Auto Subtitles Are Drafts: Why 99% Accuracy Isn’t the Finish Line

In one test clip, the auto subtitles looked almost perfect. Then one auto subtitle showed gp where the speaker had actually said HP. It was one token in a long transcript, and that was exactly the problem: nothing in the editor made it look more dangerous than the clean words around it.

Disclosure: AI helped me edit and structure this article. The gp / HP mistake came from my own build, and I checked the technical details against the code and the working editor.

I ran into this while building a subtitle editor. The ASR system already returned word-level timing and confidence values, but a polished block of text made every word look equally trustworthy. The model exposed uncertainty; the interface hid it.

That led me to a narrower engineering conclusion:

Auto subtitles are drafts. An accuracy score describes a model result; it does not define a finished review workflow.

Why auto subtitles need more than one accuracy percentage

Speech-to-text systems are often evaluated with word error rate, or WER. In its simplest form:

WER = (substitutions + deletions + insertions) / reference words

That is useful for comparing transcripts against a known reference. For auto subtitles, trouble starts when a model-level metric is turned into a product-level promise.

Suppose a 100-word transcript contains one wrong word. Its word accuracy may look excellent. But a single auto subtitle can carry very different consequences:

• Changing “and” to “an” may be harmless.
• Changing a person’s name damages trust.
• Changing 15 to 50 changes the meaning.
• Changing HP to gp made my test caption look careless.
• Dropping “not” reverses the sentence.

WER counts errors. It does not price their consequences.

Good auto subtitles also depend on things that a transcript-only score does not fully describe:

• whether words appear at the right time;
• whether cue boundaries follow the sentence;
• whether a line is readable before it disappears;
• whether punctuation helps or hurts comprehension;
• whether the user knows what remains unchecked.

A system can therefore have a strong aggregate score and still produce a bad editing experience.

Confidence is metadata, not a workflow

For auto subtitles, word-level confidence is more actionable than a single document score. It can help answer a useful question: where should a reviewer look first?

But a confidence value is still model output, not automatically a calibrated probability. Research on ASR word confidence proposes explicit score calibration so values can be compared across models. A high-confidence token can still be wrong. A missing confidence value should not silently become “high confidence.”

When an ASR response enters the app, I normalize each word into text, timing, and a finite review score. If the response includes a usable raw score, the app converts it; an absent raw score is normalized to 0, the lowest review score, so the token enters the review queue instead of looking certain. I then turn each word into an editor word that also records where it came from:

type EditorWord = {
  id: string;
  text: string;
  startMs: number;
  endMs: number;
  confidence?: number;
  source: 'asr' | 'manual';
};

The optional confidence field is deliberate at the editor layer. ASR words arrive with a normalized review score; a manually created word should not inherit a score the model never produced.

The two-step contract matters more than the TypeScript: each returned word must include text as a string, plus start and end times as non-negative numbers, while confidence must either convert to a finite review score or follow the explicit conservative 0 fallback. The editor then preserves that normalized evidence while recording whether a word came from ASR or a manual edit. A response outside that input shape is rejected instead of having missing fields invented later.

Even with that contract, rendering every confidence value is not a review workflow. A transcript with hundreds of colored words simply replaces one haystack with another.

Turn auto subtitle uncertainty into a finite review queue

When working with auto subtitles, I found it more useful to separate three things:

1. The transcript is the editable content.
2. Review targets are the subset the system asks a human to inspect.
3. Resolved targets record progress through that finite set.

A public version of the state only needs two lists:

type ReviewState = {
  candidateIds: string[];
  reviewedIds: string[];
};

function pendingIds(state: ReviewState): string[] {
  const reviewed = new Set(state.reviewedIds);
  return state.candidateIds.filter((id) => !reviewed.has(id));
}

candidateIds defines the finite review job. reviewedIds records the work the user has already done. Pending work is simply the difference between them.

There is no universal recipe for choosing review targets. In my editor, tokens are ranked by confidence and time, and the queue is deliberately bounded. If every below-threshold word were surfaced without a cap, noisy audio could create a review experience almost as expensive as rereading the whole transcript.

I do not think there is a universal confidence threshold. A sensible policy depends on the ASR system, language, audio, domain vocabulary, and the cost of a miss. The threshold should be measured against your own material, not copied from a code sample.

The resulting pipeline is straightforward:

word-level ASR result
-> validate the ASR result
-> build timed subtitle cues
-> rank review candidates
-> create a finite review queue
-> accept or edit each target
-> export

The value is that uncertainty becomes a job the user can finish. Instead of asking the user to “check everything,” the interface offers a bounded task with visible progress.

Define “All clear” precisely

The phrase “All clear” is easy to overstate. It should not mean:

• the transcript is objectively perfect;
• the model achieved a guaranteed accuracy level;
• no unflagged error exists;
• the subtitles satisfy legal, medical, or accessibility review requirements.

In this workflow, it means something narrower and testable:

Every item in the current review queue has been accepted or edited.

The user can accept a flagged word when it is already correct, or edit the cue when it is not. Accepting moves that word into the checked list. Editing changes the cue and drops target IDs that no longer point to its current words. Re-transcription creates a new transcript, so it must create a new review list too.

That last rule matters. Review state is derived from a particular transcript. If the transcript changes but the old “resolved” state survives, the UI can display completion for words the user has never seen.

“All clear” is therefore a workflow state, not a quality certificate.

Failure modes worth designing for

A high-confidence auto subtitle mistake

Names, acronyms, numbers, and domain terms can be wrong even when the model is confident. A confidence queue reduces search cost; it does not remove the need for a final watch when the content matters.

Missing confidence or an invalid input shape

Treating absent confidence as 1.0 turns missing evidence into certainty. In this implementation, an absent raw score is normalized to 0, the lowest review score, so it cannot look certain. A returned word must still provide text as a string and start and end times as non-negative numbers; a response outside that shape is rejected. Another system could choose a different explicit fallback, but it must not treat missing confidence as high confidence.

Stale review decisions

After cue editing or range re-transcription, old token IDs may no longer exist. Prune both pending and resolved IDs against the current transcript before showing completion.

Too many warnings

A review queue that flags half the transcript is not a shortcut. Bound the queue, expose why an item was selected, and provide a way to review the full text when the audio is genuinely difficult.

Source confidence leaking into translated text

If the editor has original and translated subtitle tracks, do not paint source confidence onto translated text. The uncertainty belongs to the source ASR result. A translated track can link back to unresolved source words without pretending those English token scores describe the translation.

Where this approach fits—and where it does not

A finite uncertainty queue works well for creator videos, interviews, courses, and other workflows where a person can quickly confirm a small set of likely problems.

It is not enough by itself for high-risk transcription. Legal, medical, safety, and formal accessibility workflows may require full human review, speaker verification, domain-specific checks, audit trails, and standards beyond a general-purpose creator tool.

The right claim is not “human review is no longer needed.” It is “the system can make human review smaller and more explicit.”

A builder checklist for auto subtitles

If you are turning ASR output into an editable product, I would check these before calling the workflow complete:

• Preserve word text, timing, confidence, and source.
• Normalize missing confidence conservatively, and reject words outside the required text-and-timing input shape.
• Keep transcript content separate from review progress.
• Rank uncertainty instead of painting the whole transcript.
• Bound the queue so that review remains finite.
• Let users accept a correct token without editing it.
• Invalidate stale review state after editing or re-transcription.
• Keep source-word confidence out of translated tracks.
• Define exactly what your completion label means.
• Never present queue completion as guaranteed transcript accuracy.

I built this review queue into SubtitleGeneratorafter seeing how easy it was to miss one wrong word in otherwise clean auto subtitles.Uncertain source words become a finite queue, and “All clear” appears only when that queue is empty.

How do you handle ASR uncertainty in your own products: a confidence threshold, a full transcript review, or a separate review state?