Fixing one typo in a caption used to throw away the word timings for the whole line

CapScribe burns captions into video on the iPhone, and one of its caption styles highlights each word as it is spoken. That only works because the on-device transcriber hands back a start time and a duration for every word. Then someone fixes a typo, and you have to decide what happens to those timings.

What it looked like

Until the 2.4 release, the answer was: they disappear. The edit handler set the segment's words to nil and moved on.

The renderer handles that case carefully. A segment with no word timings falls back to a static caption instead of inventing timings, and the editor shows a note saying so. That is honest, but from the user's side it is baffling. You correct "quik" to "quick", and the highlight on that line just stops. Fix a few typos across a video and a few lines go static.

Why not something simpler

Re-running recognition every time someone edits one line is a lot of machinery for a typo. Spreading the segment's duration evenly over the new words is cheap, but it throws away good data. In a four-word line where one word changed, three of the timings are still exactly right, and anything that recomputes them can only make them worse.

So the rule became: words the user didn't touch keep their timing byte for byte, and only the edited stretch gets new numbers.

What we do now

WordTimingRealigner is a pure function. It takes the old words, the old text, the new text and the segment bounds, and returns new word timings or nil.

Step one is a word-level longest common subsequence between old and new tokens, compared case-insensitively, with punctuation dropped during tokenization. Matched words become anchors and are copied across unchanged, confidence included. That alone means editing "hello world" into "Hello, world!" returns the original array, and the unit test asserts equality on the whole thing.

Step two fills the gaps. Each run of unmatched new tokens sits between two anchors, so it gets the window from the end of the previous anchor to the start of the next one, split by character count:

let totalChars = gapTokens.reduce(0) { $0 + max(1, $1.count) }
for (idx, tok) in gapTokens.enumerated() {
    let dur = idx == gapTokens.count - 1
        ? winEnd - curr   // last token absorbs rounding
        : winDuration * (Double(max(1, tok.count)) / Double(totalChars))
    output[gapStart + idx] = WordTiming(text: tok, start: curr, duration: dur, confidence: nil)
    curr += dur
}

Generated words get confidence: nil, so nothing downstream mistakes a guess for recognizer output. The last token in a gap takes whatever time is left, so floating-point drift can't push it into its neighbour.

Two details took longer than the algorithm itself.

The LCS table is built from the suffix end, which lets the backtrack walk forward. When a word repeats ("the cat saw the dog"), it takes the earliest match that still keeps the subsequence maximal, so the same edit always produces the same anchors.

Chinese has no spaces, and the stored words might be whole words or single characters. Before matching, the function checks whether the stored words agree with enumerateSubstrings(byWords) or with a one-token-per-character split, and tokenizes the new text the same way. If neither agrees, it doesn't guess.

Where it gives up

It returns nil, and that segment falls back to a static caption, when:

• nothing matches at all (the line was rewritten),
• the stored words don't validate against the old text, overlap each other, or fall outside the segment,
• an insertion lands between two anchors that touch with zero gap, so there is no window to put the new word in.

The last case is a real limitation. If the recognizer reported two words back to back and you insert a word between them, that line still loses its highlight. Borrowing time from the neighbours would fix it, but it breaks the rule that untouched words never move, and I would rather show a static line than shift a correct timestamp.

The output is checked again before it is returned: right count, every duration positive, monotonic, inside the segment. A property test runs 50 seeded random edits (replace, insert, delete, punctuation) and asserts those invariants whenever the result isn't nil.

The cost

Splitting by characters is a guess. A long word does not take proportionally longer to say than a short one, so the highlight on a changed word can land visibly early or late. For a typo fix inside one word, the guess is boxed in by the anchors on both sides, so the error can't spread past that word's old window plus any silence around it. Bigger rewrites get less precise, and a full rewrite gets nothing, which is the point.

The table is O(M×N) per edit. That is fine for a caption line and not something I would run over a whole transcript.

---

I build CapScribe, an iOS app that transcribes and burns in captions on the device: CapScribe on the App Store