Sequel to My LLM could not tell a timelapse from real time — so I taught it physics.
I build crv, an open-source tool that turns videos into something an LLM can actually read: scene-aware keyframes, a timestamped transcript, and a fused timeline. This week two of its senses started lying to it, and fixing that taught me one lesson worth writing down.
The ear lies: whisper invents captions over music
An 8-second, music-only clip came back with the caption "I'll see you next time." Nobody says anything in the clip. Whisper's decoder has seen too many outros — music at the end of a video "should" have that line, so it writes it.
The standard fix works: switch to faster-whisper and enable its Silero VAD (voice-activity detection) gate:
model.transcribe(wav,
vad_filter=True,
vad_parameters={"min_silence_duration_ms": 500},
condition_on_previous_text=False)
Segments with no detected speech never reach the model. Nothing goes in, nothing gets invented.
The bug that was mine, not whisper's
Here is the part I have not seen written down anywhere. My pipeline had a fallback: if the fast engine returns nothing, fall back to the whisper CLI. Sounds harmless — until the VAD gate correctly hears no speech and returns an empty segment list. My code read "empty" as "engine failed", fell back to the ungated CLI, and the phantom caption walked right back in through the back door.
An empty result is an answer, not an error. If you put a gated path in front of a fallback path, make sure the gate's verdict can't be overruled by your own plumbing. My manifest now says "the voice-activity gate heard no speech; music/ambient-only audio" — an honest sentence instead of a fake caption.
The eye corrects the ear: OCR as ground truth
Short-form video is wall-to-wall burned-in captions, and those captions are the video's own script. So crv Pro now OCRs every kept frame and places on-screen text on the same timeline as the ASR transcript.
Real example from a Chinese video I processed yesterday: whisper heard 猴狼 ("monkey-wolf" — not a word), while the burned-in caption at the same second clearly said 后浪 ("the rising generation", the video's whole point). Same timestamp, two readings. The manifest tells the reading LLM: prefer on-screen wording over the ASR transcript for names, numbers and terms.
The ear mishears; the eye reads the script. Cross-modal redundancy beats either sense alone.
Takeaways
- VAD-gate your ASR. Hallucinated captions poison everything downstream.
- Audit your fallback paths — a correct empty result must not trigger a fallback to the thing you were protecting against.
- If the video carries its own text, treat it as ground truth and let it correct the transcript.
Everything above ships in claude-real-video 0.7.13 (free, MIT) and crv Pro 0.8.12. All local — nothing leaves your machine.
Top comments (4)
"An empty result is an answer, not an error" is the line I wish I'd read two years ago. I hit the mirror of your fallback bug on a push-to-talk dictation tool: same faster-whisper + VAD, except each clip is its own isolated
transcribe()call of a few seconds. The knob that flips there iscondition_on_previous_text. You set it False, which is right for a long stream, but on push-to-talk I keep it True, because nothing carries from one clip to the next, so the context stays inside a single dictation instead of amplifying loops. Same knob, opposite value, purely because the clip boundary moved.The other silence your VAD gate quietly kills: a push-to-talk clip almost always has half a second of dead air at the head and tail (the time to press and release the key), and that is exactly where the phantom "thank you" gets born. Gating at the source beat every post-hoc cleanup I tried.
On the OCR-as-ground-truth call, do you ever hit a case where the burned-in caption is itself wrong (auto-generated subs) but the audio was actually right? How do you break that tie?
The push-to-talk mirror is a great proof that
condition_on_previous_texthas no universally right value — it's a property of the clip boundary, not the model. Long stream: context is fuel for loop amplification, so False. Isolated clips: context can't leak across clips because there is no across, so True buys you intra-clip coherence for free. Same knob, and the correct setting flips the moment you move the boundary. And you're right about where the phantoms are born — the dead air around the keypress is exactly the "silence that was never speech" case; gating at the source beats cleanup because post-hoc you can no longer tell confident hallucination from quiet speech.On the tie-break question — honest answer: I don't break it automatically, and that's deliberate. OCR in my pipeline isn't ground truth in the overwrite sense; it's a second independent witness. Burned-in text lands in the fused timeline as its own track with per-line OCR confidence (RapidOCR, lines under 0.6 dropped), alongside the transcript with its own confidence. When they disagree, the reading model sees both claims with timestamps and has to reconcile — same discipline as the empty-result rule: present the evidence, don't silently pick a winner. In practice most ties break themselves because the failure modes don't overlap (ASR fails on homophones, OCR fails on fonts/low contrast). Auto-generated burned-in subs are the nasty case precisely because they fail in ASR-shaped ways, so agreement between my whisper pass and the burned-in text is correlated evidence, not two independent votes. I'd rather surface that disagreement than launder it into false certainty.
That distinction between two witnesses and two correlated votes is the part I keep having to re-learn. Agreement between your whisper pass and burned-in auto-subs is not two votes, and the same trap sits one layer down: running two Whisper sizes and reading consensus as confidence. They share training data, so they fail on the same homophones and the same proper nouns. The agreement is real, the independence is not.
The only thing that ever bought me a genuinely independent witness in dictation was boring: a deterministic post-pass, plain regex mapping the way the model reliably mangles a term onto the right term. It has zero shared failure mode with the acoustic model because it is not a model. It is dumb, it is manual to maintain, and it is the one part of the pipeline whose errors I can fully enumerate.
Your per-line confidence floor is the bit I had not thought to steal. Dropping OCR lines under 0.6 instead of letting them argue is basically saying a weak witness should not get a vote at all. I have been treating my confidence signal as something to surface rather than something to gate on, and gating is probably the more honest move, especially since a low-confidence transcript line and a quiet-but-correct one look identical after the fact.
Yeah, that reframes the whole thing for me. OCR was never a second opinion. It's a witness from a different sense organ. The eye and the ear break on completely different things. Whisper drops a proper noun it never heard, but the pixels on screen literally spell it out. OCR chokes on a stylized logo, but the audio just says it. Two Whisper passes share the same blind spots, so when they agree it's really just the same guess twice. The only agreement worth anything is the kind that comes across two different senses.
Your regex post-pass is the same idea from the other side. It's independent because it's rule based, there's no shared training data to make the errors line up. Boring and deterministic, and that's exactly why it catches what the models miss. Honestly the more I sit with it, the more it feels like the whole rule is just don't trust two witnesses who went to the same school.