DEV Community

HUANGCHIHHUNG
HUANGCHIHHUNG

Posted on

The eye corrects the ear: fixing my LLM's video hallucinations with OCR and a VAD gate

#ai #whisper #llm #opensource

Sequel to My LLM could not tell a timelapse from real time — so I taught it physics.

I build crv, an open-source tool that turns videos into something an LLM can actually read: scene-aware keyframes, a timestamped transcript, and a fused timeline. This week two of its senses started lying to it, and fixing that taught me one lesson worth writing down.

The ear lies: whisper invents captions over music

An 8-second, music-only clip came back with the caption "I'll see you next time." Nobody says anything in the clip. Whisper's decoder has seen too many outros — music at the end of a video "should" have that line, so it writes it.

The standard fix works: switch to faster-whisper and enable its Silero VAD (voice-activity detection) gate:

model.transcribe(wav,
    vad_filter=True,
    vad_parameters={"min_silence_duration_ms": 500},
    condition_on_previous_text=False)
Enter fullscreen mode Exit fullscreen mode

Segments with no detected speech never reach the model. Nothing goes in, nothing gets invented.

The bug that was mine, not whisper's

Here is the part I have not seen written down anywhere. My pipeline had a fallback: if the fast engine returns nothing, fall back to the whisper CLI. Sounds harmless — until the VAD gate correctly hears no speech and returns an empty segment list. My code read "empty" as "engine failed", fell back to the ungated CLI, and the phantom caption walked right back in through the back door.

An empty result is an answer, not an error. If you put a gated path in front of a fallback path, make sure the gate's verdict can't be overruled by your own plumbing. My manifest now says "the voice-activity gate heard no speech; music/ambient-only audio" — an honest sentence instead of a fake caption.

The eye corrects the ear: OCR as ground truth

Short-form video is wall-to-wall burned-in captions, and those captions are the video's own script. So crv Pro now OCRs every kept frame and places on-screen text on the same timeline as the ASR transcript.

Real example from a Chinese video I processed yesterday: whisper heard 猴狼 ("monkey-wolf" — not a word), while the burned-in caption at the same second clearly said 后浪 ("the rising generation", the video's whole point). Same timestamp, two readings. The manifest tells the reading LLM: prefer on-screen wording over the ASR transcript for names, numbers and terms.

The ear mishears; the eye reads the script. Cross-modal redundancy beats either sense alone.

Takeaways

  1. VAD-gate your ASR. Hallucinated captions poison everything downstream.
  2. Audit your fallback paths — a correct empty result must not trigger a fallback to the thing you were protecting against.
  3. If the video carries its own text, treat it as ground truth and let it correct the transcript.

Everything above ships in claude-real-video 0.7.13 (free, MIT) and crv Pro 0.8.12. All local — nothing leaves your machine.

Top comments (0)