DEV Community

Leo Huang
Leo Huang

Posted on

Your LLM can't actually watch video. Here's the smallest fix (MIT)

#llm #ai #python #opensource

Every model card says "multimodal". Then you hand the model a real video file and discover what that means in practice: ChatGPT reads the subtitle track, Claude doesn't accept video files at all. The model narrates a video it mostly never saw.

I unpack viral videos daily for my own content work, so I couldn't route around this. I built a small tool instead.

The mechanism

claude-real-video converts a video into three things an LLM can genuinely read:

  1. Scene-aware sampled frames — ffmpeg scene scores decide where to sample, so you get a frame when the picture changes, not every N seconds. An --adaptive flag handles slow deformations (a real user bug report: fixed thresholds missed squash/stretch morphs entirely).
  2. A timestamped transcript — whisper by default; if faster-whisper is installed it runs in-process and several times faster, with automatic fallback.
  3. One MANIFEST timeline — frames and transcript merged into a single file, so the model follows the video in order instead of guessing from fragments. A --text-anchors flag force-samples frames at subtitle cues so on-screen text never falls between frames.

Then you point any LLM at the output folder — Claude, GPT, Gemini, or a local model. No API of mine in the middle, everything runs on your machine.

Usage 

pip install claude-real-video
crv "video.mp4" -o out
Enter fullscreen mode Exit fullscreen mode

Honest limitations

Not real-time — a 90-second video takes about 1–2 minutes all-in on an M-series Mac. Frame sampling can still miss motion between frames; the flags above patch the worst cases, both born from real GitHub issues.

It's MIT, currently at 1,731 stars with ~8k installs last month, which taught me the problem was never just mine:
https://github.com/HUANGCHIHHUNGLeo/claude-real-video

Top comments (0)