I Stopped Trusting YouTube Captions After an AI Agent Built the Wrong Thing

딩몬 — Thu, 27 Aug 2026 07:38:34 +0000

Last month an agent I was working with read a tutorial's auto-generated caption — "embedding dimension 136" — and built vectors at the wrong size. Every downstream call failed. The screen, if anyone had looked, said 1536.

That's the bug that became tuto: https://github.com/dingmon1019/YoutubeAnalyzer

The problem

Captions are a lossy transcript of speech, not a record of what's on screen. Tutorials, dashboards, and slides put the values that matter on screen, often never spoken aloud. An agent that only reads captions is reading half the video.

What I tried first

Just sample more frames. This gets expensive fast and doesn't solve the real problem: you still need to know which frames matter, and whether the video needs frames at all. A 28-minute music playlist has zero screen information worth $5 of vision calls.

The architecture that worked

A screen-value gate reads the transcript first and decides high or low before spending anything on vision. Low stops and asks -- a wrong verdict here costs one question, not a wasted run.
Frames are selected by signal (YouTube heatmap, chapter boundaries, activity peaks, caption cues like "click here"), not uniform sampling.
One agent, one context, sees every frame start to finish and synthesizes knowledge in the same context that saw the screens -- splitting this into separate read and synthesize agents measured 36% more expensive.
Every claim carries its evidence source (frame or transcript). When they disagree, the frame wins, and the disagreement is preserved as data, not silently resolved.

Results

On a 54-minute screen-dense tutorial: 171 knowledge items, 69 traceable to actual pixels, $4.76, 14.6 minutes wall clock. 785 tests passing.

What I'd do differently

The gate that decides whether a screen is worth reading is itself an LLM judgment call, which this project's own measurements say is noisy -- so I deliberately put it where losing is cheap (one question) instead of where it's expensive (a wasted 15-minute run).

It's a Claude Code plugin, MIT licensed: https://github.com/dingmon1019/YoutubeAnalyzer