DEV Community

Cover image for Claude can't watch your screen recording. Here's what it can read instead
Alex @Bickov
Alex @Bickov

Posted on Edited on

Claude can't watch your screen recording. Here's what it can read instead

Claude can't open your screen recording. Anthropic's vision docs list JPEG, PNG, GIF and WebP as input formats, and for a GIF only the first frame is used. Codex and ChatGPT take images too, not MP4.

That hurts most with bugs that only show up as a sequence. You click Save, a spinner shows, the page jumps back and nothing is saved. One screenshot can't show that.

So I took one 43 second repro, 3 screens and a few clicks on X.com, and sent it four ways.

1. One frame per screen

Pull a still at each screen change and send those. Claude 4.7 counts image tokens in 28px patches, ceil(width/28) x ceil(height/28), so a 1440x900 frame is 52 x 33 = 1,716 tokens. Three screens come to about 5,150 tokens.

Cheap and sharp. But Claude sees screen A and screen B with nothing in between, so the click that caused the bug is a guess.

2. One frame per second

To catch what happens between screens, sample every second instead. 43 frames is about 73,800 tokens, and once a request carries more than 20 images the per-image size limit gets stricter.

You pay 14 times more and the click still isn't there. A frame shows where the pointer was, not what it pressed.

3. Send the MP4 to Gemini

Gemini takes video directly. It samples 1 frame per second at about 100 tokens per second at default resolution, so the clip costs about 4,300 tokens.

Cheapest of the four, and the only one that keeps the motion. It still sees one frame a second, so a quick click between two samples can disappear, and the answer comes from Gemini, not from the agent that has your code open.

4. The recording as JSON

This one skips pixels. SlimSnap records the screen with voice and writes JSON next to the MP4. Each screen you visit becomes a frame with its text and positions.

The part that matters for bugs is Interactions. Every click that changes the screen becomes a step, with the control it hit and the screen it led to. Scrolls are logged too, and each spoken sentence is tied to the element under the pointer. Keystrokes are never recorded, so a password typed during the repro stays out of the file.

The same 43 seconds came to 6.3k tokens of JSON.

More than Gemini, about a twelfth of one frame per second. What you get for it is that "I clicked Save and nothing happened" reaches Claude or Codex as an actual step: the Save control, the screen before, and the screen after.

Which one to send

Token counts for the same 43 second recording sent four ways

Your bug Send
Visible on one screen One cropped screenshot
Spread over a few screens, clicks don't matter One frame per screen
A flow, and you ask Gemini The MP4
A flow, and you ask Claude or Codex The recording as JSON with the clicks

Reproduce the numbers

Tokens for one frame on Claude 4.7:

python3 -c "import math;w,h=1440,900;print(math.ceil(w/28)*math.ceil(h/28))"
Enter fullscreen mode Exit fullscreen mode

A still at a screen change:

ffmpeg -ss 4.5 -i repro.mp4 -frames:v 1 screen2.png
Enter fullscreen mode Exit fullscreen mode

One frame per second for the whole clip:

ffmpeg -i repro.mp4 -vf fps=1 frame_%02d.png
Enter fullscreen mode Exit fullscreen mode

Top comments (0)