1,377 frames in, 60 out, and none of them knew what time it was

#opensource #ai #python #ffmpeg

A user opened an issue on my open-source video tool last week that named a gap I had been shipping around for months.

He runs lectures through crv so an LLM can read them. One 22-minute lecture: 1,377 candidate frames extracted, 60 kept after dedup and --max-frames thinning. The 60 frames come out in the right order. That is all they come out with.

His complaint, in one line: the LLM can describe the slide, but it cannot tell you when the slide was on screen.

That breaks more than it sounds like:

you cannot cite visual evidence with a timestamp
you cannot line a chart up against the nearby transcript.json segments
you cannot jump from a keyframe back to that moment in the video
you cannot verify the claim afterwards

The transcript had timestamps the whole time. The frames did not.

Why the timestamps died

The pipeline goes: extract with ffmpeg, drop near-identical frames, thin down to --max-frames, rename everything to frame_001.jpg, frame_002.jpg.

Every one of those steps is lossy for position. Extraction writes files, dedup deletes some, thinning deletes more, renaming closes the gaps. By the time you are holding frame_012.jpg, the only fact left in the filename is "twelfth surviving frame", and twelfth of what is no longer recoverable from the output directory.

The tempting fix is arithmetic: timestamp = frame_number / fps. That is wrong on any variable frame rate source, which covers most screen recordings and a lot of phone video. It gives you a number that looks right and drifts.

What actually works

ffmpeg already knows. The showinfo filter prints the real PTS of every frame it passes, on the same select pass you are already running:

-vf "select=...,showinfo"

Parse that log and you get true presentation timestamps with no second decode pass. Then you carry them: attach the PTS at extraction, keep it attached through dedup, through thinning, through the rename, and write it out next to the images as frames.json:

{
  "frames": [
    {
      "file": "frame_001.jpg",
      "timestamp_sec": 18.42,
      "timestamp": "00:00:18.420",
      "selection_reason": "scene"
    }
  ]
}

selection_reason records which dedup channel kept the frame. That one is worth adding early: it is what you read when a frame you wanted is missing and you need to know which stage ate it.

The part I nearly skipped

If the showinfo log and the extracted files ever disagree on count, the tool writes no timestamps at all rather than approximate ones.

That felt overly strict while I was writing it. It is the opposite. A missing timestamp makes the model say "I don't know when". A wrong timestamp makes it cite 00:03:41 with total confidence, and nothing downstream can catch it. In a pipeline whose entire job is handing a model verifiable evidence, a plausible wrong number is the worst thing you can emit.

Did it hold up

The person who filed the issue re-ran his 22:12 lecture on the new build and checked it himself: 1,377 candidates down to 60 final frames, 60 entries in the mapping, all monotonic, no missing or extra image files. He replayed the full extraction pass against the original source and matched every final image back to its recorded timestamp. 60 out of 60.

I did not ask him to do that. It is the most useful thing anyone has done for this project.

Takeaway

If you build any extract, filter and rename pipeline that feeds an LLM, decide early where position lives. Threading an identifier through four stages is much cheaper than reconstructing it from a directory listing afterwards. And when the identifier is uncertain, emit nothing instead of something.

crv is MIT and on PyPI: