Why your video search keeps returning the wrong scene

Your search usually returns the wrong scene because of how the video was cut up before it was indexed. If you embed fixed time windows or every Nth frame, one vector can straddle two scenes, and the match lands on the neighbour. If you embed one vector per whole video, every query returns the same file. The fix is to segment first: detect shot boundaries, group shots into scenes, then embed each scene and store its start and end timestamps. A query then matches a coherent unit and returns the moment itself.

Where the wrong scene comes from

Fixed windows. Splitting every 10 seconds ignores where the cuts are. A window that holds the last three seconds of a kitchen scene and the first seven of a street scene produces an embedding that is half of each. Ask for "street at night" and you may get the kitchen timestamp.

Frame sampling without boundaries. Sampling one frame a second gives you lots of vectors, many nearly identical. The nearest neighbour is often a frame from a similar-looking shot elsewhere in the video.

One vector per video. Averaging a whole video into one embedding throws away where anything happens. Search can find the right file and still cannot tell you where in it.

How segmentation fixes it

1. Shot boundary detection. Compare consecutive frames. A hard cut spikes a histogram difference. Dissolves and fades spread the change over many frames, so an adaptive detector flags a boundary when the score beats a rolling local mean by two to three standard deviations.
2. Group shots into scenes. A shot is one camera take. A scene can span many shots, like a conversation cutting between two angles.
3. Clean up. Merge segments under about two seconds into a neighbour, and split long ones at audio silence.
4. Embed per scene, keep timestamps. One embedding per scene, stored with its start and end time. Results come back as moments.

This also cuts index size by 10 to 100 times, because you store one vector per scene.

Quick checks on your own pipeline

• Print the boundaries of the scene your query wrongly returned. If a cut falls inside it, segmentation is the problem.
• Check whether dissolves are being missed. A fixed threshold catches hard cuts and misses gradual transitions.
• Check whether on-screen text or speech carries the meaning. If so, a visual embedding alone will keep missing, and you need OCR or transcript features on the same scene.

The full walkthrough, with the algorithms and thresholds: https://mixpeek.com/guides/video-scene-segmentation

Disclosure: I work on Mixpeek.