You have a 90-second clip and you need to pick a handful of frames out of it. Not random frames. Frames that, between them, describe the whole clip well enough that a downstream model can fill in the rest. This shows up everywhere in video tooling: you stylize a few frames and warp the look across the gaps, you run an expensive diffusion pass on keyframes only, you build a thumbnail strip, you sample frames for a vision model. The whole job hangs on which frames you keep.
The lazy answer is "every tenth frame." It is one line and it is wrong in a specific, annoying way. The fancier answer is "frames where the shot cuts." That is better and still wrong, in the opposite direction. The function I want to walk through picks both, and the reason it picks both is the interesting part.
The code is SceneDetect.detecting in portable/src/visual_generation/inference.py in Wunjo Make. It is short. Around 30 lines do the whole thing.
Why "every Nth frame" is dumb
Say you take every tenth frame. The video has a hard cut at frame 47, the camera jumps from a face to a wide landscape. Your stride lands you on frames 40 and 50. Frame 40 is the face, frame 50 is the landscape, and the cut between them is invisible to you. Now whatever you do downstream has to bridge frame 40 to frame 50 as if they were the same continuous shot. They are not. A motion-based fill will try to warp a face into a mountain and produce mush. A thumbnail strip will skip the cut entirely and make two scenes look like one.
A fixed stride has no idea where the content actually changes. It samples the timeline, not the footage. On a slow, single-shot clip it oversamples, ten near-identical frames where two would do. On a fast-cut montage it straddles every cut and never lands cleanly on the start of a new shot. The stride is blind.
Why "scene cuts only" is too sparse
So detect the cuts. PySceneDetect does this well, and the function uses it:
from scenedetect import detect, ContentDetector
scene_list = detect(source_file, ContentDetector(threshold=threshold))
ContentDetector works on content difference between consecutive frames. It converts each frame to HSV, measures how much hue, saturation and value changed frame to frame, and when that change spikes past a threshold it calls a cut. There is no neural network here and no training. It is a hand-tuned heuristic, and that is fine, because shot boundaries are mostly a sudden jump in pixels and HSV captures that jump cheaply. The default threshold here is 50.0. Lower it and the detector gets twitchy, calling cuts on fast pans and lighting changes. Raise it and it misses soft cuts. Fifty is a sane middle for general footage.
Now you have a list of scenes. The problem: a scene can be ten seconds long. If you keep only the first frame of each scene, a ten-second continuous shot of someone walking across a room gives you one keyframe. Everything that happens inside that shot, the person crossing from left to right, is gone. For a thumbnail that might be fine. For anything that has to reconstruct the in-between frames, one keyframe per shot is far too little. The motion inside the shot has no anchors.
Scene cuts tell you where the content changes hard. They say nothing about the slower drift inside a shot.
Combining both
Here is the whole selection block:
if len(scene_list) > 0:
for i, scene in enumerate(scene_list):
start_frame = scene[0].get_frames()
end_frame = scene[1].get_frames()
if interval:
scene_frames.extend(range(start_frame, end_frame, interval))
scene_frames.append(end_frame - 1)
else:
scene_frames.extend([start_frame, end_frame - 1])
else:
scene_frames.append(total_frames - 1)
else:
if interval:
scene_frames = list(range(0, total_frames - 1, interval))
scene_frames.append(total_frames - 1)
else:
scene_frames = [0, total_frames - 1]
Read it from the top. For each detected scene you have a start_frame and an end_frame. The key two lines are these:
scene_frames.extend(range(start_frame, end_frame, interval))
scene_frames.append(end_frame - 1)
The first line walks the scene from its start to its end in steps of interval (default 10), and adds every one of those frames. So inside the shot you get the start, then start+10, then start+20, and so on. That is the fixed stride, but now it is reset at the start of every scene. It cannot straddle a cut, because each scene gets its own fresh stride that begins exactly on the cut. The stride does the sampling inside the shot; scene detection makes sure the stride never crosses a boundary.
The second line appends end_frame - 1, the last frame of the scene. This matters because range(start, end, interval) rarely lands on the final frame. If a scene runs from frame 100 to frame 137 with a stride of 10, range gives you 100, 110, 120, 130, and stops. Frame 136, the last real frame before the cut, never gets picked. But that last frame is the most important one to keep, because it is the frame right before the content changes. It is what a downstream fill needs to hand off cleanly to the next shot. So the code explicitly adds it. end_frame - 1, not end_frame, because the scene's end_frame is exclusive: it is the first frame of the next scene, not the last frame of this one.
Put the two ideas together and you get the property you actually want. Keyframes are dense enough inside each shot to capture motion, and they always sit cleanly on shot boundaries, never across them. Scene detection gives the structure, the stride gives the density, and the explicit last-frame append seals each scene so nothing falls through the gap at the cut.
A keyframe per cut is too coarse; every Nth frame straddles the cuts. Striding inside each detected scene, plus the scene's last frame, gives both. Photo: Unsplash
The fallback when nothing is detected
There is a second branch, and skipping it is how you ship a crash. Some inputs produce no scenes at all. A single static shot, a very short clip, a screen recording with no real cuts. PySceneDetect returns an empty list, and if your code only handles the "scenes found" path, you get zero keyframes and a confusing downstream failure.
The else covers it:
else:
if interval:
scene_frames = list(range(0, total_frames - 1, interval))
scene_frames.append(total_frames - 1)
else:
scene_frames = [0, total_frames - 1]
No scenes means the whole video is treated as one big scene. Stride across the entire thing, then append the true last frame. It degrades to plain "every Nth frame" exactly when there are no cuts to respect, which is the one case where plain stride is the right call. The combined approach and the dumb approach converge when the footage has no structure to detect.
Dedup before you read the disk
The stride and the last-frame append can collide. A short scene where end_frame - 1 happens to be a multiple of the stride from the start will have that frame in the list twice. Reading and writing the same frame twice is wasted I/O. One line handles it:
for i in sorted(set(scene_frames)):
if i <= total_frames - 1:
cap.set(cv2.CAP_PROP_POS_FRAMES, i)
ret, frame = cap.read()
if ret:
file_path = os.path.join(local_save_dir, f"{i}.jpg")
cv2.imwrite(file_path, frame)
sorted(set(scene_frames)) does two jobs in one expression. set removes the duplicates so each frame is read once. sorted puts them back in ascending order so the seek moves forward through the file instead of jumping around. Seeking forward is much cheaper than seeking backward with cv2.VideoCapture, so reading frames in order is not just tidy, it is faster.
The i <= total_frames - 1 guard is a seatbelt. total_frames from cv2.CAP_PROP_FRAME_COUNT is not always exact, some containers report a frame or two more than you can actually decode, so the guard keeps a bad count from asking for a frame that does not exist.
 |
About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn. |
Gotchas you will actually hit
The frame numbers in the output are one-based. Look at how each kept frame is recorded: the saved entry uses "frame": i + 1. The file on disk is named by the zero-based index i (f"{i}.jpg"), but the metadata reports i + 1. If you consume this output and assume the frame field matches the .jpg filename, you are off by one. The filename is the seek index, the frame field is the human-facing count.
end_frame is exclusive. The single most common bug when you reimplement this is appending end_frame instead of end_frame - 1 and writing a frame that belongs to the next scene, or running off the end of the file. PySceneDetect's scene end is the boundary, not the last frame.
The threshold is footage-dependent. threshold=50.0 is a default, not a law. Animation, screen recordings, and high-contrast edited content cut differently from handheld video. If you get too many keyframes, the detector is over-firing; raise the threshold. Too few, lower it. The stride is the other dial: a wide interval saves frames but misses fast motion inside a shot, a narrow one captures motion but costs you frames downstream.
interval=0 is a real mode. Passing a falsy interval switches every branch to "endpoints only," start and last frame of each scene, or [0, total_frames - 1] for the no-scene case. That is the right setting when you only want shot boundaries and will handle the in-between frames yourself.
Wrap-up
Keyframe selection looks trivial until you try to use the keyframes for something. A fixed stride samples the clock and ignores the footage, so it straddles every cut. Scene detection finds the cuts but leaves long shots almost unsampled. The fix is to use scene detection for structure and a per-scene stride for density, then explicitly keep the last frame of each scene so the handoff at every cut is clean. Add a fallback that treats a cutless video as one scene, dedup with sorted(set(...)) so you read each frame once and in order, and guard against an over-reported frame count.
Thirty lines, no model weights, and the output is a frame list you can actually build on. The code is SceneDetect.detecting in visual_generation/inference.py in the Wunjo Make repo. If your own keyframe picker chokes on cuts, this is the shape to copy.
References
- PySceneDetect,
ContentDetector: detects shot changes from frame-to-frame HSV content difference. Library, no associated paper. https://www.scenedetect.com
Top comments (0)