Turning a 5-minute talk into 7 images for an LLM: picking one sharp frame per scene

Nookforge Studio — Sun, 01 Nov 2026 22:03:14 +0000

If you send a video to a vision LLM, the usual first step is to sample a frame every few seconds. On a 4:44 conference talk, one frame every 10 seconds gives you 28 images. The talk has 7 slides. You pay for 21 repeats, and some of the 28 catch a slide halfway through a transition or before its last bullet appears.

What you actually want is one sharp frame per scene: per shot in a film, per slide in a talk. Here is how I built that, with numbers from real videos.

1. One cheap scan

Decode the video once, at 320 px wide in grayscale, and keep 0.5 to 4 samples per second depending on length. For each sample, compute three numbers:

Sharpness: variance of the Laplacian.
Change: mean absolute difference between this sample and the previous one, on a 64×36 thumbnail.
Flatness: whether the frame is nearly uniform (fades, black screens).

This pass is the only full decode of the video, so it sets the cost. Two ffmpeg flags help a lot on H.264: -skip_frame noref -skip_loop_filter all made the scan 1.85× faster, and since we only need a few samples per second at 320 px, nothing is lost. (VP9 and Theora ignore them.)

2. Find scene cuts with an adaptive threshold

A fixed threshold fails in both directions. Hard cuts in a film produce big jumps. Slide changes in a filmed talk jump by as little as 0.017, because the slide only fills part of the frame next to the speaker's camera. Camera motion, meanwhile, produces a steady stream of moderate differences for seconds at a time, so a low fixed threshold would cut on every pan.

So a cut is either a hard jump, or a jump that is large relative to the recent baseline:

def find_cuts(diff, fps, hard=0.15, floor=0.025, min_scene=1.0):
    """Indices where a new scene starts: hard cuts, or jumps far above the recent baseline."""
    win = max(3, int(round(fps * 2)))
    min_len = max(1, int(round(min_scene * fps)))
    cuts = [0]
    for i in range(1, len(diff)):
        if i - cuts[-1] < min_len:
            continue
        local = float(np.median(diff[max(0, i - win):i])) if i > 1 else 0.0
        if diff[i] >= hard or (diff[i] >= floor and diff[i] >= 4.0 * local + 0.004):
            cuts.append(i)
    return cuts

On a static slide deck the baseline is about 0, so a 0.017 jump stands out. During a pan the baseline is high, so motion doesn't trigger cuts.

3. Pick the best frame of each scene

Inside each scene, skip the first and last 15% (transitions live there), skip flat frames, and score the rest by sharpness divided by local motion:

score = sharpness / (1 + 20 × motion)

For slides this naturally picks the fully built slide, because more text means more edges.

4. Drop blur and duplicates

A pick is blurry if its sharpness is under 25% of the video's 75th-percentile sharpness. That catches motion blur and focus pulls without penalizing soft-looking films.

Duplicates were the interesting bug. I started with a 64-bit difference hash (a 9×8 thumbnail), the usual choice. On the talk it merged different slides: at 9×8, every slide is "white box on the left, dark webcam on the right." What works is comparing the 64×36 thumbnails directly and calling two frames duplicates only when under 1% of pixels changed noticeably. Different slides pass, and a shot the video cuts back to is kept once.

5. Grab the full-resolution frames in one pass

Starting one ffmpeg process per frame cost about a second per frame, mostly process startup plus decoding forward from the previous keyframe. One decoding pass with a select filter was 5.7× faster on a 70-second clip (21 frames in 3.7 s instead of 21 s).

My first version selected by frame number. It broke on an Ogg file that decoded 1,113 frames where 1,254 were expected, so the last targets never existed. Selecting by timestamp, the first frame at or after each target time, works on variable frame rates and dropped frames:

select='gte(t\,9.8)*lt(prev_pts*TB\,9.8)+gte(t\,16.8)*lt(prev_pts*TB\,16.8)+...'

For very long videos, seeking to each frame is still cheaper than decoding everything again, so the code picks whichever is faster using the measured scan time.

Results

Fixed sampling uses ffmpeg's fps=1/10:

Video	Every 10 s	Smart keyframes
4:44 talk with slides (high sensitivity)	28 frames, 7 distinct slides	7 frames, one per slide, all bullets visible
12:14 film, 1080p (Tears of Steel)	73 frames	159 scenes, 24 blurry picks dropped, best 100 kept
32 s trailer (Big Buck Bunny)	3 frames	16 frames, one per shot

Try it

I packaged this as Smart Video Keyframes on Apify. Pass video file URLs and get keyframes with timestamps, a contact sheet and, optionally, the on-screen text of each frame. Two ready-made examples:

Videos via Wikimedia Commons: "Research and Privacy on Wikipedia" lightning talk by Eli Asikin-Garmager (CC0), used for the cover; Tears of Steel and Big Buck Bunny by the Blender Foundation (CC BY 3.0).

Why object detectors miss small objects in big photos, and how tiling fixes it

Nookforge Studio — Sun, 01 Nov 2026 20:45:16 +0000

Point an off-the-shelf object detector at a drone photo of a marina and it will tell you there are five boats. There are dozens. The model isn't bad; the pipeline throws the boats away before the model ever sees them.

The resize problem

Most real-time detectors, YOLO and its relatives included, are trained and run at around 640×640 pixels. A 3613×2400 aerial photo gets shrunk by a factor of about 5.6 to fit. A boat that was 30 pixels long becomes about 5 pixels long, which is below what the network can reliably pick out. The same thing happens to background people in crowd shots and to cars in parking lot photos.

Sliced inference

The fix is to stop shrinking the image and look at it in pieces instead:

Cut the image into overlapping tiles at full resolution, 640 px with about 20% overlap.
Run the detector on every tile, plus one pass on the whole downscaled image so large objects that span tiles are still found.
Shift each tile's boxes back into full-image coordinates.
Merge duplicates. Plain IoU-based non-max suppression is not enough, because an object cut by a tile edge produces a partial box that barely overlaps the full one. Matching on intersection-over-smaller-box as well as IoU joins those fragments into one object.

Results

I ran the same YOLOX-S model on three public photos from Wikimedia Commons, once as a single pass and once tiled:

Photo	Single pass	Tiled
Aerial marina, 3613×2400	5 boats	91 boats
Festival crowd, 3586×2981	18 people	105 people
Parking lot, 2304×2304	18 cars	31 cars

These are detector outputs, not hand-checked counts, but the gap is the point: most of what the single pass missed was simply too small after resizing.

The trade-off

Tiling costs compute. The marina photo needs 35 tiles, so it runs 36 inferences instead of one. Two knobs keep that under control:

Only tile images that are meaningfully larger than the model's input. I start at about 1.5× the tile size.
Cap the number of tiles per image, and enlarge the tiles when the cap is hit.

When tiling isn't enough

Very dense crowds with heavily overlapping heads need density-estimation models, not detectors.
Straight-down satellite views of tiny objects need a model trained on overhead imagery; COCO-trained detectors expect side and oblique views.
Tiling finds more objects, and more false positives. Raise the confidence threshold or filter to the classes you need.

Try it

If you want to build this yourself, SAHI is a good Python starting point.

If you'd rather pass in image URLs and get counts, boxes and annotated previews back, I packaged this pipeline as Object Counter, priced per image. You can run the same three photos from this post:

Photo credits, all via Wikimedia Commons: aerial marina by GPlantam, CC BY-SA 4.0; festival crowd by Anthony O'Neil, CC BY-SA 2.0; parking lot by Tony Webster, CC BY 2.0. The cover image adapts the marina photo with detection boxes and labels and is shared under CC BY-SA 4.0.

DEV Community: Nookforge Studio

Turning a 5-minute talk into 7 images for an LLM: picking one sharp frame per scene

1. One cheap scan

2. Find scene cuts with an adaptive threshold

3. Pick the best frame of each scene

4. Drop blur and duplicates

5. Grab the full-resolution frames in one pass

Results

Try it

Why object detectors miss small objects in big photos, and how tiling fixes it

The resize problem

Sliced inference

Results

The trade-off

When tiling isn't enough

Try it