Disclosure up front: I maintain video-slide-extractor (MIT, zero dependencies) and Video2Any, the product it came out of. The benchmark below is reproducible from two scripts, and it includes a case where my own defaults lose.
If you have ever tried to turn a recorded lecture, webinar, or Zoom call back into its slides, you have probably done one of these:
- Screenshot every N seconds (what most tutorials do)
- Compare pixels between frames and keep the frame when "enough" changed
- Something smarter
I wanted actual numbers instead of vibes, so I built a benchmark where the ground truth is exact by construction — and the results changed how I document my own library's defaults.
The problem with benchmarking slide detection
The honest way to evaluate a detector is against labeled transitions: precision, recall, F1. But hand-labeling a 45-minute lecture is hours of work, the labels have their own errors, and nobody can reproduce your benchmark without your labels.
So I flipped it: render the videos from known slide sequences. If you choose when each slide appears, every transition timestamp is ground truth by construction. Two scripts, fully reproducible:
git clone https://github.com/larry-xue/video-slide-extractor
node bench/generate-fixtures.mjs # renders the videos (ffmpeg)
node bench/run.mjs # scores the methods, writes RESULTS.md
The fixtures:
- Two slide sources. A 30-slide synthetic deck drawn by ffmpeg, and the 46 real slides that Video2Any extracted from MIT OCW 6.0001 Lecture 1 (CC BY-NC-SA 4.0, fetched at run time, not committed).
-
Three variants each, same timeline:
clean(crf 23, hard cuts),noisy(crf 45 + down/upscale), andoverlay(an animated "webcam" box in the corner — the classic meeting-recording trap). - Slide durations 4–12 s from a seeded PRNG, so every machine renders identical timelines.
One caveat that belongs in the article and not in a footnote: this measures detection on re-renderings with hard cuts, not on an original classroom recording. Crossfades, camera wobble, and in-slide builds are separate failure classes (protocol here).
The three contestants
All three see the exact same input: one 160×90 RGBA frame every 2 seconds, sampled by ffmpeg.
Fixed interval — keep a frame every 10 seconds. No content awareness at all.
Naive pixel diff — a pixel "changed" if its mean |ΔRGB| > 25; keep the frame when > 2% of pixels changed since the last kept frame.
Block diff (video-slide-extractor) — split the frame into 8×8 blocks, compute each block's mean absolute difference, count a block as changed when its mean exceeds a threshold, keep the frame when > changedRatio of blocks changed. The averaging is the point: single-pixel codec noise dilutes inside a block, while a real slide change lights up whole blocks at once. Run twice: library defaults (changedRatio: 0.02) and tuned (0.10).
Results (MIT deck, 46 slides)
Matching: each detection pairs with at most one ground-truth transition within ±2.5 s. Coverage = share of slides that got at least one capture.
| Method | clean F1 | noisy F1 | overlay F1 | overlay precision |
|---|---|---|---|---|
| Fixed interval (10 s) | 0.68 | 0.68 | 0.68 | 0.76 |
| Naive pixel diff (2%) | 0.94 | 0.94 | 0.91 | 0.85 |
| Block diff (defaults, 0.02) | 0.97 | 0.97 | 0.54 | 0.37 |
| Block diff (ratio 0.10) | 0.89 | 0.89 | 0.93 | 1.0 |
Full tables (both decks, all metrics): bench/RESULTS.md.
What I learned (including where I lose)
1. Fixed-interval capture has a hard ceiling nobody mentions. Its coverage capped at ~0.78–0.80 across every variant: any slide shorter than the capture interval is simply gone, no matter how clean the video is. Tutorials that recommend "grab a frame every 10 seconds" are silently accepting a ~20% slide loss on decks with normal pacing.
2. Naive pixel diff is better than its reputation — until something animates. At a sane per-pixel threshold (25), crf-45 compression noise stayed under the trigger. Where it degrades is the webcam overlay: a small pixel population sits permanently above threshold and precision drops to 0.85.
3. My own defaults flood on webcam overlays. This is the number I care most about publishing: with the default changedRatio: 0.02, the animated overlay occupies more than 2% of blocks, so every sampled frame re-triggers — 125 captures for 46 slides, precision 0.37. Raising the ratio to 0.10 restores precision to 1.0. Thresholds are inputs to report, not constants to trust. (The full product deals with this properly — per-video auto-calibration and masking the animated region — but the library's README now says this out loud instead of hoping.)
4. Neither detector hits recall 1.0 on clean input. A few consecutive MIT slides differ by roughly one bullet line; at 160×90 that change hovers at the trigger floor. Incremental builds are the honest hard case for any frame-differencing approach.
The takeaway
If you are building a video-to-slides pipeline in JavaScript:
- Don't use fixed-interval capture; the miss rate is structural.
- Frame differencing needs noise absorption (blocks, or a robust threshold) and a story for persistently-animated regions — masking, calibration, or a higher trigger ratio.
- Whatever you build, benchmark it against constructed ground truth. It costs two scripts and removes all labeling excuses: methodology.
The detector is ~80 lines, MIT-licensed, zero dependencies, runs in the browser and Node: github.com/larry-xue/video-slide-extractor. The finished product built on top of it (local processing, no upload) is video2any.com.
Top comments (0)