How to evaluate a slide-change detector without fooling yourself

I maintain video-slide-extractor,
a small JavaScript frame-differencing package used in a browser video-to-slides
pipeline. The detector is intentionally simple. Evaluating it honestly is not.

The easiest demo is to show a lecture clip, export a plausible deck, and say it
worked. That proves almost nothing.

A detector can produce a good-looking output while missing brief slides,
capturing half-faded transitions, or keeping the same slide five times. It can
also look inaccurate because the evaluator sampled too slowly to ever see the
missing slide.

Here is the protocol I now use.

First, define the task

Three jobs are often mixed together:

1. Slide-change detection: find timestamps where the visible frame changed.
2. Original-slide recovery: choose a clean representative image for every slide shown in the recording.
3. Deck generation: use a transcript or model to create a new presentation.

A frame-difference detector only addresses the first job. It should not get
credit for OCR or export quality, and it should not be blamed for a PowerPoint
writer stretching an image.

Build ground truth before tuning

Label the source video manually:

slide_id,start_seconds,end_seconds,notes
1,0.0,42.4,title slide
2,42.4,88.1,incremental bullet builds
3,88.1,123.7,crossfade transition

Do this before changing thresholds. Otherwise it is too easy to redefine
"correct" around the output you already have.

Record the source, license, duration, resolution, and access date. If the video
cannot be redistributed, publish the labels and source link instead.

Freeze the sampling configuration

The sampling interval is part of the detector.

If a slide is visible for 1.4 seconds and you sample every 2 seconds, the
detector cannot recover it. That is a sampling false negative, not a threshold
false negative.

For every run, record:

• sample interval;
• comparison resolution;
• crop or mask;
• detector options;
• runtime and package version.

For video-slide-extractor, that looks like:

const detect = createSlideDetector(160, 90, {
  blockSize: 8,
  blockDelta: 14,
  changedRatio: 0.02,
});

The defaults are starting points, not an accuracy guarantee.

Match detections once

Give each detected timestamp a tolerance window, such as two seconds, and match
it to at most one labeled transition.

• A matched detection is a true positive.
• An unmatched detection is a false positive.
• An unmatched label is a false negative.

Then calculate:

precision = TP / (TP + FP)
recall    = TP / (TP + FN)
F1        = 2 * precision * recall / (precision + recall)

This catches a common bad evaluation: counting two detections around the same
transition as two successes.

Score visual usability separately

A timestamp can be correct while the captured image is unusable.

Track at least:

• transition frames;
• consecutive duplicates;
• a slide revisited later;
• low-resolution or blurred captures;
• frames dominated by a presenter or webcam overlay.

I report duplicate rate and transition-frame rate separately from detection F1.
Those failures often belong to temporal cleanup and export-frame selection, not
the first-pass change detector.

Tag the failure class

The aggregate number tells you whether a version improved. The failure class
tells you what to build next.

Useful tags include:

• incremental build;
• crossfade or wipe;
• cursor/webcam motion;
• camera footage around a screen;
• compression noise;
• repeated earlier slide;
• sample interval too wide.

For example, raising changedRatio can reduce cursor-related false positives
while making small text-only changes easier to miss. A single "accuracy" number
hides that tradeoff.

Compare equal pipelines

When comparing two libraries, use the same decoded frames and sampling interval.
When comparing two finished products, report the entire pipeline instead:

• what input each accepted;
• whether processing was local or remote;
• how much manual cleanup was needed;
• whether the final PPTX contained images, OCR text, or native objects.

Do not rank a transcript-to-new-deck generator against an original-slide
recovery tool as if they produced the same result.

Publish the failures

The minimum reproducible evidence is:

• ground-truth labels;
• detector configuration;
• detected timestamps;
• metric calculation;
• package/runtime versions;
• known blockers and excluded cases.

I added a reusable version of this protocol to the repository:

Evaluate a Slide-Change Detector

The detector itself is zero-dependency JavaScript and accepts ordered RGBA
frames in the browser or Node:

npm install video-slide-extractor

The larger lesson is simple: show the labels, the settings, and the ugly frames.
If a comparison publishes only the winners, it is marketing, not a benchmark.

Comments

[Wren Calloway]

The tolerance-window matching hides a failure mode worth calling out: incremental bullet builds break the one-to-one assumption in a way a crossfade doesn't. A slide that reveals bullets over 40 seconds is arguably one "slide" in your ground truth but N legitimate visible-frame changes to a difference detector. Whether those extra detections are false positives or true positives depends entirely on whether the downstream job wants the final composed slide or every intermediate state — and that's a labeling decision, not a detector property.

Which means your CSV schema quietly encodes a policy. If slide 2 is one row spanning 42.4–88.1 with "incremental bullet builds" in notes, you've decided builds are noise, and every build detection scores as FP no matter how well the detector performed at its literal task. I'd add a column that marks whether a labeled span is expected to be single-capture or multi-capture, so the F1 number reflects the task you chose rather than an implicit one. Otherwise two evaluators using the same protocol on the same video can honestly report different recall purely from how they collapsed builds — and neither will know that's why they disagree.

[larry]

This is a fair hit, and I think the right fix is one step stronger than a per-row flag.

The problem with a single-capture / multi-capture column is that it still bakes one collapse decision into the labels — it just makes the decision visible. If the downstream job changes (final composed slide vs. every build state), you're relabeling the same video.

What I'd rather do is label at the finest granularity the video actually has: every visible-frame change gets its own row, typed.

event_id,seconds,type,parent_slide
1,0.0,slide,1
2,42.4,slide,2
3,55.0,build,2
4,63.2,build,2
5,71.8,build,2
6,88.1,slide,3

Then the collapse policy becomes a scoring parameter instead of a labeling decision. count_builds=events scores all six rows as expected detections. count_builds=collapse merges build rows into their parent slide — and here your point splits further into two sub-policies worth naming: builds as ignorable (detections inside the span are neither TP nor FP) versus builds as errors (a final-composed-slide exporter genuinely wants the detector quiet during builds, because every build detection becomes a near-duplicate to clean up). A capture-for-OCR pipeline probably wants ignorable; a one-image-per-slide deck exporter wants errors.

The concrete win over encoding policy in the labels is exactly your two-evaluators scenario: they share one label file, and if their recall numbers differ, the difference lives in a declared, reportable scoring flag — not in how each of them silently collapsed the CSV.

I'll update the evaluation doc in the repo to split the schema this way. Thanks — the notes field was doing hand-waving work that belonged in the schema.

[Frank]

This is super helpful! I'm curious if