DEV Community

Cover image for PySceneDetect Finds Scene Cuts. It Does Not Find Highlights.
Jacob Yang
Jacob Yang

Posted on

PySceneDetect Finds Scene Cuts. It Does Not Find Highlights.

A hard cut is a discontinuity between adjacent frames. A clip-worthy moment is a judgment about payoff, reaction, and whether a new viewer can follow what just happened. Those two events can land on the same timestamp, and often they do not.

I wanted a number for that gap, not a vibe. I did not use a Twitch VOD and I did not use footage from a commercial game. Riot art, publisher HUDs, and other people's broadcasts are not mine to republish as fixtures. Instead I rendered 24 seconds of original geometric video, planted the cuts and the "highlight-like" events myself, and ran PySceneDetect 0.7.3 against those labels. The scores below are cut detection versus planted labels on synthetic footage. They are not highlight accuracy on real gameplay.

What PySceneDetect says it measures

The project's current homepage, fetched 22 November 2026, describes PySceneDetect as a tool for detecting shot changes and splitting a video into clips. The stable release listed that day was v0.7.3 (8 November 2026), which matched the installed package. The official detector reference is clear about the job. ContentDetector "compares the difference in content between adjacent frames against a set threshold/score, which if exceeded, triggers a scene cut." The difference is calculated in HSV, and the Python default threshold is 27.0. AdaptiveDetector is similar, "except the threshold isn't fixed, but is a rolling average of adjacent frame changes," which "can help mitigate false detections in situations such as fast camera motions." ThresholdDetector "detects slow transitions using average pixel intensity in RGB (fade in/fade out)."

The same page notes a default that matters if you switch between the CLI and Python: min_scene_len defaults to 15 frames in Python and to 0.6 seconds on the CLI. I used the Python API and left every detector at library defaults, including ContentDetector(threshold=27.0) and min_scene_len=15. I did not tune after seeing the labels.

PySceneDetect is BSD 3-Clause software by Brandon Castellano. The fixture I built is original and is released under CC0 1.0 in the experiment folder. This article is not affiliated with the project.

A fixture with planted cuts and planted "highlights"

The clip is 1280×720 at 30 fps. Five hard cuts sit at 2.0, 5.0, 9.0, 14.0, and 18.0 seconds, each a new background color and a new geometric subject. A one-second fade to black starts at 21.0 seconds and holds black until 24.0. Inside scenes that already exist, I planted three highlight-like events that are not cuts:

  • a four-frame white flash and a yellow rectangle at 7.5 seconds
  • an 880 Hz beep at 11.5 seconds, 0.30 seconds long, over a quiet noise bed
  • a short horizontal pan at 16.0 seconds

The bouncing disk in the green scene is motion without a cut. If a detector fires there, that is a false scene cut, not a success.

A second, deliberately crude highlight heuristic looks for peaks in mean absolute frame difference and in per-frame audio RMS. It is a teaching baseline, not a highlight model. The question is whether cut detectors recover the highlight events, and whether the peak heuristic can be mistaken for a cut detector.

Measured cuts

I compared detections to planted hard cuts with a 5-frame tolerance, 0.1667 seconds at 30 fps. Fade scoring used a 0.5-second window around fade start. Highlight scoring used 0.25 seconds.

Method Against Predicted times (s) P R F1
ContentDetector, defaults 5 hard cuts 2.00, 7.50, 9.00, 14.00, 18.00, 22.00 0.667 0.800 0.727
AdaptiveDetector, defaults 5 hard cuts 2.00, 5.00, 7.50, 9.00, 14.00, 18.00, 22.00 0.714 1.000 0.833
ThresholdDetector, defaults fade at 21.0 none 0.000 0.000 0.000
Highlight heuristic 3 highlight-like events 10 peaks, including all 3 0.300 1.000 0.462
ContentDetector 3 highlight-like events 1 of 3 (the flash) 0.167 0.333 0.222
Highlight heuristic 5 hard cuts all 5, plus 5 extras 0.500 1.000 0.667

Three stacked panels for the 24-second synthetic fixture: mean absolute frame difference, audio RMS, and event marks. Planted hard cuts, a fade band, highlight-like events, and ContentDetector cut times are drawn on the bottom panel.

Figure 1. Motion and audio on the synthetic fixture, with planted labels and ContentDetector cuts. The 5.0-second hard cut is a small motion spike and was missed at the default threshold. The 7.5-second flash is a large spike and was reported as a cut. The beep at 11.5 seconds is obvious in audio and invisible to the cut detector.

AdaptiveDetector found all five hard cuts. ContentDetector missed the cut at 5.0 seconds. Looking at the motion trace, that transition is a milder color change than the others; at threshold 27.0 it did not qualify. I left the threshold alone. Default parameters are the measurement, not a prelude to a better-looking number.

Both content and adaptive detectors also fired at 7.5 seconds, which is the flash, and at 22.0 seconds, which is the first fully black frame after the fade. Those are false positives against the hard-cut list. The flash is a useful misspecification: a highlight-like event that looks like a cut to an HSV difference score. The 22.0-second mark is the end of the fade, not the 21.0-second start.

ThresholdDetector at its Python defaults (threshold=12 on average RGB, add_final_scene=False, min_scene_len=15) reported no cuts at all on this fade-to-black-and-hold. The official docs note that the CLI detect-threshold command adds a final scene by default and that the Python class does not. I did not change add_final_scene after the fact. A fade that a human can see is not guaranteed to become a cut list under the defaults you actually called.

Six stills taken immediately after each ContentDetector cut: a cyan rectangle, the washed-out flash with a yellow box, a yellow triangle, a red disk on a grid, two colored squares, and a fully black frame.

Figure 2. Frames immediately after the ContentDetector cuts. The 7.50-second still is the planted flash, not a new scene. The 22.00-second still is the hold on black after the fade. There is no 5.00-second still because that hard cut was not detected.

The Python call was the official one:

from scenedetect import ContentDetector, AdaptiveDetector, ThresholdDetector
from scenedetect import open_video
from scenedetect.scene_manager import SceneManager

def detect_cuts(video_path, detector):
    video = open_video(str(video_path))
    manager = SceneManager()
    manager.add_detector(detector)
    manager.detect_scenes(video)
    scenes = manager.get_scene_list()
    return [round(s[0].seconds, 4) for s in scenes[1:]]

content_cuts = detect_cuts(path, ContentDetector())
adaptive_cuts = detect_cuts(path, AdaptiveDetector())
threshold_cuts = detect_cuts(path, ThresholdDetector())
Enter fullscreen mode Exit fullscreen mode

Cuts are the wrong label for highlights

The highlight heuristic recovered all three planted highlight-like events: the flash near 7.63 seconds, the beep at 11.7 seconds, and the pan at 16.03 seconds. It also fired on every hard cut, on a bounce peak at 6.07 seconds, and near the end of the fade. Precision against the highlight labels was 0.300. Recall was 1.000. That is what an unfiltered peak detector does. It is sensitive, and it cannot tell a scene change from a flash from a camera pan.

ContentDetector recovered only the flash among the three highlight-like events. It missed the beep, which has almost no visual change, and it missed the pan, which is visible in the motion trace as a short plateau rather than a single spike above the default cut threshold. Against the highlight labels, F1 was 0.222. That is the result I wanted measured, not improved: a scene-cut detector is not a highlight detector, even when one of your highlights happens to look like a cut.

The inverse is also true. The peak heuristic found all five hard cuts, so you could pretend it is a cut detector with F1 0.667, worse than AdaptiveDetector's 0.833 on this fixture, and with no ability to say which peaks are edits.

If you are turning a long recording into short posts, both mistakes are expensive. A cut list will split on flashes and miss quiet punchlines. A raw motion-and-audio peak list will hand you a pile of scene changes and loading-screen pops along with the moments you actually wanted.

A shortlist still needs an editor

That gap is the practical one in a VOD workflow. FragCut, an AI gaming clip maker, starts from a saved broadcast and builds a shortlist of moments where something changes, and the pyscenedetect measurements above are about cut points, not about which of those moments is worth posting.

The product page, fetched 22 November 2026 and last updated 18 November 2026, says to pull candidates from places where something actually changes: chat spikes, round wins, deaths, jokes, raids, boss attempts, or scoreboard swings. It also says AI can scan a long VOD and suggest candidates, but you still decide what is worth posting, and you still review each one like an editor. For a three-hour stream it suggests aiming for 10 to 20 candidates rather than 60. I did not run FragCut on a VOD, and I am not going to treat the synthetic F1 scores as a product benchmark.

The same distinction showed up in a different form when I measured a synthetic kill-banner detector with OpenCV. That article was about a HUD event, which is closer to a highlight cue than to a shot change. Neither a banner match nor a scene cut is a complete definition of a publishable clip.

Limits

The fixture is original geometric video under CC0. It is not a substitute for a labeled game dataset, and the highlight-like events are flashes, beeps, and pans I drew, not clutches or joke timings. I used PySceneDetect 0.7.3 at documented Python defaults; a different min_scene_len, threshold, or CLI default would change the list. ThresholdDetector may behave differently if you set add_final_scene=True, which I did not do in this run. The highlight heuristic's peak thresholds are my choices, labeled as such, and they should not be read as part of PySceneDetect.

The code, fixture generator, labels, licenses, and results.json are in devto-fragcut-pyscenedetect-code/. Run python -m pytest test_matching.py -q and python run_experiment.py to regenerate the video and the figures.

Top comments (0)