Auto-highlights with a small VLM or how the model votes yes or no on every 10-second cut

You point a tool at a 40-minute gameplay video and ask for a one-minute highlight reel. What should it actually do?

The naive answer is "find the exciting parts." But a model has no idea what exciting means for your video until you tell it. A goal in football, a punchline in a podcast, and a jump scare in a let's-play are all highlights, and none of them look alike. So the real problem is two problems stacked on top of each other. First, figure out what counts as a highlight for this specific video. Then go through the video and decide, piece by piece, whether each piece is one of those.

That is exactly the shape of the code in portable/src/visual_generation/highlights/processing.py in Wunjo Make. It uses a small vision-language model called SmolVLM2 and, for the audio path, OpenAI's Whisper. What I like about it is that the "scoring" is not a regression head or a learned ranker. It is the model answering the word yes. Let me walk through how that works and where the sharp edges are.

The whole pipeline in one breath

Here is the visual path, top to bottom, from generate_visual_highlight_moments:

duration = self.get_video_duration_seconds(video_path)
description = self.process_video_analyser(video_path)
highlight_response = self.process_highlight(description, highlight_description)
self.cut_scenes(video_path, save_dir, segment_length)   # 10s chunks via ffmpeg
# ...shuffle segment indices, chunk them...
if self.process_video_segment(segment_path, highlight_response):
    kept_segments.append((start_time, end_time))

Four model calls, in order:

1. Describe the video. Ask the VLM what kind of video this is.
2. Invent the highlight menu. Feed that description back and ask for a list of dramatic moments that would make good highlights for this type of video.
3. Cut into segments. Use ffmpeg to split the file into 10-second clips. No model here.
4. Vote per segment. For each clip, show it to the VLM with the menu and ask: does this clip contain one of these? Keep the ones where the answer is yes.

The model wrapper, and the one line that parses the answer

class SmolVLM2:
    def __init__(self, model_path: str = "HuggingFaceTB/SmolVLM2-2.2B-Instruct", device: str = "cuda", download_root: str = None):
        self.processor = AutoProcessor.from_pretrained(model_path)
        self.model = AutoModelForImageTextToText.from_pretrained(
            model_path,
            torch_dtype=torch.bfloat16,
            device_map=self.device,
            _attn_implementation="flash_attention_2" if FLASH_ATTN_2_AVAILABLE else None
        )

SmolVLM2 is a small multimodal model, the 2.2B-Instruct checkpoint here. Small matters: this runs per segment in a loop, so a 70B model would be unusable on a desktop. It loads in bfloat16 and turns on FlashAttention 2 only if the import succeeded at the top of the file, otherwise it passes None and lets transformers fall back.

The generate method is where the interface to the rest of the pipeline lives:

def generate(self, messages: list, max_new_tokens: int = 64):
    inputs = self.processor.apply_chat_template(
        messages[:8192],
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(self.model.device, dtype=torch.bfloat16)
    generated_ids = self.model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.7)
    generated_output = self.processor.decode(generated_ids[0], skip_special_tokens=True).lower().split("assistant: ")[1]
    return generated_output

Three things to notice here, because they all affect the highlight decision.

do_sample=True, temperature=0.7. The voting is not deterministic. The same clip with the same prompt can come back yes once and no the next run. For a highlight reel that is fine, even pleasant. You get variety. For a unit test it is a headache.

max_new_tokens=64 for the per-segment call. The model is not asked to write an essay, it is asked for a quick verdict. Sixty-four tokens is enough for "yes" plus a one-line justification and nothing more.

.lower().split("assistant: ")[1]. This is the cheap, fragile glue that turns chat output into a usable string. It lowercases everything (so the later "yes" in response check is case-insensitive) and keeps only the text after the literal assistant: marker. If the decoded output ever does not contain that exact marker, this line raises an IndexError. Worth knowing if you swap the checkpoint.

The model writes its own rubric

This is my favorite part. Before scoring anything, the code asks the model to enumerate what a highlight even is for this video. Look at the system messages:

SYSTEM_MESSAGES = {
    "highlight_editor": "You are a highlight editor. List archetypal dramatic moments that would make compelling highlights if they appear in the video. Each moment should be specific enough to be recognizable but generic enough to potentially exist in other videos of this type.",
    "highlight_assistant": "You are a helpful visual-language assistant ... Highlights should be rare and important events in the video in question.",
    "highlight_analyzer": "You are a video highlight analyzer. Your role is to identify moments that have high dramatic value ... Be categorical and choose only the brightest moments. ..."
}

And the prompt that builds the menu:

@staticmethod
def prompt_highlight(description: str, highlight_description: str = None, behaviour: int = None):
    prompts = {
        1: "List potential highlight moments to look for in this video:",
        2: "List dramatic moments that would make compelling highlights if they appear in the video. ...:"
    }
    behaviour = 1 if highlight_description is None and behaviour is None else 2
    highlight_description = prompts[behaviour] if highlight_description is None else highlight_description
    return [
        {"role": "system", "content": [{"type": "text", "text": SYSTEM_MESSAGES["highlight_editor" if behaviour == 1 else "highlight_assistant"]}]},
        {"role": "user", "content": [{"type": "text", "text": f"""Here is a description of a video:\n\n{description}\n\n{highlight_description}"""}]}
    ]

The behaviour switch decides the personality. If the caller passed nothing, behaviour is 1 and the model plays "highlight editor", listing archetypal moments. If the caller passed their own highlight_description (say, "show me every time someone laughs"), behaviour becomes 2 and the system role switches to "highlight assistant". The wording in those two system prompts is doing real work: "specific enough to be recognizable but generic enough to potentially exist in other videos of this type" is what keeps the menu from being too tied to one exact frame.

There is one more wrinkle in process_highlight:

def process_highlight(self, description: str, highlight_description: str = None) -> str:
    if highlight_description is not None:
        highlight_description = f"List potential {highlight_description.lower()} to look for in this video:"
    highlight_response_messages = self.prompt_highlight(description, highlight_description)
    highlight_response = self.generate(highlight_response_messages, max_new_tokens=256)
    return highlight_response

When you give your own description, it is wrapped into "List potential {your text} to look for in this video:". So a user request like "funny reactions" becomes "List potential funny reactions to look for in this video:". The output of this call, up to 256 tokens, is the rubric. It is plain text. It gets pasted verbatim into every per-segment prompt later.

The yes/no vote, and why scoring is just "yes" in response

Here is the prompt that scores one clip:

@staticmethod
def prompt_video_segment(video_segment_path: str, highlight_types: str):
    return [
        {"role": "system", "content": [{"type": "text", "text": SYSTEM_MESSAGES["highlight_analyzer"]}]},
        {"role": "user", "content": [
            {"type": "video", "path": video_segment_path},
            {"type": "text", "text": f"""Given these highlight examples:\n{highlight_types}\n\nDoes this video contain a moment that matches the core action of one of the highlights? Answer with:\n'yes' or 'no'\nIf yes, justify it"""}]}
    ]

And the scoring:

def process_video_segment(self, video_segment_path: str, highlight_types: str) -> bool:
    messages = self.prompt_video_segment(video_segment_path, highlight_types)
    response = self.generate(messages, max_new_tokens=64)
    print(f"Segment response {response}")
    return "yes" in response

That is the entire scorer. No score between 0 and 1, no ranking, no threshold you tune. The clip is shown to the model alongside the rubric, the model is told to answer yes or no and justify a yes, and the code does a substring check for yes on the lowercased text. It is a vote, not a rank.

This is a real design choice with real consequences. The upside: it is dead simple and it leans on the model's own judgment, which is the whole point of using a VLM. The downside is the substring check. The word "yes" appearing anywhere in 64 tokens flips the clip to "keep". A justification like "there is no clear yes-or-no moment here" contains yes and would be kept. In practice the instruction to lead with yes or no makes this rare, but it is the failure mode to watch if you fork this.

Stopping early: shuffle, chunk, budget

You do not want to score all 240 clips of a 40-minute video. You want roughly one minute of highlights and then stop. Here is how the loop manages that:

duration_limit_sec = duration_limit_min * 60
duration_highlight = 0

indices = list(range(len(segments_path)))
random.shuffle(indices)

chunk_size = 10
chunks = [indices[i:i + chunk_size] for i in range(0, len(indices), chunk_size)]

for num, chunk in enumerate(chunks):
    for i in chunk:
        segment_path = segments_path[i]
        start_time = float(i * segment_length)
        end_time = min(float(i * segment_length + segment_length), duration)
        if duration_highlight + segment_length > duration_limit_sec:
            break
        if self.process_video_segment(segment_path, highlight_response):
            kept_segments.append((start_time, end_time))
            duration_highlight += segment_length
    else:
        continue   # limit not reached, process next chunk
    break          # limit reached, stop

Three deliberate moves here.

The indices are shuffled before scoring. If you scanned in order and your budget filled up after the first six clips, your highlight reel would be the first sixty seconds of the video, every time. Shuffling spreads the sampled clips across the whole timeline, so a one-minute reel can pull a moment from the start, the middle, and the end.

The duration_highlight counter is the budget. Every kept clip adds segment_length (ten seconds). The check if duration_highlight + segment_length > duration_limit_sec: break stops adding once the next clip would overflow the requested length. So duration_limit_min=1 gives you up to six ten-second clips.

The for/else/break is the early exit. This is one of Python's least-used features. The else on a for loop runs only if the loop finished without hitting break. So: if a chunk completes without filling the budget, else: continue moves to the next chunk. If the budget filled mid-chunk and we breaked out of the inner loop, the else is skipped, and the outer break ends everything. Chunking by ten just bounds how often the budget is checked.

One last line that matters:

kept_segments.sort(key=lambda x: x[0])

We scored in shuffled order, so the kept clips are out of order. This sorts them back by start time, so the reel plays chronologically. Without it your highlights would jump around in time.

The audio twin: Whisper instead of frames

generate_text_highlight_moments is the same machine with a different sensor. Instead of looking at frames, it reads what was said. The transcription comes from the Whisper class:

class Whisper:
    def __init__(self, download_root: str, model_id: str = "large-v3-turbo", device: str = "cuda"):
        self.model = whisper.load_model(model_id, device=device, download_root=download_root)

    def __call__(self, audio_path, language=None, task="transcribe"):
        audio = self.load_audio(audio_path)
        output = whisper.transcribe(self.model, audio, detect_disfluencies=True, vad=True, language=language, task=task)
        return output

It uses large-v3-turbo with vad=True (voice activity detection, so silence is dropped) and detect_disfluencies=True (it marks ums and hesitations). The transcript gets grouped into roughly 30-second blocks by get_timestamp, each block carrying start, end, and text. Those blocks are the segments now. The scorer is process_text_segment, which asks the same yes/no question about a chunk of transcript text, and the same "yes" in response vote decides it. Same shuffle, same budget, same early exit, same final sort.

There is also a small cleanup helper worth a mention, because small VLMs love to repeat themselves:

@staticmethod
def strip_repeats(text, min_words=2, max_words=6):
    pattern = re.compile(r'\b((?:\w+\s*){' + str(min_words) + ',' + str(max_words) + r'})\1+', flags=re.IGNORECASE)
    return pattern.sub(r'\1', text)

It collapses a phrase of 2 to 6 words that repeats back-to-back into a single copy. The text path runs the description through this before re-summarizing. It is a band-aid for a known habit of small models, and it is honest about being one.

---

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

---

A highlight reel is just the clips that survived the vote, stitched back in time order. Photo: Unsplash

Stitching the survivors

Once you have kept_segments, the cut is plain ffmpeg. concatenate_scenes builds a filter_complex that trims and concatenates each kept range:

for i, (start_sec, end_sec) in enumerate(scene_times):
    filter_complex_parts.append(f"[0:v]trim=start={start_sec}:end={end_sec},setpts=PTS-STARTPTS[v{i}];")
    filter_complex_parts.append(f"[0:a]atrim=start={start_sec}:end={end_sec},asetpts=PTS-STARTPTS[a{i}];")
    concat_inputs.append(f"[v{i}][a{i}]")

concat_filter = f"{''.join(concat_inputs)}concat=n={len(scene_times)}:v=1:a=1[outv][outa]"

setpts=PTS-STARTPTS resets the timestamps on each trimmed piece so they line up at zero before concatenation. Audio gets the same treatment with asetpts. The output is re-encoded to H.264 with AAC audio. There is a separate concatenate_audio for audio-only output. Both honor a DEBUG env var: when DEBUG=True the command runs through os.system so you see ffmpeg's output, otherwise it runs silently through subprocess.run.

Gotchas if you run this yourself

A few things that will bite you, all visible in the code:

• Sampling makes it non-reproducible. do_sample=True, temperature=0.7 means two runs on the same video give different reels. Set do_sample=False in generate if you need repeatability while debugging.
• The "yes" in response substring is greedy. Any clip whose 64-token answer contains the letters yes is kept. If you change the checkpoint or the prompt, check that the model still leads with a clean yes/no.
• The split("assistant: ")[1] will throw on the wrong format. A different chat template that does not emit that exact lowercase marker breaks generate with an IndexError. This is the first thing to check when swapping models.
• cut_scenes re-reads the directory with os.listdir. Segments are matched by index against segments_path, and start_time is computed as i * segment_length. That assumes ffmpeg produced evenly sized chunks in sorted order. The last chunk is shorter, which is why end_time is clamped with min(..., duration).
• The budget is coarse. It counts in whole segment_length units (10s), not in actual kept seconds, so a one-minute request gives "up to six clips", not "exactly sixty seconds".

That is the whole selection engine. No trained ranker, no score threshold, just a small model that writes its own rubric, then votes yes or no on each ten-second slice while a budget and a shuffle keep the reel short and spread out. The simplicity is the feature. You can read every decision the system makes, because every decision is a word the model said.

References

• Marafioti et al. "SmolVLM: Redefining Small and Efficient Multimodal Models." 2025. arXiv:2504.05299. https://arxiv.org/abs/2504.05299
• Radford, Kim, Xu, Brockman, McLeavey, Sutskever. "Robust Speech Recognition via Large-Scale Weak Supervision." 2022. arXiv:2212.04356 (Whisper). https://arxiv.org/abs/2212.04356
• Source file: portable/src/visual_generation/highlights/processing.py in Wunjo Make.