How to find highlight clips in a transcript with a sliding window in Python

Once a transcript has word timestamps, finding candidate short clips in a long recording does not need a model. A window of fixed length, started at every word and scored by how many words fit inside it, finds the stretches where the most is said. The video editor in the osFoundry desktop app ships exactly this as its Suggest clips button, in Rust (find_highlights in canvas_transcript.rs). Below it is rebuilt in Python and run on a real 5:07 interview, with measurements of what each rule does, how window length changes the answer, when the best-first selection stops being optimal, and how to make the scoring pass linear.

The input

The recording is a NASA interview with astronaut Christina Koch. Its Whisper transcript has 1,007 words over 306.97 seconds, 3.28 words a second on average. Whisper returns words without punctuation in this mode, and the only fields the algorithm needs are the times:

{"words": [
  {"index": 0, "text": "And",   "start_ms": 0,   "end_ms": 160},
  {"index": 1, "text": "we'll", "start_ms": 160, "end_ms": 280},
  {"index": 2, "text": "get",   "start_ms": 280, "end_ms": 380},
  ...
], "duration_ms": 306970}

One window per word

Every word starts a candidate window. A word counts if it ends before the window does, and the scan stops at the first word that doesn't:

def candidates(words, window_ms):
    out = []
    for i, w in enumerate(words):
        start = w["start_ms"]
        end = start + window_ms
        n = 0
        for x in words[i:]:
            if x["end_ms"] > end:
                break
            n += 1
        if n:
            out.append((start, end, n))
    return out

Starting windows on words rather than on a fixed grid (every second, say) means a candidate always opens at the beginning of a word, never in the middle of one. The score divides by the window length, not by the time actually spoken, so the 44 windows that start in the last 15 seconds of the clip score low on their own, without a special case.

For this interview that gives 1,007 candidates. The median holds 49 words, the best 67.

Best first, no overlaps

The candidates are sorted by score, ties broken by position so the shortlist comes out the same on every run, and taken from the top while they don't overlap anything already chosen:

def find_highlights(words, window_ms=15_000, count=3):
    ranked = sorted(candidates(words, window_ms), key=lambda c: (-c[2], c[0]))
    chosen = []
    for c in ranked:
        if len(chosen) >= count:
            break
        if any(c[0] < k[1] and k[0] < c[1] for k in chosen):
            continue  # overlaps a better window
        chosen.append(c)
    return sorted(chosen)

With the app's settings, 15 seconds and three clips, the Python version returns the same three ranges the app shows:

clip	words	words/s
0:45.1 – 1:00.1	64	4.27
2:03.4 – 2:18.4	67	4.47
2:31.5 – 2:46.5	65	4.33

The overlap rule does most of the work. Without it the top three are 2:03.4, 2:03.6 and 2:00.2, the same passage three times, all within 3.4 seconds of each other. Dense speech is dense for a while, so the best windows cluster: the top 19 candidates all start between 1:59.7 and 2:04.4. The second and third picks are candidates number 20 and 21 in the ranking, and 18 better-scored windows were skipped to reach them. The two tied at 67 (2:03.4 and 2:03.6) are why the tie-break on position matters.

Window length changes the answer

The same transcript with other window lengths:

window	picks (start, words/s)
10 s	0:51.2 (4.5), 2:08.5 (4.7), 3:15.3 (4.5)
15 s	0:45.1 (4.27), 2:03.4 (4.47), 2:31.5 (4.33)
30 s	0:25.1 (3.8), 1:48.5 (3.93), 2:31.5 (3.93)
60 s	0:00.0 (3.57), 1:23.7 (3.48), 2:31.0 (3.77)

Short windows reward a single fast burst and produce higher peaks. Long windows average over pauses and questions and drift toward the overall rate of 3.28. The stretch around 2:31 survives at every length from 15 to 60 seconds, which is a decent sign that it is a sustained answer and not one quick sentence. Running two lengths and looking for the overlap is a cheap way to rank the candidates by more than one number.

Is best-first optimal?

Taking the best window first is a greedy choice. To check it, the exact answer is a small dynamic program over the candidates sorted by start time: for each candidate, either skip it or take it and jump to the first candidate that starts after it ends.

import bisect, functools

def best_total(words, window_ms, count):
    cands = sorted(candidates(words, window_ms))
    starts = [c[0] for c in cands]
    nxt = [bisect.bisect_left(starts, c[1]) for c in cands]

    @functools.lru_cache(None)
    def best(i, k):
        if k == 0 or i >= len(cands):
            return 0
        return max(best(i + 1, k), cands[i][2] + best(nxt[i], k - 1))

    return best(0, count)

Total words in the chosen windows, greedy against exact:

window × clips	best-first	exact
15 s × 3	196 words, 3 clips	196
15 s × 10	595 words, 10 clips	595
30 s × 5	563 words, 5 clips	565
30 s × 10	764 words, 8 clips	989
60 s × 5	849 words, 5 clips	991

For a short shortlist of short windows, greedy loses nothing here: at 15 seconds it matched the exact total for 3, 5 and 10 clips. It falls apart when the windows start to fill the recording. Ten 30-second windows fit in 307 seconds, but best-first places its first picks wherever the peaks are, the gaps left between them are shorter than 30 seconds, and it returns 8 clips where the exact answer fits 10. The same thing costs 142 words at 60 s × 5. So the greedy version is the right tool for "give me three candidates", and the wrong one for "split this talk into chapters", where the DP is only a few lines more.

Make the scoring pass linear

The rescan in candidates reads every word in every window, so its cost grows with the window length times the number of words. Because both start and end times only move forward, the window's last word only moves forward too, and one pointer is enough:

def candidates_two_pointer(words, window_ms):
    out, j = [], 0
    for i, w in enumerate(words):
        j = max(j, i)
        end = w["start_ms"] + window_ms
        while j < len(words) and words[j]["end_ms"] <= end:
            j += 1
        if j > i:
            out.append((w["start_ms"], end, j - i))
    return out

Both versions return identical candidates on this transcript. Timed with Python 3.9 on an Apple M1, median of 7 runs, scoring only, with the 1-hour case made by repeating the interview 12 times:

transcript	words	rescan	two-pointer
5 min	1,007	2.9 ms	0.4 ms
1 h	12,084	190.5 ms	5.4 ms

The app's Rust loop is the rescan version. In Python, on a long podcast or with several window lengths at once, the two-pointer version is the one to keep. It does rely on the end times being in order, which held for every word here but is worth asserting on transcripts merged from several sources.

What the score does not know

Words per second measures how much is said, not whether it is worth hearing. The comment above find_highlights in the app's source calls it a heuristic and draws the line deliberately: a model that picks highlights would be a different feature, one that costs money and needs a key, and quietly upgrading this one would turn a free operation into a charge the user never approved. The panel in the app says the same to the user, "a starting point, not a verdict". In practice that means the three windows are where to start listening, and the cut points are still set by ear.

The full script, with every measurement above, is about 120 lines of standard-library Python. In the osFoundry desktop app the same function sits behind the Suggest clips button in the video editor's transcript panel, next to a transcription step that runs on your own cloud API key. The app is free from the osFoundry download page.