Auto-generate video chapters with Whisper + an LLM (without hallucinated timestamps)

TL;DR

We'll auto-generate chapters for a video. The trick: transcribe with timestamps, let the LLM choose chapter boundaries by segment index (never raw time), then map indices back to real timestamps in code. That sidesteps the one thing LLMs are reliably bad at: echoing exact timecodes. Output is a chapters.vtt file plus YouTube-style stamps.

📦 Code: github.com/USER/llm-video-chapters (replace before publishing)

Pasting a transcript into a chat model and asking for "chapters with timestamps" produces great titles and fabricated times. LLMs don't preserve exact timestamps across long inputs, and long-context models also skip the middle. The fix is to stop asking the model to be a clock. Let's build the pipeline that does it right.

1. Transcribe with real timestamps

Use WhisperX (or faster-whisper) for aligned segment timestamps. Each segment has a start, end, and text.

pip install whisperx

# transcribe.py
import whisperx, json

device = "cpu"  # or "cuda"
model = whisperx.load_model("large-v3", device, compute_type="int8")

audio = whisperx.load_audio("talk.mp4")
result = model.transcribe(audio, batch_size=8)

# result["segments"] -> [{"start": 0.0, "end": 4.12, "text": "..."}, ...]
with open("segments.json", "w") as f:
    json.dump(result["segments"], f, indent=2)

print(f"{len(result['segments'])} segments")

python transcribe.py
# 214 segments

💡 Tip: FFmpeg 8.0's native whisper filter can also emit timestamped JSON if you'd rather not add a Python dependency. Either way, the shape you need is a list of segments with start times.

2. Number the segments, keep times in a lookup

The model is about to see indexed text. It is not going to see the raw timestamps. We keep those in a table keyed by index.

# build_index.py
import json

segments = json.load(open("segments.json"))

# Lookup table: index -> start time (seconds)
start_times = {i: seg["start"] for i, seg in enumerate(segments)}

# Numbered transcript the model will read
numbered = "\n".join(f"[{i}] {seg['text'].strip()}" for i, seg in enumerate(segments))

open("numbered.txt", "w").write(numbered)

numbered.txt looks like:

[0] Welcome everyone, thanks for joining.
[1] Today we're going to talk about database indexing.
[2] But first, a quick recap of last week.
...
[24] Okay, let's actually set up the schema.

3. Ask the LLM for boundaries by index

This is where correctness lives. The prompt asks for JSON with a start_segment (an integer) and a title. The model never touches a timestamp.

# chapter.py
import json, os
from openai import OpenAI  # any chat LLM works; this is just an example client

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
numbered = open("numbered.txt").read()
segments = json.load(open("segments.json"))
duration_min = segments[-1]["end"] / 60

# Scale chapter count to length so short videos aren't over-segmented
target = "3-5" if duration_min < 15 else "8-14"

prompt = f"""You are segmenting a video transcript into chapters.
The transcript is a numbered list of segments: [index] text.

Return ONLY JSON: a list of objects with "start_segment" (the index where a
new chapter begins) and "title" (a short, specific 2-6 word label).
Rules:
- The first chapter MUST start at segment 0.
- Choose boundaries where the topic clearly changes.
- Aim for {target} chapters for this {duration_min:.0f}-minute video.
- Do NOT include timestamps. Only segment indices.

Transcript:
{numbered}
"""

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    response_format={"type": "json_object"},
    temperature=0.2,
)

chapters = json.loads(resp.choices[0].message.content)["chapters"]
json.dump(chapters, open("chapters_raw.json", "w"), indent=2)
print(chapters)

python chapter.py
# [{'start_segment': 0, 'title': 'Introduction'},
#  {'start_segment': 24, 'title': 'Setting up the schema'},
#  {'start_segment': 58, 'title': 'B-tree vs hash indexes'}, ...]

4. Reattach real timestamps in code

The model gave us where (indices). Code supplies when (real times from Whisper). This step cannot hallucinate.

# finalize.py
import json

segments = json.load(open("segments.json"))
chapters = json.load(open("chapters_raw.json"))
start_times = {i: seg["start"] for i, seg in enumerate(segments)}
video_end = segments[-1]["end"]

# Attach start/end times; each chapter ends where the next begins
final = []
for n, ch in enumerate(chapters):
    start = start_times[ch["start_segment"]]
    end = (start_times[chapters[n + 1]["start_segment"]]
           if n + 1 < len(chapters) else video_end)
    final.append({"start": start, "end": end, "title": ch["title"]})

json.dump(final, open("chapters.json", "w"), indent=2)

5. Emit WebVTT + YouTube stamps

Standard, boring output formats. A WebVTT chapters track renders markers on the scrub bar via <track kind="chapters">.

# emit.py
import json

def ts(sec):  # seconds -> HH:MM:SS.mmm
    h, rem = divmod(sec, 3600)
    m, s = divmod(rem, 60)
    return f"{int(h):02d}:{int(m):02d}:{s:06.3f}"

def yt(sec):  # seconds -> mm:ss (or h:mm:ss)
    h, rem = divmod(int(sec), 3600)
    m, s = divmod(rem, 60)
    return f"{h}:{m:02d}:{s:02d}" if h else f"{m:02d}:{s:02d}"

chapters = json.load(open("chapters.json"))

# WebVTT
vtt = ["WEBVTT", ""]
for ch in chapters:
    vtt += [f"{ts(ch['start'])} --> {ts(ch['end'])}", ch["title"], ""]
open("chapters.vtt", "w").write("\n".join(vtt))

# YouTube-style (first stamp MUST be 00:00 or the platform ignores all of them)
print("\n".join(f"{yt(ch['start'])} {ch['title']}" for ch in chapters))

chapters.vtt:

WEBVTT

00:00:00.000 --> 00:01:38.400
Introduction

00:01:38.400 --> 00:04:02.900
Setting up the schema

Attach it to any HTML5 player:

<video controls>
  <source src="talk.mp4" type="video/mp4" />
  <track kind="chapters" src="chapters.vtt" srclang="en" label="Chapters" default />
</video>

Things to still watch 👀

• Title style: constrain length in the prompt or you'll get a mix of terse labels and rambling sentences.
• Granularity: scaling chapter count to duration (we did this) avoids over-segmenting short clips and under-segmenting long ones, the long-context omission problem.
• Review high-stakes videos: timestamps are guaranteed correct; the model's judgment about where a topic turns is not. A quick human pass fixes the rare bad boundary.

What's next

• Batch it across your library and store chapters.json next to each asset.
• Add on-screen frame captions to the transcript for videos where the audio alone is thin (the Chapter-Llama CVPR 2025 approach blends both).
• Feed the same segments into search or a summary endpoint. You already did the expensive part (transcription); reuse it.

The whole idea in one line: let the model decide the seams, let your code keep the time. #tutorial