DEV Community

Mason K
Mason K

Posted on

Add speaker labels to video captions with WhisperX and WebVTT voice spans

#python #ffmpeg #video #tutorial

TL;DR

We'll turn a two-person video into a WebVTT caption file where each cue says who is speaking. Pipeline: ffmpeg extracts audio, whisperx --diarize transcribes and labels speakers, a small Python script groups words into <v Speaker> cues.

Plain speech-to-text gives you a wall of words. For interviews, podcasts and meetings, readers need to know who said what. We'll use WhisperX (Whisper with voice activity detection, forced alignment and pyannote-based diarization) and the standard WebVTT voice span so any player can show the names. #python #video

You'll need python 3.10+, ffmpeg, a Hugging Face account, and ideally a CUDA GPU. CPU works with --compute_type int8 --device cpu, just slower.

1. 🔑 Get access to the diarization model

WhisperX's diarization uses pyannote's speaker-diarization-community-1 pipeline, which is gated on Hugging Face:

  1. Create a read token at hf.co/settings/tokens.
  2. Open the pyannote/speaker-diarization-community-1 model page and accept the user conditions. 
export HF_TOKEN=hf_xxx   # never commit this
Enter fullscreen mode Exit fullscreen mode

⚠️ Note: if you skip the user-conditions step, the pipeline fails to load with an authorization error even with a valid token. Accept the conditions first.

2. 🎧 Extract audio from the video 

# extract_audio.sh
ffmpeg -y -i interview.mp4 -vn -ac 1 -ar 16000 audio.wav
Enter fullscreen mode Exit fullscreen mode

Mono, 16 kHz is what speech models work with internally, and a WAV file avoids decoder surprises.

3. 🗣️ Transcribe and diarize 

pip install whisperx
whisperx audio.wav --model large-v2 --diarize --hf_token "$HF_TOKEN" \
  --output_format json --output_dir out
Enter fullscreen mode Exit fullscreen mode

On CPU:

whisperx audio.wav --model large-v2 --diarize --hf_token "$HF_TOKEN" \
  --compute_type int8 --device cpu --output_format json --output_dir out
Enter fullscreen mode Exit fullscreen mode

The README lists the CPU flags and the diarize flag. WhisperX does four things in order: voice activity detection, Whisper transcription, wav2vec2 forced alignment (word-level timestamps), then pyannote diarization with speakers assigned to words.

💡 Tip: WhisperX's README claims 70x realtime for large-v2 batched transcription. That's their best case. Time it on your own hardware before planning capacity.

4. 🔍 Look at the JSON

Each segment has a speaker, and each aligned word carries one too (shortened example):

{
  "segments": [
    {
      "start": 0.5, "end": 4.2,
      "text": "Welcome back to the show.",
      "speaker": "SPEAKER_00",
      "words": [
        {"word": "Welcome", "start": 0.5, "end": 1.0, "speaker": "SPEAKER_00"},
        {"word": "42", "speaker": "SPEAKER_01"}
      ]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Notice the last word: numbers and symbols often come back with no start/end. Our converter has to skip those or it will crash.

⚠️ Note: the exact JSON fields can shift between WhisperX versions. Open one real output file and confirm words[].speaker exists before trusting the script below.

5. 🧩 Group words into WebVTT cues

The rules:

  • a new cue starts when the speaker changes,
  • or after a pause longer than 1.5 s,
  • or when a cue would run longer than 7 s (so captions stay readable). 
#!/usr/bin/env python3
"""Convert WhisperX JSON (with --diarize) into a WebVTT file using <v Speaker> voice spans."""
from __future__ import annotations

import json
import sys

MAX_CUE_SECONDS = 7.0   # split long monologues so captions stay readable
GAP_SECONDS = 1.5       # a pause this long starts a new cue even for the same speaker


def ts(seconds: float) -> str:
    ms = round(seconds * 1000)
    h, ms = divmod(ms, 3_600_000)
    m, ms = divmod(ms, 60_000)
    s, ms = divmod(ms, 1000)
    return f"{h:02d}:{m:02d}:{s:02d}.{ms:03d}"


def words_with_speakers(data: dict) -> list[dict]:
    out = []
    last_speaker = "UNKNOWN"
    for seg in data["segments"]:
        for w in seg.get("words", []):
            if "start" not in w:          # numbers and symbols often get no timestamp
                continue
            last_speaker = w.get("speaker", seg.get("speaker", last_speaker))
            out.append({"word": w["word"], "start": w["start"], "end": w["end"], "speaker": last_speaker})
    return out


def build_cues(words: list[dict]) -> list[dict]:
    cues, cur = [], None
    for w in words:
        new = (
            cur is None
            or w["speaker"] != cur["speaker"]
            or w["start"] - cur["end"] > GAP_SECONDS
            or w["end"] - cur["start"] > MAX_CUE_SECONDS
        )
        if new:
            cur = {"speaker": w["speaker"], "start": w["start"], "end": w["end"], "text": [w["word"]]}
            cues.append(cur)
        else:
            cur["text"].append(w["word"])
            cur["end"] = w["end"]
    return cues


def to_vtt(cues: list[dict], names: dict[str, str]) -> str:
    lines = ["WEBVTT", ""]
    for i, c in enumerate(cues, 1):
        name = names.get(c["speaker"], c["speaker"])
        lines += [str(i), f"{ts(c['start'])} --> {ts(c['end'])}", f"<v {name}>{' '.join(c['text'])}", ""]
    return "\n".join(lines)


if __name__ == "__main__":
    data = json.load(open(sys.argv[1]))
    names = json.loads(sys.argv[2]) if len(sys.argv) > 2 else {}
    print(to_vtt(build_cues(words_with_speakers(data)), names))
Enter fullscreen mode Exit fullscreen mode

Run it, mapping the generic labels to names you know:

python3 diar_to_vtt.py out/audio.json '{"SPEAKER_00":"Host","SPEAKER_01":"Guest"}' > captions.vtt
Enter fullscreen mode Exit fullscreen mode

On a small hand-written sample in the same shape as WhisperX output, the result is:

WEBVTT

1
00:00:00.500 --> 00:00:02.100
<v Host>Welcome back to the show.

2
00:00:02.600 --> 00:00:04.100
<v Guest>Thanks for having me.
Enter fullscreen mode Exit fullscreen mode

The <v Name> syntax is the WebVTT voice span from the W3C spec. The closing </v> can be omitted when the span covers the whole cue, which is the case here.

6. ▶️ Attach it to a player 

<!-- player.html -->
<video controls src="interview.mp4">
  <track kind="captions" srclang="en" label="English" src="captions.vtt" default />
</video>
Enter fullscreen mode Exit fullscreen mode

HLS players take the same file as a subtitles rendition if you package it. Style the speaker with the ::cue(v[voice="Host"]) selector.

/* captions.css */
video::cue(v[voice="Host"]) { color: #ffd166; }
video::cue(v[voice="Guest"]) { color: #8ecae6; }
Enter fullscreen mode Exit fullscreen mode

7. 🚧 Limits to know before shipping

Limit What happens What to do
Generic labels Output says SPEAKER_00, not a name Map by hand, from invite metadata, or a speaker-ID step
Per-file labels SPEAKER_00 in one file is not SPEAKER_00 in another Carry identity yourself across recordings
Overlapping speech One word gets one speaker Review a sample of crosstalk-heavy content
Model accuracy pyannote reports 11.2 DER on VoxConverse for community-1 (project's own benchmark) Score on your own labelled sample

What's next

  • Score your own accuracy: hand-label ten minutes of your content and compute diarization error rate against it.
  • Index the speaker field so search can answer "what did the guest say about X".
  • If your content has fixed hosts, look at speaker-embedding matching to name them automatically.

Wrapping up: three commands and one small script give you captions that identify speakers, in a format every player already understands.

Top comments (0)