Build an AI dubbing pipeline: faster-whisper + XTTS-v2 + FFmpeg

TL;DR

We're building a script that takes a video in English and produces the same video narrated in Spanish, in a cloned version of the original speaker's voice. Stack: faster-whisper for timestamped transcription, an LLM (or any MT engine) for translation, XTTS-v2 for voice-cloned synthesis, FFmpeg for surgery. We'll also handle the problem every demo skips: translated audio that doesn't fit its time slot.

📦 Code: github.com/USER/repo (replace before publishing)

If you'd rather start from a finished system, Softcatala's open-dubbing and KrillinAI are full pipelines behind one CLI. This post builds the minimal version by hand so you understand what those tools are doing, and where they break.

0. Setup and a licensing warning ⚠️

Python 3.10–3.12. The original Coqui company shut down in early 2024; the maintained fork of their TTS library is published by Idiap as coqui-tts:

$ python -m venv dub && source dub/bin/activate
$ pip install faster-whisper coqui-tts
$ ffmpeg -version | head -1   # 6.0+ is fine, 8.x current

⚠️ Note: the XTTS-v2 model weights ship under the Coqui Public Model License, which restricts commercial use. Prototype freely, but before dubbed videos ship to paying customers, someone must read that license and possibly swap the synthesis step for a commercially licensed model or paid API. Voice cloning also requires the speaker's consent. Get it in writing.

1. Extract audio and transcribe with word timestamps 🎙️

# pull mono 16k audio for the ASR step
$ ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -y source.wav

# dub/transcribe.py
from faster_whisper import WhisperModel

model = WhisperModel("large-v3-turbo", compute_type="int8")
segments, info = model.transcribe("source.wav", word_timestamps=True)

lines = []
for seg in segments:
    lines.append({
        "start": seg.start,
        "end": seg.end,
        "text": seg.text.strip(),
    })
print(f"language={info.language} segments={len(lines)}")

The timestamps are the skeleton of the whole pipeline. Every downstream step preserves start/end per segment, because that's where the translated speech has to fit back.

2. Translate with a length budget 🌍

Per-segment MT gives you sentences that are individually fine and collectively wrong (inconsistent terminology, drifting register). Feed the whole transcript to your translation step with context, and, crucially, give it a length constraint per segment. This is the single biggest lever against sync drift:

# dub/translate.py (engine-agnostic sketch)
PROMPT = """Translate this video narration from English to Spanish.
Rules:
- Keep terminology consistent (glossary: {glossary})
- Each numbered line must be speakable within its duration.
  Line 3 has 2.8s. Line 7 has 4.1s. Prefer shorter phrasings.
- Return the same numbered lines, translated."""

Whether the engine is an LLM, a local NLLB/M2M model, or a cloud MT API matters less than the contract: same segments in, same segments out, lengths respected. Have a native speaker skim the output. One reviewer-hour here prevents most of the embarrassment this pipeline can produce.

3. Clone the voice and synthesize 🗣️

XTTS-v2 supports 17 languages and clones a voice from a few seconds of clean reference audio. Cut a reference clip of the original narrator (no music, no crosstalk):

$ ffmpeg -i source.wav -ss 00:00:12 -t 8 -y reference.wav

# dub/synthesize.py
from TTS.api import TTS

tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")

for i, seg in enumerate(translated_segments):
    tts.tts_to_file(
        text=seg["text_es"],
        speaker_wav="reference.wav",
        language="es",
        file_path=f"segments/{i:04d}.wav",
    )

First run downloads the weights; after that it's local. GPU strongly recommended; CPU works for short content if you're patient.

4. The boss fight: fitting audio back into time slots ⏱️

Spanish runs longer than English as a rule. Some synthesized segments will overflow their slots, and naive concatenation drifts out of sync within minutes. Measure first:

# dub/align.py
import soundfile as sf

report = []
for i, seg in enumerate(translated_segments):
    audio, sr = sf.read(f"segments/{i:04d}.wav")
    actual = len(audio) / sr
    slot = seg["end"] - seg["start"]
    report.append((i, slot, actual, actual / slot))

for i, slot, actual, ratio in report:
    flag = "⚠️ OVERFLOW" if ratio > 1.1 else "ok"
    print(f"seg {i:04d}  slot={slot:.2f}s  synth={actual:.2f}s  ratio={ratio:.2f}  {flag}")

seg 0007  slot=4.10s  synth=5.23s  ratio=1.28  ⚠️ OVERFLOW
seg 0012  slot=2.80s  synth=2.91s  ratio=1.04  ok

Then apply fixes in escalating order:

1. Absorb into silence. If the next segment starts late, let the audio spill into the gap. Free and inaudible.
2. Time-stretch gently. FFmpeg's atempo up to ~1.1 is usually imperceptible on speech; beyond that it sounds rushed:

$ ffmpeg -i segments/0007.wav -filter:a "atempo=1.12" -y segments/0007_fit.wav

1. Re-translate the outliers. Anything still over ratio ~1.2 goes back to step 2 with a tighter length budget. Retranslating five bad segments beats stretching fifty.

Build the final track by placing each segment at its original start on a silent canvas, then remux against the untouched video stream:

# assemble placed segments into one track (adelay per segment, amix), then:
$ ffmpeg -i input.mp4 -i dubbed_es.wav \
    -map 0:v -map 1:a -c:v copy -shortest -y output_es.mp4

-c:v copy matters: the video stream is never re-encoded, so the dub costs nothing in visual quality.

5. Ship it as a track, not a fork 📦

Don't create tutorial_es_final_v2.mp4 files. Mux the dub as an additional audio track and let the player expose a language menu:

$ ffmpeg -i input.mp4 -i dubbed_es.wav \
    -map 0 -map 1:a -c copy \
    -metadata:s:a:0 language=eng -metadata:s:a:1 language=spa \
    -y output_multilang.mp4

For HLS delivery, each language becomes an audio rendition in the master playlist; one video ladder, N audio tracks, and the player switches without a second stream.

Things that will bite you 🧾

A short list from the failure modes this kind of pipeline reliably produces:

• Numbers, units, and code. TTS models mangle "0.00096" and "av1_vulkan" in every language. Pre-process the script: expand numbers to words in the target language, and decide whether code identifiers stay English (they should).
• Background music. If the source audio has music under the voice, your extracted track carries it into transcription fine, but your dubbed track loses it entirely. Either separate stems first (Demucs works) or accept music-free dubs; mixing the original music bed back under the synthesized voice is the professional-sounding middle path.
• Speaker changes. One reference clip means one voice. Interviews and multi-presenter webinars need diarization (who speaks when) before synthesis, which is where the prebuilt pipelines earn their keep.
• Acronyms and product names. Add them to the glossary with explicit pronunciation guidance, or enjoy hearing your product's name pronounced five different ways across one video.
• Silence is load-bearing. Don't trim inter-segment gaps to make room; viewers use those pauses to process what's on screen.

What's next

• Automate the pipeline against your upload flow and keep a per-language glossary file under version control; translation consistency is what makes a library feel professionally localized.
• The overflow report from step 4 is your QA dashboard. Track the overflow rate per language over time; it tells you when your translation prompt or TTS pacing regressed.
• When you outgrow the DIY version: open-dubbing and KrillinAI cover more edge cases (multi-speaker, subtitle sync), and lip-sync models exist as a heavier stage for talking-head content.