Create an AI Dubbing Tool for Video Content

Why Build Your Own AI Dubbing Tool?

If you’ve ever needed to localize a video, add a narration track, or simply replace a shaky on‑camera voice, doing it manually can be a nightmare. Traditional dubbing requires hiring voice actors, booking studios, and spending hours syncing audio to video. With modern text‑to‑speech (TTS) and voice‑cloning tech, you can automate most of that workflow in a few lines of code.

In this post I’ll walk you through a practical, end‑to‑end pipeline for creating an AI dubbing tool:

1. Extract the original audio from a video file.
2. Transcribe (or translate) the speech to text.
3. Generate a new voice track with a TTS service.
4. Mix the synthesized audio back into the video.

The heavy lifting for the voice generation will be done by ElevenLabs, a high‑quality, low‑latency TTS platform that also offers voice cloning. Let’s dive in.

---

1. Prerequisites

Tool	Reason
Python 3.9+	Core scripting language
ffmpeg	Audio/video extraction & re‑encoding
pydub	Simple audio manipulation
requests	HTTP calls to ElevenLabs API
OpenAI Whisper (or any ASR)	Optional transcription step
ElevenLabs API key	Access to the TTS endpoint

You can install the Python dependencies with:

pip install pydub requests openai-whisper

Tip: If you’re on Windows, make sure ffmpeg.exe is on your PATH. On macOS/Linux you can usually install it via Homebrew or apt.

---

2. Extract the Original Audio

First, we need the raw audio track from the source video. ffmpeg makes this a one‑liner:

ffmpeg -i input_video.mp4 -vn -acodec pcm_s16le -ar 22050 -ac 1 original.wav

• -vn – ignore the video stream
• -acodec pcm_s16le – uncompressed WAV (easier for later processing)
• -ar 22050 – 22.05 kHz sample rate (ElevenLabs works well with this)
• -ac 1 – mono channel (reduces file size)

You now have original.wav ready for transcription or direct voice cloning.

---

3. (Optional) Transcribe & Translate

If you need subtitles or want to dub into a different language, run the audio through a speech‑to‑text model. Here’s a quick Whisper example:

import whisper

model = whisper.load_model("base")
result = model.transcribe("original.wav")
text = result["text"]
print("Transcribed text:", text)

For translation, Whisper can output directly in another language:

result = model.transcribe("original.wav", language="en", task="translate")
translated = result["text"]
print("Translated to English:", translated)

Save the final script you want to feed into the TTS engine into a variable called script.

---

4. Generate the New Voice with ElevenLabs

ElevenLabs offers a simple REST API. Below is a Python wrapper that sends the script and receives an audio file.

import requests

ELEVEN_API_KEY = "YOUR_ELEVENLABS_API_KEY"
VOICE_ID = "EXAMPLE_VOICE_ID"   # you can use a default voice or a cloned one

def synthesize(text: str, output_path: str):
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{VOICE_ID}"
    headers = {
        "xi-api-key": ELEVEN_API_KEY,
        "Content-Type": "application/json"
    }
    payload = {
        "text": text,
        "model_id": "eleven_monolingual_v1",
        "voice_settings": {
            "stability": 0.75,
            "similarity_boost": 0.85
        }
    }

    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()

    with open(output_path, "wb") as f:
        f.write(response.content)
    print(f"Saved synthesized audio to {output_path}")

# Example usage:
script = "Hello, world! This is an AI‑generated dub."
synthesize(script, "dub.wav")

Using the API from curl

If you prefer a quick terminal test, here's the equivalent curl command:

curl -X POST "https://api.elevenlabs.io/v1/text-to-speech/EXAMPLE_VOICE_ID" \
  -H "xi-api-key: YOUR_ELEVENLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "text": "Hello, world! This is an AI‑generated dub.",
        "model_id": "eleven_monolingual_v1",
        "voice_settings": {
          "stability": 0.75,
          "similarity_boost": 0.85
        }
      }' \
  --output dub.wav

Note: Replace EXAMPLE_VOICE_ID with the ID of the voice you want to use. ElevenLabs lets you clone your own voice with just a few minutes of audio – perfect for brand‑consistent dubbing.

---

5. Align & Mix the New Audio

The synthesized audio will usually be slightly shorter or longer than the original. A simple way to keep sync is to stretch/compress the track to match the original duration using pydub.

from pydub import AudioSegment

original = AudioSegment.from_wav("original.wav")
dub = AudioSegment.from_wav("dub.wav")

# Match length
dub = dub.set_frame_rate(original.frame_rate)
dub = dub.set_channels(original.channels)
dub = dub.set_sample_width(original.sample_width)

# Speed up or slow down to fit the original duration
ratio = len(original) / len(dub)
dub = dub.speedup(playback_speed=ratio, chunk_size=150, crossfade=25)

dub.export("dub_aligned.wav", format="wav")

Now we can replace the original audio track in the video:

ffmpeg -i input_video.mp4 -i dub_aligned.wav -c:v copy -map 0:v:0 -map 1:a:0 -shortest final_dubbed.mp4

• -c:v copy – keep the original video codec (no re‑encoding)
• -map – select streams (video from input 0, audio from input 1)
• -shortest – stops encoding when the shortest stream ends (prevents extra silence)

You now have final_dubbed.mp4 with an AI‑generated voice track!

---

6. Putting It All Together

Below is a minimal script that ties the steps together. Feel free to expand it with error handling, batch processing, or a web UI.

import os
import subprocess
from pathlib import Path
import whisper
import requests
from pydub import AudioSegment

# ---- Config ----
VIDEO_IN = "input_video.mp4"
VIDEO_OUT = "final_dubbed.mp4"
ELEVEN_API_KEY = "YOUR_ELEVENLABS_API_KEY"
VOICE_ID = "EXAMPLE_VOICE_ID"

# 1. Extract audio
subprocess.run([
    "ffmpeg", "-i", VIDEO_IN, "-vn",
    "-acodec", "pcm_s16le", "-ar", "22050", "-ac", "1",
    "original.wav"
], check=True)

# 2. Transcribe (or just set a script)
model = whisper.load_model("base")
result = model.transcribe("original.wav")
script = result["text"]
print("Script to dub:", script)

# 3. Synthesize with ElevenLabs
def synthesize(text, out_path):
    url = f"https://api.elevenlabs.io/v1/text-to-speech/{VOICE_ID}"
    headers = {"xi-api-key": ELEVEN_API_KEY, "Content-Type": "application/json"}
    payload = {"text": text, "model_id": "eleven_monolingual_v1"}
    resp = requests.post(url, json=payload, headers=headers)
    resp.raise_for_status()
    Path(out_path).write_bytes(resp.content)

synthesize(script, "dub.wav")

# 4. Align audio length
orig = AudioSegment.from_wav("original.wav")
dub = AudioSegment.from_wav("dub.wav")
ratio = len(orig) / len(dub)
dub = dub.speedup(playback_speed=ratio)
dub.export("dub_aligned.wav", format="wav")

# 5. Merge back into video
subprocess.run([
    "ffmpeg", "-i", VIDEO_IN, "-i", "dub_aligned.wav",
    "-c:v", "copy", "-map", "0:v:0", "-map", "1:a:0",
    "-shortest", VIDEO_OUT
], check=True)

print("✅ Dubbing complete! Check:", VIDEO_OUT)

You can run this script with python dub_tool.py (after filling in your ElevenLabs API key and voice ID). The result is a fully dubbed video with just a few seconds of processing time.

---

7. Going Further

• Batch processing – Loop over a folder of videos, store each transcript in a database.
• Custom voice cloning – Upload a 30‑second sample of your brand’s voice to ElevenLabs, then use that VOICE_ID for consistent dubbing.
• Web UI – Use Flask or FastAPI to let users upload a video, choose a voice, and download the dubbed result.
• Subtitle sync – Export the Whisper timestamps and embed them as SRT files for closed captions.

All of these extensions build on the same core steps we covered, so you can adapt the pipeline to your product’s needs.

---

8. Wrap‑Up & Call to Action

Building an AI dubbing tool is now a weekend project thanks to modern TTS services. By leveraging ElevenLabs for high‑quality voice synthesis, you get natural‑sounding output without the overhead of recording studios.

Ready to give it a spin? Grab your free trial and API key from ElevenLabs, plug it into the snippets above, and start dubbing your videos in minutes. Happy coding!