The Hidden Engineering Problem in Real-Time Movie Dubbing
I've been building a tool called LiveDub — it plays movies in your browser and dubs them into another language in real time. Everything runs on your own machine. No API calls leave the computer. You hear translated speech while watching, just like you'd hear subtitles, but spoken aloud.
People ask me how it works by looking at the pipeline:
browser extension captures tab audio → Whisper transcribes → LLM translates → neural voice speaks back
That diagram is accurate but misleading. It makes the whole thing sound like a simple chain of well-behaved components. The real challenge isn't any single step — it's keeping them all in sync with what's actually happening on screen.
Translation has natural latency
Here's something most people don't realize about dubbing: you can't translate while someone is speaking. You have to wait until they stop talking before the pipeline can even begin processing. Then Whisper needs a moment to finish its transcription, then the LLM translates it, then the voice synthesizer generates audio.
That means there's always some delay between what happens on screen and what you hear in the dubbed track. If that delay is too short, the dub runs out of material — it finishes translating before the person on screen actually stops speaking, and then it has nothing to say while they keep talking. That sounds ridiculous but it's one of the most common bugs I've had to fix.
The buffer strategy
The solution is a sliding audio buffer. Instead of feeding translated text into the voice synthesizer immediately, I hold each segment for a few seconds in a queue. This does two things:
- It gives Whisper and the LLM time to finish processing without creating gaps
- It creates enough headroom that if someone speaks longer than expected, the dubbed track still has material queued up when they stop talking
The tricky part is choosing the right buffer size. Too short and you get the "runs out of dub" problem described above. Too long and there's an obvious lag — you see a character finish speaking on screen, then wait another five seconds for their translated voice to arrive.
In practice, I found that holding translated segments in a 3-5 second buffer works well enough that most viewers don't notice the delay at all. The brain fills in the gap because the dub stays consistent and doesn't stutter or overlap.
Ducking the original audio
Another thing people tend to overlook: you can't just layer dubbed audio on top of the original soundtrack. It sounds like two different movies playing simultaneously, which is worse than no dub at all.
The browser extension I built for LiveDub intercepts the tab's audio output and measures its volume in real time. When a translated segment is being spoken, it ducks (lowers) the original audio by about 15-20dB during that window. The ducking fades back up when the dubbed speech ends.
This isn't perfect — there are edge cases where background music or sound effects bleed through — but for dialogue-heavy scenes it works remarkably well. And crucially, everything happens locally. No cloud processing, no latency from round-tripping to a server.
Speaker switching by pitch
One feature I'm particularly proud of: the dub switches between male and female voices based on audio pitch analysis. When the system detects that someone is speaking with a higher fundamental frequency (typically a woman or child), it switches to the female voice model, and vice versa. This means you don't get a single narrator's voice for every character — different speakers actually sound different.
The pitch detection runs on each audio segment before synthesis, so it adapts dynamically. It's not perfect — sometimes a low male voice is shouting and sounds higher than usual — but the false positive rate is low enough that it's useful in practice.
What it takes to run
Windows only (the browser extension approach works best there). Python 3.10+, LM Studio installed for local LLM inference, and one small translation model (~2GB download). No GPU required — it runs on CPU fine, though a discrete GPU makes Whisper noticeably faster.
After the initial setup and model downloads, everything is completely offline. Your audio never leaves your machine.
The tool costs $5 as a one-time purchase on Gumroad. No subscription, no recurring fees. If you've ever wanted to watch foreign content with translated dialogue spoken aloud without depending on any cloud service, it might be worth trying.
Top comments (0)