The moment things got weird was around the third scene. A character said one short line, and my dub played a long, wordy stretch that I didn't recognize. I rewound, read it, and it clicked: nothing was mistranslated. The model had rewritten the line. Tidy, polished, and somehow longer than anything that was actually spoken.
I'd built LiveDub, a little tool where a browser extension grabs a tab's audio, Whisper turns it into text, a local LLM translates it, and Kokoro speaks the result out loud over the original. All of it runs on my PC, so I had trace logs over everything. The culprit was never the speech-to-text part.
It was the translation step.
Here's what I hit: local LLMs today are damn eager to help. Hand one a sentence in one language and ask for it in another, and a lot of them won't do the single, literal thing you asked. They guess your job is film adaptation. They fill in the actor's mood. They compress three lines into something sharper. Professional translators are trained to do the opposite, but we don't hand them a fresh prompt every line.
On paper the output looked good. Better, some sessions, than the subtitles in the actual movie, which is the trap. I was re-captioning a film scene by scene, so every reinterpretation pushed the audio a beat off the actors' mouths, and the extra words dragged the synthesized voice so long that whole exchanges drifted.
The fix was the system prompt, and it took me three attempts to get the tone right. The version that won looks roughly like this:
You are a faithful interpreter. Translate the exact meaning and tone, nothing more.
Do not expand, condense, or paraphrase for drama, and do not add your own reaction.
If the line is short and flat, return short and flat. Output only the translated line.
That last line - "output only the translated line" - is the one that carried the most weight. LLMs respond to instructions about the shape of the reply almost as much as to what's in it. Give them nothing to perform into.
It took a couple of hours of back-and-forth to land on that balance. The day it clicked was unglamorous: a flat, slightly rude, two-word line came back as a flat, slightly rude, two-word line. Almost dull to watch, and that was the win. With real-time dubbing tooling, success isn't a better rendering. It's a faithful one, and no more.
If you're pointing a local LLM at translation, don't work around it line by line. Bake the constraint into the prompt first, guard the shape, and test against lines you already know. And if you're curious how the whole thing hangs together - browser tab audio capture for any video, streaming Whisper, voice switching by pitch - I packaged the complete offline Windows-only tool at https://symshah.gumroad.com/l/livedub for a one-time $19. No cloud, no monthly bill, no the model quietly "improving" your script.
Top comments (0)