Getting Tab Audio Out of a Browser Was the Hardest Part of My Dubbing Tool

I keep getting asked what the hardest part of building a real-time dubbing tool was. Most people assume it's the model. It wasn't. The week that made me question my life choices was spent on something far dumber: capturing just one tab's audio and handing it to a process on my own machine, without recording my whole desktop or a mic track of me muttering at the screen.

A browser tab isn't a file on disk. You can't point at it like a URL and slurp it. The audio is being rendered live inside an open page, surrounded by however many other tabs and whatever else the OS happens to be playing. Record the desktop and you get everything, half of it other people's notifications. Grab the mic and you get your own voice layered over the track, which is worse. The honest answer is to live inside the page itself, as the thing that taps the stream before it ever reaches the system mixer. An extension running in the tab is the only thing that gets you exactly the one piece of audio you're trying to read, and none of the noise.

Once that's working, the rest is a pipeline I've written about in pieces before, so here's the wide view: the extension shuttles audio frames to a local Whisper that transcribes in slices rather than waiting for the whole film, each slice goes to a local model in LM Studio for translation, and a neural TTS called Kokoro speaks the result with one of two voices picked by the pitch of the original speaker. All of it happens on a Windows box with Python 3.10+ and a small model around 2GB. Nothing leaves the machine, and once the initial setup is done the whole thing runs with the network cable unplugged.

The streaming part is where the plumbing actually earns its keep. You can't translate a movie as one giant block, you'd be wrapping up sometime after the post-credits scene. So Whisper gets fed in short windows, each window comes back as a sentence, and each sentence only gets translated once the speaker has finished it. That's the couple-of-words delay every tool like this has. It's not a flaw, it's a rule of the game: you cannot speak a line you haven't heard yet. A real-time studio sitting on a ten-thousand-dollar GPU box lives under the same constraint.

Loosely, the loop looks like this:

while video plays:
    chunk = frame.wait_for_next()          # audio grabbed by the extension
    line  = whisper.transcribe(chunk)     # local transcription
    out   = local_llm.translate(line)     # LM Studio, not the cloud
    voice = pick_for_pitch(line)
    audio = kokoro.speak(out, voice)
    duck_and_play(audio)                  # original audio dips underneath

When that finally clicked, the payoff felt borderline unfair. I put a French crime film on with a Spanish dub track and it kept pace in something close to real time while the original volume ducked underneath. No GPU required, though a decent one tightens the gap between a sentence ending and the next one landing.

If you'd rather watch a whole working pipeline than solder these pieces together yourself, I packaged it as a single Windows download, browser extension, local transcription, translation, and voice choosing, one up-front price and no subscription: https://symshah.gumroad.com/l/livedub