Whisper streaming transcription is a contradiction in terms, and you find that out about two hours into building it. Whisper is a batch model. It wants a whole file and it'll sit there chewing on it until it's done, then hand you a transcript. Live audio doesn't work that way.
I built a real-time video dubber that runs completely on my own machine, nothing uploaded and offline after setup. Browser tab audio comes out, gets transcribed, translated, and spoken back over the top of the original track. The transcription is the first and fussiest leg of that chain, and it's where most people give up.
Whisper streaming transcription: what actually works
The trick is that you don't really stream Whisper at all - you fake it. Feed it a rolling window of a few seconds and keep whatever it agrees on twice in a row. That "local agreement" rule is the whole game. Three things make it hold up for me:
- Use faster-whisper, not the vanilla OpenAI package. Same models, several times faster, and it runs on CPU without complaining. One line: pip install faster-whisper.
- Voice activity detection in front of the buffer. Once I added it, the hallucinations from transcribing silence mostly disappeared.
- Trim the buffer at a confirmed sentence and carry the tail over. Whisper gets sloppier the more context you dump on it, so keep the window short.
And the honest bit: there's always a delay. Whisper won't commit a sentence until it's more or less finished, because a half-heard word changes everything that follows. Mine holds back about a sentence before it commits and starts translating. Every real-time dubber does this, cloud or not. Anything promising you word-for-word simultaneous dubbing in a language this close to real is either a different class of model or overselling.
Why keep it local? Because dubbing a film means pushing a whole evening of audio, and every word of it, through a service someone else runs. On my machine nothing leaves it, and once the models are cached the internet doesn't matter. Whisper here, an LLM in LM Studio for the translation, a small neural TTS for the speaking.
I wrapped the whole chain into a Windows app and a browser extension so I don't have to think about these loops anymore. If the idea of dubbing a film without uploading a frame of it sounds useful, it's at symshah.gumroad.com/l/livedub.
FAQ
- Does it need a GPU? No. faster-whisper keeps up on CPU if you keep the model small. A GPU makes it snappier, that's all.
- How much delay? About a sentence. It can't translate words it hasn't heard finish yet.
- Can it hear any video? Tab audio capture means anything that plays in your browser, plus local files. Getting that capture loop working was its own little war.
Top comments (0)