We just launched BhaShaVox, an AI studio that translates, dubs and voice-clones content into 125+ languages, including 20+ Indian languages. This post is the engineering story: how a dub request moves through the system, what was hard, and what we'd do differently.
The shape of the problem
Dubbing a video is not one AI call. It's a chain:
Speech-to-text (who said what, and when)
Translation (keeping meaning, not just words)
Text-to-speech, in a library voice or a cloned voice
Timing, so each speaker's new audio fits their original slot
Mixing it back into the video, with the background music kept
Each step can take seconds to minutes. A web request can't wait that long.
The architecture
Next.js frontend (app router)
Node + Express API, organised as routes → controllers → services
Bull queues on Redis for anything heavy (dubbing, STT, TTS)
Python workers for the studio pipeline (dubbing, voice cloning, vocal separation), called from Node
Socket.io (with the Redis adapter) to push live progress to the browser
MongoDB for app data, Google Cloud Storage for media
mediasoup (an SFU) for live voice/video translation in AutoTalk
Deployed on GCP Cloud Run
The request flow
The API never does the heavy work itself. It validates the request, puts a job on the queue, and returns a job ID straight away:
// simplified
router.post('/dub', auth, async (req, res) => {
const job = await dubQueue.add({
userId: req.user.id,
mediaUrl: req.body.mediaUrl,
targetLanguages: req.body.languages,
voiceId: req.body.voiceId,
});
res.json({ jobId: job.id });
});
A worker picks it up and reports progress over Socket.io as each stage finishes:
// simplified
dubQueue.process(async (job) => {
const emit = (stage, pct) =>
io.to(job.data.userId).emit('dub:progress', { jobId: job.id, stage, pct });
const transcript = await stt(job.data.mediaUrl); emit('transcribed', 25);
const translated = await translate(transcript, job.data); emit('translated', 50);
const audio = await tts(translated, job.data.voiceId); emit('voiced', 80);
const output = await mixAndUpload(audio, job.data); emit('done', 100);
return output;
});
The browser shows a live progress bar instead of a spinner that might hang.
What was hard
- Indian languages need their own path We learned quickly that one provider doesn't fit every language. For speech-to-text we use a fallback chain: an Indic-specialised model is tried first for Indian languages, then general-purpose providers if it fails or returns an empty transcript. Translation works the same way. When either the source or target is an Indian language, we route it to the model that handled Indic text best in our tests.
Small things matter too. Hindi, Bengali and several other Indian languages end sentences with a danda (।), not a full stop. Our clean-up step originally only checked for ., ! and ?, so it kept "fixing" perfectly correct Hindi by adding a stray full stop. Now the danda counts as a sentence end:
if (!cleaned.match(/[.!?।]$/)) { /* add punctuation */ }
LLMs talk too much
When you use an LLM for translation, sometimes it doesn't just translate. It replies with things like "However, I will continue to translate the text as per the rules provided:" and that line ends up spoken in someone's voice in the final video. We now run every translation through a validation step that rejects meta-commentary and instruction echoes before it reaches text-to-speech.Timing: translated speech never fits
A 2-second English line can become 3 seconds of Tamil. If you just place the new audio on the timeline, speakers start talking over each other.
We handle it in three layers:
Split before speaking. If a line's text is too long for its time slot at natural speaking speed, we split it at clause boundaries, then commas, then words, so each chunk can be voiced comfortably.
Speed up only as much as needed. If the generated audio is still longer than the slot (with ~6% slack), we compute the exact tempo required, speech_length / slot_length, and apply a pitch-preserving speed-up with ffmpeg's atempo filter. atempo only accepts 0.5–2.0 per filter, so larger factors are chained. We never slow speech down, and there's a ceiling so it doesn't turn into chipmunk audio.
Trim and cut as a last resort. We trim leading silence and breaths from each clip, and only hard-cut a tail if it would overlap the next speaker.
simplified
needed = speech_sec / slot_sec
if speech_sec <= slot_sec * 1.06:
return wav # fits, leave it natural
tempo = min(needed, MAX_TEMPO) # only as fast as required
wav = atempo_chain(wav, tempo) # pitch-preserving
Keeping the background
A dub with the music stripped out sounds wrong. We run vocal separation (Demucs) once per video to get an instrumental bed and a clean vocal track. The instrumental goes under the new dub. The clean vocals become the reference for voice cloning, so background noise doesn't leak into the cloned voice.GPU cost
We keep GPUs only for the studio work (dubbing and cloning). Text translation and speech-to-text run on API-based models, which keeps per-job cost low enough to offer a free trial.
What we'd do differently
Our first timing fix was a fixed speed-up: if the dub was too long, play it at 1.2x, and if it was still too long, 1.45x. It was simple and it was wrong. Short lines got rushed for no reason, and long ones still overflowed. Calculating the exact tempo each line needs was a small change that fixed most of our "sounds robotic" feedback.
The bigger lesson: build validation between AI steps from day one. Every model in a chain can return something unexpected, and the step after it will happily turn that into audio.
Try it
BhaShaVox is live with a free 14-day trial: https://www.bhashavox.com
If you work with Indian languages, I'd love to hear how the dubs sound in yours. Questions about the stack are welcome in the comments.
Top comments (0)