I wanted to dub short videos without sending the source media to a hosted dubbing platform. The hard part is not producing one synthetic sentence. It is keeping the original speech timing, preserving the background, translating the transcript, cloning a voice, and rebuilding a video that still feels like the same video.
That led me to build Video Dubbing Translator, a local-first, browser-based pipeline for short-form video dubbing.
The pipeline
The current flow is:
video → vocal/background separation → Whisper/WhisperX → translation → VoxCPM2 voice cloning → duration fitting → reconstruction → optional LatentSync
The interface is designed around inspectable stages rather than one opaque “dub this” button. I can keep the extracted vocal and background tracks separate, see the transcript and translation, synthesize each translated segment, fit its duration, and then rebuild the final video.
Translation currently uses Google Translate, so “local-first” does not mean “fully offline.” The media processing and speech synthesis run locally, while translation is the deliberate network-dependent step.
Why VoxCPM2
The project previously used XTTS-v2. I migrated the active voice-cloning path completely to VoxCPM2 because it is the direction I want this project to grow around: reference-voice cloning, native 48 kHz speech output, and an official model/source with an Apache-2.0 license.
This is not a claim that one model wins every voice or language test. It is a practical integration decision. VoxCPM2 gives the pipeline a clear voice-cloning backend, and the 48 kHz output fits naturally into the reconstruction stage.
A real English → Turkish run
I ran an end-to-end test on a 15-second English clip:
- The original audio was separated into vocal and background tracks.
- Whisper produced the transcript and WhisperX supplied word-level timing.
- The transcript was translated to Turkish with Google Translate.
- Four Turkish segments were synthesized with VoxCPM2 using a reference voice.
- Each segment was duration-fitted before the tracks were mixed back together.
- The result was reconstructed as H.264/AAC video with 48 kHz stereo audio.
The rendered output is about 14.88 seconds long. Running Turkish ASR over the final audio recovered all four intended Turkish sentences. That is useful evidence that the pipeline works end to end, but it is one short integration test—not a scientific benchmark.
Here are the artifacts:
The parts that mattered most
Timing is a first-class output
A translated sentence rarely has the same duration as the source sentence. I therefore treat duration fitting as its own stage instead of hoping the synthesizer happens to match. The target duration comes from the timed transcript, and the generated segment is adjusted before reconstruction.
Keep the background intact
For a short clip, the background is part of the identity of the scene. Separating it before synthesis means the new speech can replace the original vocal track without throwing away the surrounding sound.
Keep the audio contract explicit
The current demo is rebuilt with 48 kHz stereo audio and H.264/AAC video. Making the output contract explicit catches format problems early and makes the resulting file easier to inspect or share.
Lip-sync is optional
LatentSync is an optional stage, not a hidden requirement of the base pipeline. The core demo is speech dubbing and reconstruction; lip-sync can be added when the project and hardware are ready for that extra pass.
What is not solved yet
There are several important boundaries:
- Google Translate means the pipeline is not fully offline.
- CPU synthesis is slow. My smoke run used an 8 GB M1 with CPU/bfloat16 settings; CUDA has not been validated in this project.
- There is no production-grade speaker diarization yet.
- Multiple speakers, overlaps, and rapid turn-taking need more work than the single-speaker demo.
- LatentSync is optional and is not part of the base validation claim.
- A short clip cannot establish broad quality, latency, or language coverage.
Those limits are part of why I am publishing the implementation and the artifacts instead of presenting the demo as a finished dubbing product.
Try it and contribute
The repository is MIT-licensed and includes the browser UI, processing stages, setup notes, and the demo links:
github.com/kadirb4rut/video-dubbing-translator
I would especially like feedback on voice similarity, timing behavior, installation on different GPUs, multilingual edge cases, and how the pipeline behaves with more than one speaker. The goal is a reproducible local-first workflow that people can inspect and improve—not a black-box hosted service.
If you are building with WhisperX, VoxCPM2, or local video tooling, I would be interested in what you would change first.
Top comments (0)