Building ALAD: Real-Time AI Video Audio Dubbing in Chrome with Gemini API & WebSockets
Imagine watching a video, live stream, or online course in a language you don't speak, and hearing it dubbed in real-time into your native language—right inside your browser, with almost zero latency.
That is exactly why I built ALAD (Live Audio Dubbing / LAD), an open-source Chrome Extension that leverages Google’s Gemini API and WebSockets to capture live tab audio, translate it on the fly, and play back natural dubbed audio alongside the original video.
In this article, I’ll walk through why I created ALAD, how its architecture works under the hood, the technical hurdles of handling real-time audio in Chrome Manifest V3, and how you can try or contribute to the project!
💡 The Problem: Why Real-Time Dubbing?
Subtitles are great, but they require your constant visual attention. If you are watching a technical tutorial, a lecture, or a fast-paced presentation in a foreign language, scanning subtitles while trying to focus on code or visual slides can be exhausting.
Existing dubbing solutions usually fall into two categories:
Offline post-processing tools: High quality, but slow and impossible for live or streaming video.
Heavy cloud SaaS platforms: Expensive, subscription-based, and locked behind proprietary players.
I wanted a solution that was lightweight, open-source, privacy-friendly, and integrated directly into the browser.
🚀 Introducing ALAD (Live Audio Dubbing)
ALAD connects your active browser tab directly to Gemini's real-time multimodal capabilities using WebSockets.
Key Features:
🎙️ Real-Time Tab Audio Capture: Captures clear raw audio directly from YouTube, Udemy, Twitch, or any HTML5 video player without capturing room ambient noise.
⚡ Low-Latency Streaming: Streams audio chunks over WebSockets for rapid response and minimal delay between visual speaker cues and audio output.
🤖 Powered by Gemini API: Uses Gemini's advanced multimodal audio understanding and natural speech generation.
🎚️ Smart Audio Ducking & Volume Balancing: Lowers the original video audio slightly while playing the dubbed translation so you can comfortably hear both or focus solely on the translated audio.
⚙️ Manifest V3 Compliant: Built strictly for modern Chrome extension security and performance standards.
🔑 BYO Key (Bring Your Own API Key): Keeps your requests private and cost-effective—your API key is stored locally in your browser.
🛠️ Tech Stack & Architecture
Here is how data flows through ALAD in real time:
[ Active Tab (HTML5 Video) ]
│
▼ (tabCapture API / Web Audio API)
[ Audio Processing & PCM Chunking ]
│
▼ (WebSocket Stream)
[ ALAD Background Engine ]
│
▼ (Gemini API Multimodal Endpoint)
[ AI Translation & Audio Synthesis ]
│
▼ (AudioBuffer Queue / Playback)
[ Dubbed Audio Streamed to User ]
Main Technologies Used:
Frontend / Popup: HTML5, CSS3, Modern JavaScript (ESNext)
Extension Platform: Chrome Extension API Manifest V3 (Service Workers, Offscreen Documents, tabCapture)
Audio Engineering: Web Audio API (AudioContext, PCM 16-bit encoding)
Real-Time Transport: WebSockets for low-latency full-duplex binary audio streaming
AI Engine: Google Gemini API (Multimodal Live API)
🔬 Under the Hood: Key Technical Challenges
Developing a real-time audio extension on Manifest V3 came with a few interesting technical challenges:
- Tab Audio Capture in Manifest V3 In Manifest V3, background scripts run inside transient Service Workers, which do not have access to the DOM or AudioContext.
To solve this, ALAD utilizes Chrome’s Offscreen API (chrome.offscreen). When audio capturing starts:
The background service worker spawns an offscreen document.
The offscreen document calls chrome.tabCapture.getMediaStreamId() to capture the active tab's media stream.
Raw audio is piped into a MediaStreamAudioSourceNode inside an AudioContext.
...JavaScript
// Offscreen script snippet for tab audio capturing
const stream = await navigator.mediaDevices.getUserMedia({
audio: {
mandatory: {
chromeMediaSource: 'tab',
chromeMediaSourceId: streamId
}
}
});
const audioContext = new AudioContext({ sampleRate: 16000 });
const source = audioContext.createMediaStreamSource(stream);
// Process and chunk PCM audio data...
...
- PCM Audio Chunking & WebSocket Encoding For Gemini to process live audio seamlessly, audio must be downsampled (typically to 16kHz mono) and chunked into base64-encoded PCM frames before being pushed through the WebSocket connection.
We process incoming audio buffer arrays, convert floating-point samples to 16-bit signed integers (PCM16), and dispatch small binary frames continuously:
JavaScript
function convertFloat32ToPCM16(buffer) {
let l = buffer.length;
let buf = new Int16Array(l);
while (l--) {
buf[l] = Math.min(1, Math.max(-1, buffer[l])) * 0x7FFF;
}
return buf.buffer;
}
- Jitter-Free Audio Queue & Playback Synchronization Receiving streamed audio chunks over a network means packet arrival times can fluctuate. If played back immediately upon receipt, the synthesized voice would sound choppy.
ALAD implements a custom Audio Queue Scheduler. Received PCM chunks are decoded into AudioBuffer objects and scheduled sequentially on the timeline (audioContext.currentTime), creating continuous, stutter-free real-time dubbing.
📦 How to Install and Run ALAD Locally
You can test ALAD right now on your machine in just a few minutes:
Prerequisites:
Google Chrome (or any Chromium-based browser like Brave or Edge)
A Gemini API key (obtainable for free from Google AI Studio)
Installation Steps:
Clone the repository:
Bash
git clone https://github.com/navidseyedain/ALAD.git
cd ALAD
Load into Chrome:
Open Chrome and navigate to chrome://extensions/
Enable Developer mode (toggle in the top right corner).
Click Load unpacked.
Select the ALAD project directory.
Configure & Start Dubbing:
Click the ALAD icon in your Chrome toolbar.
Paste your Gemini API key in the settings tab.
Select your target language.
Open any video on YouTube, click Start Live Dubbing, and enjoy!
🛣️ What's Next? (Roadmap)
ALAD is an active open-source project, and there are several exciting features planned:
[ ] Multi-speaker Detection: Identifying different voices in the video and assigning distinct synthetic voices.
[ ] Offline Cache & Subtitle Overlay: Generating dual synchronized subtitles alongside audio dubbing.
[ ] Chrome Web Store Publishing: Packaging and releasing ALAD to the official Chrome Web Store for one-click installation.
[ ] Custom Voice Parameters: Pitch, speed, and emotion adjustments for dubbed voices.
🤝 Open Source & Contributions
ALAD is completely open-source and released under the MIT License. Contributions are super welcome! Whether you want to fix a bug, improve audio synchronization, refine the UI, or add support for new languages, feel free to open an issue or pull request.
🌐 GitHub Repository: https://github.com/navidseyedain/ALAD
👤 Author: Navid Seyedain (@navidseyedain)
If you find this project useful or interesting, please consider dropping a ⭐ Star on the repository to support development!
Thank you for reading! Feel free to leave your thoughts, feedback, or questions in the comments below.
Top comments (0)