DEV Community

Cover image for How I Built ALAD Mobile: Real-Time AI Audio Dubbing for Any Android App
navidseyedain
navidseyedain

Posted on

How I Built ALAD Mobile: Real-Time AI Audio Dubbing for Any Android App

#ai #opensource #android #kotlin

Have you ever wanted to watch a foreign YouTube video, a live stream on Twitch, or a show on Netflix without constantly staring at subtitles?

To solve this, I built ALAD Mobile (Live Audio Dubbing) — an open-source native Android application that captures internal system audio in real-time, streams it through Google's Gemini Multimodal Live API, and instantly plays back the translated voice over any active media.

💡 What It Does

ALAD Mobile creates a system-wide floating overlay widget that sits comfortably over your active applications. Once activated:

  • 🎧 Intercepts Audio: It captures the internal playback audio of any app playing media.
  • ⚡ Live Translation: Streams audio chunks to Gemini's bidirectional Multimodal Live API via WebSockets.
  • 🎙️ Real-Time Playback: Plays back the dubbed voice in real time in your selected language (supporting 78 languages).
  • 📱 Seamless Overlay: Lets you toggle dubbing or switch target languages without ever leaving YouTube, Netflix, or Spotify.

🛠️ Tech Stack & Architecture

Building low-latency real-time voice translation on mobile required a carefully structured stack:

  • Language: 100% Kotlin
  • UI Framework: Jetpack Compose (Material 3)
  • Async Programming: Kotlin Coroutines, StateFlow, and SharedFlow
  • Networking & Streaming: WebSockets for low-latency bidirectional audio streaming
  • Android Audio APIs: MediaProjection / AudioRecord for internal audio capture + AudioTrack for low-latency playback
  • System Service: Floating Overlay Window using TYPE_APPLICATION_OVERLAY
  • AI Engine: Google Gemini Multimodal Live API

🧠 How It Works Under the Hood

1. Internal Audio Capture

Using Android's MediaProjection API and AudioRecord, the app captures raw PCM audio streams from the device's internal sound engine without relying on external microphone input.

2. Bidirectional Live WebSockets

Traditional REST API requests introduce too much latency for live translation. ALAD Mobile opens a persistent WebSocket connection directly to Gemini's Live API, continuously streaming PCM audio chunks in real-time.

3. Audio Track Playback

As Gemini processes and translates the stream, incoming audio payloads are decoded and pushed directly into a low-latency AudioTrack buffer for smooth, near-instant voice playback.

4. Floating System Overlay

To ensure zero friction, a Floating Control Widget built with Android's System Alert Window lets users control dubbing and switch target languages seamlessly over any running app.

🚀 Technical Challenges & Lessons Learned

  • Minimizing Latency: Balancing buffer sizes between raw PCM capture and WebSocket transmission frames was crucial for keeping speech output synchronized with active playback.
  • State Management: Handling connection lifecycle events, audio buffer underruns, and background service states seamlessly using Kotlin StateFlow.

🌐 Open Source & How to Try It

ALAD Mobile is 100% Free and Open Source. It runs using a free API key directly from Google AI Studio — no subscriptions or user accounts required.

If you find this project interesting, feel free to drop a ⭐ star on GitHub, submit a PR, or leave your architectural feedback below!

Top comments (0)