DEV Community: navidseyedain

How I Built an In-Place Android Screen Translator with Jetpack Compose, ML Kit, and Gemini Vision

navidseyedain — Sat, 29 Aug 2026 13:29:23 +0000

Language barriers in mobile apps, games, and foreign media are a constant friction point. Traditional screen translation tools on Android usually force users to take manual screenshots, freeze the screen, jump between applications, or deal with rigid line-by-line OCR card overlays that block the view.
To solve this, I built ALST (AI Live Screen Translation) — an open-source, real-time Android screen translator that translates any screen content in-place and renders the translated text directly over the original screen coordinates without ever leaving your active app.
In this article, I'll walk through the system architecture, how I integrated Gemini 3.6 Flash for single-pass vision translation, and how to prevent memory leaks during continuous high-density screen capturing in Android 14+.
🏗️ High-Level System Architecture
ALST is built following Clean Architecture and MVVM/MVI design patterns. The codebase is strictly partitioned into single-responsibility modules:
core/capture: MediaProjection, VirtualDisplay, & ImageReader pipeline
core/ocr: Google ML Kit Text Recognition v2 engine
core/translator: Dual translation engine (ML Kit On-Device + Gemini Flash)
core/overlay: WindowManager floating views & Compose Canvas rendering
service: ScreenTranslatorService (Foreground) & QSTranslateTileService
data: Jetpack DataStore preferences (BYOK API keys, language options)
ui: Material 3 Glassmorphic Dashboard & Overlay UI
🧠 1. Single-Pass Vision AI vs. Traditional Chaining
In conventional translation tools, the pipeline is split into three heavy steps:
Capture screen frame -> Run local OCR -> Extract text blocks & coordinates.
Send extracted raw text to a Translation API -> Receive translated strings.
Draw text cards over the screen.
While this works, it often loses contextual meaning (e.g., in video games, manga, or slang-heavy social posts).
With ALST, when running in Cloud AI Mode, the app uses Google's Gemini 3.6 Flash Multimodal Vision API:
Raw bitmap frame buffers are sent directly to the model.
In a single pass, Gemini extracts the text, understands local context/idioms, and returns both the translated text and exact bounding box coordinates (Rect).
This dramatically improves translation quality and contextual awareness while reducing pipeline complexity.
📴 2. Dual-Engine Architecture (Cloud + 100% Offline)
Recognizing that users aren't always connected to high-speed internet, ALST implements a flexible Dual-Engine System:
Cloud Engine (Gemini 3.6 Flash): Uses the com.google.ai.client.generativeai SDK with a Bring-Your-Own-Key (BYOK) model for deep contextual translation.
On-Device Engine (Google ML Kit): Combines ML Kit Text Recognition v2 with ML Kit On-Device Translation. It operates 100% offline with sub-50ms latency and zero server dependencies.
Users can toggle seamlessly between these two engines inside the app dashboard, persisted via Jetpack DataStore Preferences.
🔋 3. Zero-Leak Memory Engineering in Android 14+
Capturing raw 1440p / 4K screen frames produces bitmaps that consume over 15MB–20MB of RAM per frame. Doing this continuously using MediaProjection and ImageReader quickly leads to OutOfMemoryError (OOM) crashes if buffer recycling isn't managed strictly.
Here is how ALST handles zero-leak memory management inside ScreenCaptureManager:

suspend fun captureSingleFrame(): Bitmap? = withContext(Dispatchers.Default) {
    val image = imageReader?.acquireLatestImage() ?: return@withContext null
    try {
        val planes = image.planes
        val buffer = planes[0].buffer
        val pixelStride = planes[0].pixelStride
        val rowStride = planes[0].rowStride
        val rowPadding = rowStride - pixelStride * screenWidth

        val bitmap = Bitmap.createBitmap(
            screenWidth + rowPadding / pixelStride,
            screenHeight,
            Bitmap.Config.ARGB_8888
        )
        bitmap.copyPixelsFromBuffer(buffer)
        Bitmap.createBitmap(bitmap, 0, 0, screenWidth, screenHeight)
    } finally {
        // CRITICAL: Always close the image buffer immediately to return it to VirtualDisplay
        image.close()
    }
}

🎯 4. In-Place Overlay & Spatial Coordinate Mapping
To render translated text boxes directly over original text without distorting the layout:
ALST creates a dynamic WindowManager view of type TYPE_APPLICATION_OVERLAY.
Coordinates returned from OCR/Gemini are scaled against physical screen density (DisplayMetrics) and system navigation/notch insets.
Using Jetpack Compose Canvas, dark translucent glassmorphic cards with rounded corners are drawn exactly over original bounding boxes, placing high-contrast translated text right where your eyes expect it.
📱 5. Deep System Integration
ALST integrates directly into Android system controls for maximum convenience:
Draggable Floating Action Button (FAB): A frosted overlay button with magnetic screen-edge snapping.
Quick Settings Tile (TileService): Allows users to trigger screen translation directly from the Android status bar pull-down menu.
Android 14/15 Compliance: Runs via a Foreground Service registered with foregroundServiceType="mediaProjection" and handles runtime permissions securely via a translucent trampoline activity.
🔒 6. Privacy First
ALST operates strictly on a Bring Your Own Key (BYOK) model:
No intermediary proxy servers.
No telemetry or user tracking.
API keys stay stored strictly inside local sandboxed DataStore storage.
Captured screen frames exist purely in volatile RAM during processing and are immediately garbage-collected.
📦 Source Code & Links
ALST is 100% free and open-source under the MIT License.

⭐️ GitHub Repository: https://github.com/navidseyedain/ALSTMobile
🚀 Latest Release (APK): https://github.com/navidseyedain/ALSTMobile/releases/tag/v1.0.0

If you find the architecture interesting or useful, feel free to drop a star ⭐️ on the GitHub repository or open an issue for feature requests and discussions!

[Boost]

navidseyedain — Sat, 29 Aug 2026 12:53:19 +0000

How I Built ALAD Mobile: Real-Time AI Audio Dubbing for Any Android App

navidseyedain — Tue, 11 Aug 2026 01:07:52 +0000

Have you ever wanted to watch a foreign YouTube video, a live stream on Twitch, or a show on Netflix without constantly staring at subtitles?

To solve this, I built ALAD Mobile (Live Audio Dubbing) — an open-source native Android application that captures internal system audio in real-time, streams it through Google's Gemini Multimodal Live API, and instantly plays back the translated voice over any active media.

💡 What It Does

ALAD Mobile creates a system-wide floating overlay widget that sits comfortably over your active applications. Once activated:

🎧 Intercepts Audio: It captures the internal playback audio of any app playing media.
⚡ Live Translation: Streams audio chunks to Gemini's bidirectional Multimodal Live API via WebSockets.
🎙️ Real-Time Playback: Plays back the dubbed voice in real time in your selected language (supporting 78 languages).
📱 Seamless Overlay: Lets you toggle dubbing or switch target languages without ever leaving YouTube, Netflix, or Spotify.

🛠️ Tech Stack & Architecture

Building low-latency real-time voice translation on mobile required a carefully structured stack:

Language: 100% Kotlin
UI Framework: Jetpack Compose (Material 3)
Async Programming: Kotlin Coroutines, StateFlow, and SharedFlow
Networking & Streaming: WebSockets for low-latency bidirectional audio streaming
Android Audio APIs: MediaProjection / AudioRecord for internal audio capture + AudioTrack for low-latency playback
System Service: Floating Overlay Window using TYPE_APPLICATION_OVERLAY
AI Engine: Google Gemini Multimodal Live API

🧠 How It Works Under the Hood

1. Internal Audio Capture

Using Android's MediaProjection API and AudioRecord, the app captures raw PCM audio streams from the device's internal sound engine without relying on external microphone input.

2. Bidirectional Live WebSockets

Traditional REST API requests introduce too much latency for live translation. ALAD Mobile opens a persistent WebSocket connection directly to Gemini's Live API, continuously streaming PCM audio chunks in real-time.

3. Audio Track Playback

As Gemini processes and translates the stream, incoming audio payloads are decoded and pushed directly into a low-latency AudioTrack buffer for smooth, near-instant voice playback.

4. Floating System Overlay

To ensure zero friction, a Floating Control Widget built with Android's System Alert Window lets users control dubbing and switch target languages seamlessly over any running app.

🚀 Technical Challenges & Lessons Learned

Minimizing Latency: Balancing buffer sizes between raw PCM capture and WebSocket transmission frames was crucial for keeping speech output synchronized with active playback.
State Management: Handling connection lifecycle events, audio buffer underruns, and background service states seamlessly using Kotlin StateFlow.

🌐 Open Source & How to Try It

ALAD Mobile is 100% Free and Open Source. It runs using a free API key directly from Google AI Studio — no subscriptions or user accounts required.

🔗 GitHub Repository: navidseyedain/ALAD-Mobile

If you find this project interesting, feel free to drop a ⭐ star on GitHub, submit a PR, or leave your architectural feedback below!

Building ALAD: Real-Time AI Video Audio Dubbing in Chrome with Gemini API & WebSockets

navidseyedain — Wed, 05 Aug 2026 22:05:14 +0000

Building ALAD: Real-Time AI Video Audio Dubbing in Chrome with Gemini API & WebSockets
Imagine watching a video, live stream, or online course in a language you don't speak, and hearing it dubbed in real-time into your native language—right inside your browser, with almost zero latency.

That is exactly why I built ALAD (Live Audio Dubbing / LAD), an open-source Chrome Extension that leverages Google’s Gemini API and WebSockets to capture live tab audio, translate it on the fly, and play back natural dubbed audio alongside the original video.

In this article, I’ll walk through why I created ALAD, how its architecture works under the hood, the technical hurdles of handling real-time audio in Chrome Manifest V3, and how you can try or contribute to the project!

💡 The Problem: Why Real-Time Dubbing?
Subtitles are great, but they require your constant visual attention. If you are watching a technical tutorial, a lecture, or a fast-paced presentation in a foreign language, scanning subtitles while trying to focus on code or visual slides can be exhausting.

Existing dubbing solutions usually fall into two categories:

Offline post-processing tools: High quality, but slow and impossible for live or streaming video.

Heavy cloud SaaS platforms: Expensive, subscription-based, and locked behind proprietary players.

I wanted a solution that was lightweight, open-source, privacy-friendly, and integrated directly into the browser.

🚀 Introducing ALAD (Live Audio Dubbing)
ALAD connects your active browser tab directly to Gemini's real-time multimodal capabilities using WebSockets.

Key Features:
🎙️ Real-Time Tab Audio Capture: Captures clear raw audio directly from YouTube, Udemy, Twitch, or any HTML5 video player without capturing room ambient noise.

⚡ Low-Latency Streaming: Streams audio chunks over WebSockets for rapid response and minimal delay between visual speaker cues and audio output.

🤖 Powered by Gemini API: Uses Gemini's advanced multimodal audio understanding and natural speech generation.

🎚️ Smart Audio Ducking & Volume Balancing: Lowers the original video audio slightly while playing the dubbed translation so you can comfortably hear both or focus solely on the translated audio.

⚙️ Manifest V3 Compliant: Built strictly for modern Chrome extension security and performance standards.

🔑 BYO Key (Bring Your Own API Key): Keeps your requests private and cost-effective—your API key is stored locally in your browser.

🛠️ Tech Stack & Architecture
Here is how data flows through ALAD in real time:

[ Active Tab (HTML5 Video) ]
│
▼ (tabCapture API / Web Audio API)
[ Audio Processing & PCM Chunking ]
│
▼ (WebSocket Stream)
[ ALAD Background Engine ]
│
▼ (Gemini API Multimodal Endpoint)
[ AI Translation & Audio Synthesis ]
│
▼ (AudioBuffer Queue / Playback)
[ Dubbed Audio Streamed to User ]
Main Technologies Used:
Frontend / Popup: HTML5, CSS3, Modern JavaScript (ESNext)

Extension Platform: Chrome Extension API Manifest V3 (Service Workers, Offscreen Documents, tabCapture)

Audio Engineering: Web Audio API (AudioContext, PCM 16-bit encoding)

Real-Time Transport: WebSockets for low-latency full-duplex binary audio streaming

AI Engine: Google Gemini API (Multimodal Live API)

🔬 Under the Hood: Key Technical Challenges
Developing a real-time audio extension on Manifest V3 came with a few interesting technical challenges:

Tab Audio Capture in Manifest V3 In Manifest V3, background scripts run inside transient Service Workers, which do not have access to the DOM or AudioContext.

To solve this, ALAD utilizes Chrome’s Offscreen API (chrome.offscreen). When audio capturing starts:

The background service worker spawns an offscreen document.

The offscreen document calls chrome.tabCapture.getMediaStreamId() to capture the active tab's media stream.

Raw audio is piped into a MediaStreamAudioSourceNode inside an AudioContext.

...JavaScript
// Offscreen script snippet for tab audio capturing
const stream = await navigator.mediaDevices.getUserMedia({
audio: {
mandatory: {
chromeMediaSource: 'tab',
chromeMediaSourceId: streamId
}
}
});

const audioContext = new AudioContext({ sampleRate: 16000 });
const source = audioContext.createMediaStreamSource(stream);
// Process and chunk PCM audio data...

...

PCM Audio Chunking & WebSocket Encoding For Gemini to process live audio seamlessly, audio must be downsampled (typically to 16kHz mono) and chunked into base64-encoded PCM frames before being pushed through the WebSocket connection.

We process incoming audio buffer arrays, convert floating-point samples to 16-bit signed integers (PCM16), and dispatch small binary frames continuously:

JavaScript
function convertFloat32ToPCM16(buffer) {
let l = buffer.length;
let buf = new Int16Array(l);
while (l--) {
buf[l] = Math.min(1, Math.max(-1, buffer[l])) * 0x7FFF;
}
return buf.buffer;
}

Jitter-Free Audio Queue & Playback Synchronization Receiving streamed audio chunks over a network means packet arrival times can fluctuate. If played back immediately upon receipt, the synthesized voice would sound choppy.

ALAD implements a custom Audio Queue Scheduler. Received PCM chunks are decoded into AudioBuffer objects and scheduled sequentially on the timeline (audioContext.currentTime), creating continuous, stutter-free real-time dubbing.

📦 How to Install and Run ALAD Locally
You can test ALAD right now on your machine in just a few minutes:

Prerequisites:
Google Chrome (or any Chromium-based browser like Brave or Edge)

A Gemini API key (obtainable for free from Google AI Studio)

Installation Steps:
Clone the repository:

Bash
git clone https://github.com/navidseyedain/ALAD.git
cd ALAD
Load into Chrome:

Open Chrome and navigate to chrome://extensions/

Enable Developer mode (toggle in the top right corner).

Click Load unpacked.

Select the ALAD project directory.

Configure & Start Dubbing:

Click the ALAD icon in your Chrome toolbar.

Paste your Gemini API key in the settings tab.

Select your target language.

Open any video on YouTube, click Start Live Dubbing, and enjoy!

🛣️ What's Next? (Roadmap)
ALAD is an active open-source project, and there are several exciting features planned:

[ ] Multi-speaker Detection: Identifying different voices in the video and assigning distinct synthetic voices.

[ ] Offline Cache & Subtitle Overlay: Generating dual synchronized subtitles alongside audio dubbing.

[ ] Chrome Web Store Publishing: Packaging and releasing ALAD to the official Chrome Web Store for one-click installation.

[ ] Custom Voice Parameters: Pitch, speed, and emotion adjustments for dubbed voices.

🤝 Open Source & Contributions
ALAD is completely open-source and released under the MIT License. Contributions are super welcome! Whether you want to fix a bug, improve audio synchronization, refine the UI, or add support for new languages, feel free to open an issue or pull request.

🌐 GitHub Repository: https://github.com/navidseyedain/ALAD

👤 Author: Navid Seyedain (@navidseyedain)

If you find this project useful or interesting, please consider dropping a ⭐ Star on the repository to support development!

Thank you for reading! Feel free to leave your thoughts, feedback, or questions in the comments below.

Gemini SEO - Parallel Multi-Agent SEO & GEO Engine

navidseyedain — Mon, 27 Jul 2026 17:39:41 +0000

Search Engine Optimization (SEO) is evolving rapidly into Generative Engine Optimization (GEO). To help developers and webmasters optimize their content for modern AI search engines, I built Gemini SEO.

🚀 What is Gemini SEO?

Gemini SEO is a high-performance, parallel multi-agent engine built with TypeScript and powered by Google's Gemini API. It automates deep technical SEO and GEO audits using specialized autonomous AI agents running concurrently.

✨ Key Features

⚡ Parallel Multi-Agent Execution: Runs multiple AI agents in parallel for fast, multi-dimensional analysis.
🤖 Generative Engine Optimization (GEO): Prepares your content not just for traditional search engines, but for modern AI search platforms.
🛡️ Built with TypeScript: Fully typed, scalable, and maintainable codebase.
🎯 Actionable Insights: Generates structured reports and actionable optimization suggestions.

🛠️ Getting Started

You can easily set up and run the project locally:


bash
# Clone the repository
git clone [https://github.com/navidseyedain/gemini-seo.git](https://github.com/navidseyedain/gemini-seo.git)

# Navigate to directory
cd gemini-seo

# Install dependencies
npm install

Make sure to set your Gemini API key in your environment variables before running the engine.
🔗 Open Source & Contribution
The project is completely open-source under the MIT License. I would love to get feedback from the community!
GitHub Repository: navidseyedain/gemini-seo
⭐ If you find this project useful, please consider giving it a star on GitHub! It helps the project reach more developers.
Feel free to leave your thoughts, questions, or feature requests in the comments below!