Getting Started with a Real‑Time Voice Translation App
Building a real‑time voice translation app feels like science‑fiction, but with today’s cloud APIs it’s a very achievable side‑project. In this article we’ll walk through the whole stack: capturing audio, transcribing it, translating the text, and finally turning the translated text back into speech. We’ll keep the example language‑agnostic (Python on the backend, JavaScript on the frontend) and we’ll lean on a few battle‑tested services. The star of the show is ElevenLabs for its high‑quality text‑to‑speech (TTS) and voice‑cloning capabilities. You can start experimenting right away with their free tier at https://try.elevenlabs.io/kr07zfuqn1bp.
1. Why Real‑Time Matters
If you’re building a language‑learning app, a travel chatbot, or a customer‑support tool, latency is everything. A 2‑second delay feels natural; a 5‑second lag is noticeable. That means we need:
- Fast speech‑to‑text (STT) – ideally < 1 s per spoken sentence.
- Quick translation – most cloud translation APIs return within 200 – 400 ms.
- Low‑latency TTS – the synthesized voice should be ready in < 1 s.
The rest of this guide assumes you already have an API key for a STT provider (Google Cloud Speech‑to‑Text, Azure Speech, or Whisper on your own server) and a translation service (Google Translate, DeepL, or Microsoft Translator). We’ll plug in ElevenLabs for the TTS step.
2. High‑Level Architecture
+----------------+ +-------------------+ +-----------------+
| Browser (JS) | <--->| Node.js/Flask API| <--->| ElevenLabs TTS |
+----------------+ +-------------------+ +-----------------+
^ ^ ^
| | |
| WebSocket (audio) | REST (transcribe/translate)
| | |
| | |
Mic input <--- STT API <--- Translation API <--- TTS Engine
- Front‑end: captures mic audio via the Web Audio API, streams to the backend over a WebSocket.
- Backend: receives audio chunks, forwards them to the STT service, then to the translation service, and finally to ElevenLabs for synthesis.
- ElevenLabs: returns an MP3 or raw PCM stream that the browser plays back.
3. Front‑End: Capturing and Streaming Audio
Below is a minimal JavaScript snippet that records mic input and streams it to a server using WebSocket. We’re using the MediaRecorder API for simplicity.
const socket = new WebSocket('wss://your-server.com/stream');
const mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm' });
mediaRecorder.ondataavailable = (e) => {
// Send each chunk as binary data
socket.send(e.data);
};
socket.addEventListener('message', (event) => {
// event.data is the TTS audio blob from ElevenLabs
const audio = new Audio(URL.createObjectURL(event.data));
audio.play();
});
mediaRecorder.start(250); // send 250ms chunks
What to watch for
- Chunk size: smaller chunks give lower latency but increase overhead. 250 ms is a good compromise.
-
Browser support: most modern browsers support
MediaRecorder. For older browsers you may need a polyfill or a server‑side WebRTC solution.
4. Back‑End: Orchestrating the Pipeline
We’ll use Python + FastAPI for the backend because of its async support and excellent typing. The core flow is:
- Receive audio chunk
- Append to an in‑memory buffer
- When the buffer reaches a threshold (or after a silence detection), send to the STT API
- Translate the resulting text
- Send the translated text to ElevenLabs and stream the returned audio back to the client
# main.py
from fastapi import FastAPI, WebSocket
import aiohttp
import asyncio
import json
app = FastAPI()
# Replace with your real keys
STT_ENDPOINT = "https://api.speech-to-text.com/v1/recognize"
TRANSLATE_ENDPOINT = "https://api.translate.com/v1/translate"
ELEVENLABS_API = "https://api.elevenlabs.io/v1/text-to-speech"
ELEVENLABS_API_KEY = "YOUR_ELEVENLABS_KEY"
async def transcribe_audio(audio_bytes: bytes) -> str:
async with aiohttp.ClientSession() as session:
async with session.post(
STT_ENDPOINT,
headers={"Authorization": f"Bearer {STT_ENDPOINT}"},
data=audio_bytes,
) as resp:
result = await resp.json()
return result["text"]
async def translate_text(text: str, target_lang: str = "es") -> str:
async with aiohttp.ClientSession() as session:
payload = {"q": text, "target": target_lang}
async with session.post(
TRANSLATE_ENDPOINT,
json=payload,
headers={"Authorization": f"Bearer {TRANSLATE_ENDPOINT}"},
) as resp:
result = await resp.json()
return result["translatedText"]
async def synthesize_speech(text: str) -> bytes:
async with aiohttp.ClientSession() as session:
payload = {
"text": text,
"voice_settings": {"pitch": 0, "speed": 1}
}
async with session.post(
ELEVENLABS_API,
json=payload,
headers={
"xi-api-key": ELEVENLABS_API_KEY,
"Content-Type": "application/json",
},
) as resp:
# ElevenLabs streams MP3
return await resp.content.read()
@app.websocket("/stream")
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
buffer = bytearray()
while True:
data = await websocket.receive_bytes()
buffer.extend(data)
# Simple logic: every 3 seconds of audio we process
if len(buffer) > 300_000: # roughly 3 s of 96 kHz audio
text = await transcribe_audio(buffer)
translated = await translate_text(text, target_lang="de")
audio = await synthesize_speech(translated)
await websocket.send_bytes(audio)
buffer.clear()
Why this works
- Async I/O: All calls to STT, translation, and ElevenLabs run concurrently, keeping the WebSocket alive.
- Buffering strategy: We keep a sliding window of audio. A more sophisticated version would use VAD (voice activity detection) to detect pauses and trigger processing only when the user stops speaking.
5. ElevenLabs: Why It’s the TTS of Choice
ElevenLabs offers:
- Human‑like voices – up to 30 + voices with fine‑grained control over pitch, speed, and timbre.
- Voice cloning – upload a few minutes of audio and clone a custom voice. Perfect for branded assistants.
- Low latency – the API returns a short MP3 stream in < 1 s.
Tip: The free tier gives you a generous amount of usage per month. If you’re building a prototype, you’ll be fine. When you go live, just swap the API key in the
ELEVENLABS_API_KEYvariable.
6. Deployment Checklist
| Step | What to do | Notes |
|---|---|---|
| 1 | Deploy the FastAPI app to a platform that supports WebSockets (e.g., Fly.io, Render, Railway). | Make sure you set CORS_ALLOW_ORIGIN to your domain. |
| 2 | Secure your API keys. Store them in environment variables or a secrets manager. | Don’t hard‑code them in the repo. |
| 3 | Enable HTTPS on the front‑end. Browsers block microphone access on insecure origins. | Let’s Encrypt is free. |
| 4 | Test latency. Use chrome://inspect or websocket.org to measure round‑trip time. |
Aim for < 500 ms end‑to‑end. |
| 5 | Optimize VAD. If you notice unnecessary STT calls, add a simple VAD library like webrtcvad. |
Reduces costs and latency. |
7. Going Beyond the Basics
-
Multilingual support – add a language picker that changes the
target_langparameter. - Batch translation – for long monologues, chunk the text and stream partial TTS back to the user.
- Voice cloning – let users upload a sample of their voice and use ElevenLabs to clone it for personalized TTS.
- Edge caching – cache translated snippets so repeated phrases are served instantly.
8. Final Thoughts
With a handful of lines of code and a few paid services, you can turn a microphone into a real‑time, multilingual chatbot. The key is to keep the data path tight: mic → STT → translation → TTS → speaker. ElevenLabs makes the last hop the smoothest, delivering natural‑sounding voices that feel like a real person.
Ready to bring your voice translation idea to life? Sign up for ElevenLabs today and unlock high‑quality TTS and voice‑cloning for your app.
Try ElevenLabs now: https://try.elevenlabs.io/kr07zfuqn1bp
Happy coding!
Top comments (0)