DEV Community

VoiceDeveloper
VoiceDeveloper

Posted on

Build a Real-Time Voice Translation App

#webdev #tutorial #javascript #ai

Why Build a Real‑Time Voice Translation App?

You’ve probably heard about those “real‑time” translation tools that let you talk to someone who speaks another language and instantly understand them. The tech stack behind that is a fascinating blend of speech recognition, machine translation, and speech synthesis. If you’re a developer who loves working with AI and wants to add voice to your next product, building a real‑time voice translation app is a great way to get hands‑on with cutting‑edge tools and see instant results.

In this article we’ll walk through a minimal, end‑to‑end architecture, show you how to hook everything together with Python, and highlight where ElevenLabs comes in as a powerful TTS and voice‑cloning partner.

High‑Level Architecture 

User Device  ──►  STT (Speech → Text)  ──►  MT (Text → Text)  ──►  TTS (Text → Speech)  ──►  User Device
Enter fullscreen mode Exit fullscreen mode
  1. Speech‑to‑Text (STT) – Capture audio from the microphone and transcribe it in real time.
  2. Machine Translation (MT) – Translate the transcribed text from the source language to the target language.
  3. Text‑to‑Speech (TTS) – Convert the translated text back into spoken audio.
  4. Voice Cloning (optional) – Make the output sound like a specific speaker, e.g., your own voice.

We’ll use the following open‑source and cloud services:

Component Suggested Tool
STT vosk (offline) or Google Speech‑to‑Text (cloud)
MT OpenAI’s GPT‑4o (or other LLM) or Google Translate API
TTS ElevenLabs (high‑quality voices, voice cloning)
Voice Cloning ElevenLabs voice cloning API

1. Capture Live Audio

The simplest way to stream microphone data in Python is with pyaudio. Below is a minimal example that reads chunks of audio and sends them to a WebSocket or HTTP endpoint for STT.

import pyaudio
import wave
import requests
import json

CHUNK = 1024
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000  # 16kHz is a good balance for STT
DEVICE_INDEX = None  # use default mic

p = pyaudio.PyAudio()
stream = p.open(format=FORMAT,
                channels=CHANNELS,
                rate=RATE,
                input=True,
                frames_per_buffer=CHUNK,
                input_device_index=DEVICE_INDEX)

print("Listening… press Ctrl+C to stop")

try:
    while True:
        data = stream.read(CHUNK)
        # Send raw PCM to STT endpoint
        response = requests.post(
            "https://api.stt.example.com/v1/recognize",
            headers={"Content-Type": "audio/l16; rate=16000"},
            data=data,
        )
        transcript = response.json()["transcript"]
        print("STT:", transcript)
except KeyboardInterrupt:
    pass

stream.stop_stream()
stream.close()
p.terminate()
Enter fullscreen mode Exit fullscreen mode

Tip: If you want to keep latency low, use a streaming STT API (e.g., Google Cloud Speech‑to‑Text) that returns partial results.

2. Translate the Text

Once you have the transcription, the next step is to translate it. If you’re already using OpenAI for LLM work, GPT‑4o can do translation in a single request:

import openai

openai.api_key = "sk-..."

def translate(text, source_lang="en", target_lang="es"):
    prompt = f"Translate the following {source_lang} text to {target_lang}:\n\n{text}"
    response = openai.ChatCompletion.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return response.choices[0].message.content.strip()

translated = translate(transcript, source_lang="en", target_lang="es")
print("MT:", translated)
Enter fullscreen mode Exit fullscreen mode

If you need a free or lower‑cost option, the Google Translate API or DeepL API works well too. Just make sure your translation latency stays under ~200 ms to keep the whole pipeline feeling real‑time.

3. Speak It Back – TTS with ElevenLabs

ElevenLabs offers extremely natural voices and a straightforward REST API. You can even clone a voice you like, which is perfect if you want the assistant to sound like you.

import requests

ELEVENLABS_API_KEY = "YOUR_ELEVENLABS_API_KEY"
ELEVENLABS_URL = "https://api.elevenlabs.io/v1/text-to-speech"

def synthesize(text, voice_id="EXAVITQu4vr4xnSDxMaL"):
    headers = {
        "accept": "audio/mpeg",
        "xi-api-key": ELEVENLABS_API_KEY,
    }
    payload = {
        "text": text,
        "voice_settings": {"stability": 0.5, "similarity_boost": 0.75},
    }
    response = requests.post(f"{ELEVENLABS_URL}/{voice_id}", headers=headers, json=payload)
    response.raise_for_status()
    return response.content  # raw MP3 bytes

audio_bytes = synthesize(translated)
with open("output.mp3", "wb") as f:
    f.write(audio_bytes)
print("Saved MP3")
Enter fullscreen mode Exit fullscreen mode

Voice Cloning

If you want the output to sound like a particular person, ElevenLabs lets you upload a few minutes of their speech and generate a new voice ID. The docs walk you through the upload and training steps, but the key call looks like this:

CLONE_URL = "https://api.elevenlabs.io/v1/voice-cloning"

payload = {"audio_url": "https://mycdn.com/samples/voice.mp3"}
response = requests.post(CLONE_URL, headers={"xi-api-key": ELEVENLABS_API_KEY}, json=payload)
new_voice_id = response.json()["voice_id"]
Enter fullscreen mode Exit fullscreen mode

You can then pass new_voice_id into the synthesize call above.

Why ElevenLabs?

  • Real‑time: Their API responds in <200 ms for short sentences.
  • Voice quality: Human‑like intonation and prosody.
  • Voice cloning: Perfect for personal assistants, brand voices, or accessibility.

If you’re ready to try ElevenLabs, you can sign up and get a free tier via this link: https://try.elevenlabs.io/kr07zfuqn1bp

4. Putting It All Together

Below is a very basic Flask app that ties everything together. It listens on /voice, receives raw audio, transcribes, translates, synthesizes, and streams the MP3 back.

from flask import Flask, request, Response
import requests, json
import openai

app = Flask(__name__)
ELEVENLABS_API_KEY = "YOUR_ELEVENLABS_API_KEY"
openai.api_key = "YOUR_OPENAI_KEY"

@app.route("/voice", methods=["POST"])
def voice_endpoint():
    audio = request.data  # raw PCM
    # 1. STT
    stt_resp = requests.post(
        "https://api.stt.example.com/v1/recognize",
        headers={"Content-Type": "audio/l16; rate=16000"},
        data=audio,
    )
    transcript = stt_resp.json()["transcript"]

    # 2. MT
    translated = translate(transcript, source_lang="en", target_lang="fr")

    # 3. TTS
    mp3 = synthesize(translated, voice_id="EXAVITQu4vr4xnSDxMaL")

    return Response(mp3, mimetype="audio/mpeg")

if __name__ == "__main__":
    app.run(port=5000)
Enter fullscreen mode Exit fullscreen mode

On the client side, you can stream microphone data to /voice via WebSocket or HTTP POST and play the returned MP3 with the browser’s <audio> tag.

5. Tips for Low Latency

Component Optimization
STT Use a local model (e.g., vosk) or a low‑latency cloud endpoint.
MT Cache frequent phrases; batch small requests if latency is high.
TTS Pre‑generate common responses; use ElevenLabs’ “streaming” endpoint if available.
Network Keep all services in the same region; use HTTP/2 or gRPC for lower overhead.

6. Next Steps

  1. Add speaker diarization so you can identify who is speaking in a multi‑person conversation.
  2. Persist the conversation in a database for later retrieval or training.
  3. Support multiple target languages and let users pick from a dropdown.
  4. Integrate a UI – a simple React component that shows live subtitles and plays the translated audio.

Call to Action

Ready to turn your idea into a voice‑powered reality? ElevenLabs gives you the industry‑leading TTS and voice‑cloning technology you need, with a generous free tier and easy integration. Sign up today and start building your own real‑time voice translation app:

Try ElevenLabs now: https://try.elevenlabs.io/kr07zfuqn1bp

Top comments (0)