Last October, someone posted "Ask HN: Real-time speech-to-speech translation?" on Hacker News. 158 points, 70 comments. They wanted a Babelfish, something that listens, translates, and speaks, all offline. Every alternative people suggested (RTranslator, 3PO, Samsung Interpreter) failed on at least one axis: too awkward, requires cloud, not open source.
I decided to build one for English and Mandarin.
What it does
You speak English, it says the same thing in Mandarin. You speak Mandarin, it says it in English. Everything runs on your machine.
npm install -g live-translate
live-translate start
live-translate
# Hold SPACE, speak, release, hear translation
No API keys. No cloud. Works in airplane mode. Works behind the Great Firewall.
Why I built this
The existing options all have the same problem.
Google Translate sends your audio to Google's servers, is blocked in China, and renders "我请你吃饭" as "I invite you to eat rice" instead of "I'll treat you to dinner."
Microsoft Translator is not much better. A paying Azure customer publicly filed a ticket saying Mandarin output is "literal or word-for-word" while Spanish and Portuguese work fine.
DeepL has no voice mode at all for Chinese, and academic testing found it mishandles basic Chinese grammar.
ChatGPT Voice has excellent quality when it works, but it forgets it's supposed to be translating and starts answering your questions instead.
I wanted something private (my conversations with my family are not Google's business), that worked offline (I travel to places with no signal), and that was open source so I could inspect and fix it.
The architecture
Three ML models chained together:
Microphone
|
Whisper (speech to text + language detection)
|
Ollama + Qwen 2.5 (text to translated text)
|
Piper TTS (translated text to speech)
|
Speaker
Whisper handles speech recognition and language detection in one step. It tells me "this is English" or "this is Mandarin" so the routing is free.
Qwen 2.5 via Ollama handles translation. I started with Opus-MT (Helsinki-NLP's lightweight model) but the quality was bad. It would repeat "你好 你好 你好" for a simple "Hello." Qwen 2.5 at 7B parameters produces natural, idiomatic output.
Piper TTS handles speech synthesis. It's fast, MIT licensed, and has decent voices for both English and Chinese.
All three run locally. No data leaves your machine. The only network call is pulling the Docker containers on first install.
One thing I learned the hard way
Translation quality is dramatically worse in one direction. English to Chinese is decent across most tools. Chinese to English, the direction where a Mandarin speaker needs to be understood, is significantly worse. A clinical study tested live spoken translation and found that English to Chinese scored 62-76% acceptable while Chinese to English scored only 36-41%.
This matters because a live conversation needs both directions to work. Most translation demos only show the easy direction. I benchmark both.
The Whisper hallucination problem
Whisper has a known issue. During silence or background noise, it hallucinates phrases from its training data. The most common ones are "Translated by Amara.org Community" and "Transcribed by Otter.ai", because Whisper was trained on YouTube subtitle files that contained these credit lines.
Common hallucinations: "Translated by Amara.org Community" and "Transcribed by Otter.ai"
I added filtering for known hallucination strings and surface Whisper's confidence score to the user. If confidence is low, you see a warning instead of garbage output.
It's also an MCP
If you use Claude Code or Claude Desktop, you can use the same engine as an MCP server:
npx @anthropic-ai/claude-code mcp add live-translate-mcp
Give Claude an audio file and it'll transcribe, translate, and save the translated audio, all locally.
waxberry-dev
/
live-translate-mcp
MCP server for local speech translation (EN ↔ 中文) via Whisper + Claude + Piper
live-translate-mcp
Real-time English ↔ Mandarin speech translation for Claude — powered by Whisper, Claude AI, and Piper TTS.
Give Claude the ability to listen, translate, and speak. live-translate-mcp is a Model Context Protocol (MCP) server that adds speech translation as a native tool inside Claude Desktop and Claude Code. Hand it an audio file, and it transcribes, translates, synthesises, and plays the result — entirely on your machine, with Claude handling the translation.
What it does
Step
Technology
Where it runs
Speech → Text
OpenAI Whisper (via @huggingface/transformers)
Local
Text → Translation
Claude (Opus 4.8)
Anthropic API
Translation → Speech
Piper TTS (ONNX)
Local
Audio never leaves your machine except for the translated text sent to the Claude API. ASR and TTS run fully on-device.
Tools
translate_file
Translate a WAV audio file. Pass an absolute path — the server transcribes it, translates the text via Claude, synthesises speech, saves…
Try it
npm install -g live-translate
live-translate doctor # check prerequisites
live-translate start # start the translation backend
live-translate # hold SPACE to speak
Run Click here for prerequisites
live-translate doctor to verify your setup.
The whole thing is MIT licensed. PRs welcome, especially if you speak Cantonese. That's the most-requested language to add and Whisper already supports it.
waxberry-dev
/
live-translate
Fully local English ↔ Mandarin speech translator. Open source, offline, privacy-first. CLI + MCP server.
live-translate — English ↔ 中文
Fully local speech-to-speech translator between English and Mandarin Chinese No cloud APIs required. All audio processing runs on your machine. MIT licensed.
Install
npm install -g live-translate
Quick Start
live-translate # press SPACE to record, SPACE again to translate
# downloads models and starts services automatically on first run
live-translate stop # stop services when done
Commands
Command
Description
live-translate
Start translating (SPACE to start/stop recording, Q or Ctrl-C to quit)
live-translate config
Configure your translation provider
live-translate start
Download models and start backend services (runs automatically when needed)
live-translate stop
Stop all backend services
live-translate status
Show service health and active provider
live-translate doctor
Check prerequisites
Translation Providers
Run live-translate config to choose a backend. The default is Opus-MT (fully local, no API key needed).
Provider
Type
Quality
Cost
Opus-MT (default)
Local model
Good
Free
Ollama (Qwen 2.5)
Local LLM
High
Free — needs
Website: waxberry.dev
If you found this useful, a star on GitHub means a lot. And if you try it, tell me how the translation quality is for you, especially Mandarin to English. That's the direction I'm working hardest to improve.
Top comments (1)
I’m really excited to get Cantonese support added! Since Whisper already handles the transcription, the main hurdle is fine-tuning the Piper TTS voice to sound natural. If you speak Cantonese and are down to run a few test strings through the engine, please drop a comment or open an issue on the repo! I'd love to know if the current models handle Yue Chinese grammar correctly.