I built Parley, a desktop live interpreter, because I join a lot of international meetings and my English isn't great. This post is about the audio plumbing, which turned out to be most of the work. No code to sell you here, just what I learned.
The constraint that shapes everything
I wanted spoken translation in both directions during a normal call, without asking anyone else to install anything, and without writing a plugin for each meeting platform.
The only interface every meeting app shares is: it picks a microphone and a speaker. So the interpreter has to be a microphone and a speaker, and sit between the user's real devices and the meeting:
Outgoing: physical mic → capture → cloud translation → translated speech → virtual mic → meeting app
Incoming: meeting app → capture → cloud translation → translated speech → physical headphones
That simple picture hides most of the hard parts.
1. Getting audio into the meeting
The meeting app needs a microphone it can select that carries the translated speech.
- macOS. The app installs a virtual audio driver that exposes two devices: ParleyVirtualMic, which the meeting uses as its microphone, and ParleyMeetingBus, which the meeting uses as its speaker so Parley can capture what the other side says. The user's real mic and headphones are selected inside Parley.
- Windows 11. The installer bundles the signed VB-CABLE driver. Translated speech goes into the cable; the meeting app selects CABLE Output as its microphone.
Installing an audio driver is the most intimidating step for non-technical users, so I spent a lot of effort on doing it automatically and explaining it in plain language. On Windows the installer now discovers the endpoints itself, with no driver downloads or device IDs to type.
2. Getting audio out of the meeting, and only the meeting
For the listening direction, you want the other participants' voices and nothing else: not your music, not a YouTube tab, not notification sounds.
- On macOS, routing the meeting's speaker to ParleyMeetingBus gives Parley exactly the meeting's output.
- On Windows, Parley captures the meeting app's process directly, so the user picks "Zoom" (or whichever app) as the listening source instead of all system audio.
A privacy note I put in the docs because it matters: if you capture system audio rather than one process, you may capture more than the conversation you meant to translate.
3. Feedback loops are the number-one failure
If translated speech plays through laptop speakers, the mic hears it, translates it again, and you get an echo chamber in two languages. Rules I ended up enforcing in the UX:
- Headphones are strongly recommended in meeting mode, and the first-run flow says so.
- The meeting only ever hears the virtual mic, never the physical mic directly.
- In in-person (venue) mode, where translated speech does go to room speakers, the setup guide tells you to keep the microphone away from them.
4. Streaming and the latency/stability trade-off
Capture is resampled to 16 kHz mono PCM and streamed to the realtime model in ~100 ms chunks. The models are the part I don't build: Parley is powered by state-of-the-art large language models, and today the managed options are Qwen (60 input / 29 spoken output languages, Cantonese input-only) and Doubao (Chinese ↔ English), both with live voice cloning. I don't wait for the user to finish a sentence; the model's own turn detection decides when to speak. Translated audio is decoded and queued for playback as soon as each delta arrives.
The interesting decision was the playback buffer. A short pre-buffer lowers delay but stutters when the network hiccups or the model emits short fragments. I experimented with ~200 ms and went back to about one second, because choppy translated speech is worse for comprehension than a slightly later start. Most of the delay people feel is still the model's: Qwen reports about 2.3 s LAAL and Doubao about 2.53 s speech output latency in their own evaluations (different metrics, not directly comparable).
5. Devices change mid-call
Bluetooth headphones disconnect, sample rates change when you plug in a USB mic, the OS switches default devices. Each of those can silently kill one direction. The audio layer (built on cubeb, in Rust) has to notice, reopen the device, and keep the session alive, while the UI (Slint) tells the user what happened instead of just going quiet.
6. Letting you hear your own translation
Once the outgoing direction worked, the obvious question from my own calls was: what did they actually hear? So the meeting bar has a translated voice monitor. It plays the model's translated output of what you said into your own headphones. It never plays your raw microphone, and it doesn't open a second model session.
A few rules kept it from breaking anything else:
- It's an optional branch at the point where translated speech leaves the model. The full stream still goes to the virtual mic; the monitor copy goes through its own bounded queue, and if that queue backs up it drops its own stale audio rather than slowing the meeting mic.
- It's mixed into the one output stream Parley already has open to your headphones. No ducking, no automatic volume changes to the meeting audio, and a soft knee near full scale instead of hard clipping.
- The volume control (0–100%, default 15%, on a perceptual curve) changes only the monitor's gain. It never touches the system/device volume, the meeting audio, or what's sent to the meeting. Gain changes ramp over a few milliseconds to avoid clicks, and off or 0% is bit-exact bypass.
- Gating follows the translated samples, not your mic, so a translation that arrives after you stop talking isn't cut off.
- It's still not echo cancellation: if you use speakers, the mic can pick them up. Headphones remain the rule.
7. One input, many outputs
The venue mode was a nice generalisation: one speaker, several languages at once, each language sent to its own physical speaker or headphone output with its own playback queue (a single output works too). It's unlocked by a one-time $149 venue licence, then billed by usage like everything else. Captions are grouped by language in the history, and SRT export covers the whole session.
8. Billing that matches what users expect
People don't think in "streams". They think "my meeting lasted 30 minutes". So a two-way meeting, or a venue session with three languages, is one session charge: billing counts the connected time once (by the millisecond, from when the connection is ready, silence included), not once per direction or output. Short sessions aren't rounded up to a full minute. There's no software to buy and no subscription for meeting translation; you pay per minute from a wallet. It complicated the server side, but it makes the price explainable in one sentence.
What I'd tell anyone building something similar
- Design the first-run audio setup before you design anything else.
- Treat feedback prevention as a product feature, not a bug fix.
- Pick buffer sizes for intelligibility, not benchmark latency.
- Expect device changes constantly and make recovery visible.
- Price in the unit users think in.
Disclosure: I'm the founder of Parley. Parley runs on macOS 14.5+ and Windows 11 x64.
Top comments (0)