At Trilyo we build a chat app with built-in translation. Our next update lets users call any phone number in the world, including landlines, with live AI translation during the call. The person who answers doesn't need an app. They just pick up the phone.
This isn't a code walkthrough. It's a high-level look at the problems that turn up when you put translation in the middle of a real phone call, and the tradeoffs we chose.
1. Turn-taking beats pretending to be simultaneous
Human interpreters at conferences can work simultaneously. Machine translation of speech, as a pipeline, needs enough of a sentence to translate meaning rather than individual words. Word order is different across languages, and a verb at the end of a sentence can change everything before it.
So we designed the experience around turns. One person speaks, the system detects the end of the utterance, the speech is translated, and the other person hears it in their language. Then the roles swap.
That brings its own sub-problems:
- End-of-turn detection. Wait too little and you cut people off in the middle of a thought. Wait too long and the call feels sluggish. People pause to think, so a moment of silence doesn't always mean they've finished.
- Overlap. On a phone, people talk over each other, say "uh-huh," or start answering early. The system has to decide what counts as a turn and what is just backchannel noise.
- Setting expectations. If users know the call goes turn by turn, they adapt: shorter sentences and letting the other side finish. A clear mental model helps more than squeezing out a little more speed.
2. Latency is a budget, not a single number
Every turn includes a short pause while translation happens. That pause is made up of several stages: capturing the end of speech, recognizing it, translating, producing the translated speech and playing it back over the line. Each stage has a speed-versus-quality tradeoff:
- Translating too early risks translating half a sentence badly.
- Faster speech synthesis can sound less natural.
- Some language pairs are harder than others, and the budget doesn't spread evenly across them.
We chose not to promise a speed we can't guarantee for every language and every network. We describe the call honestly, as turn by turn with a short pause, and keep working to make that pause shorter.
3. Telephone audio is not app audio
Calls between two app users can use wideband audio. The public phone network, and landlines especially, often carries narrowband audio: a limited frequency range, compression and sometimes line noise.
Speech recognition on that kind of audio is noticeably harder:
- Consonants that tell similar words apart can be lost.
- Background noise, speakerphones and old handsets make it worse.
- Accents and fast speech add more uncertainty.
The translated speech we send back goes through the same narrow channel, so it has to stay intelligible on an old landline handset, not just sound nice on studio headphones.
4. The callee never agreed to use your app
This is the part we found most interesting, and it isn't technical. In an app-to-app call, both people chose the product. In a call to a phone number, the person answering made no such choice. They don't know what Trilyo is, and they may not expect an AI to be on the call.
So every translated call starts with a short announcement to the person being called:
"Call from <caller name>, translated by AI."
This does several things:
- Transparency. The other person knows right away that the call is translated by AI.
- Identity. They know who is calling, which matters for trust, and especially for unknown numbers.
- Expectations. Once they know a translator is in the loop, the turn-by-turn rhythm feels natural rather than strange.
We think any product that puts AI between two people on a phone should be upfront about it. A short announcement is a small cost for that.
5. A synthetic voice, said plainly
The person on the other end hears a synthesized AI voice speaking the translation. It isn't a recreation of the caller's voice. We think that's the honest default: the meaning comes from the caller, and the voice clearly belongs to the translator.
6. Paying for what you use
Phone calls cost money in the real world, so translated calls use minutes that users buy in packages inside the app. There are no free minutes for this feature. A clear, prepaid model keeps it simple for users and keeps the feature sustainable.
What we learned
- Design the conversation model before you optimize the pipeline. Turn-taking with a clear mental model works better than half-simultaneous output that confuses people.
- Be honest about latency. Describe the experience as it is, and then improve it.
- Plan for the worst audio path, not the best one.
- Transparency is a feature. Tell the person on the other end what is happening.
Try it
Translated calls to any phone number are coming in Trilyo's next update, version 1.7.3, which is now in store review. Trilyo is available on the App Store. Learn more at trilyo.app.
Top comments (1)
Framing latency as a budget across capture, recognition, translation and playback is a useful way to look at it, because it shows which stage to optimise first. Telling users up front that the call runs turn by turn also seems wise, since a short pause then reads as part of the design and not as a fault. Did you find that the end-of-turn timeout needed to differ between languages?