There's a bug every voice agent team hits eventually. The agent asks a question, gets three words in, stops, and says "Sorry, go ahead." Nobody said anything. The agent heard itself.
In a browser demo you rarely see it. The browser's echo canceller is decent, you're wearing headphones, and the network is clean. On a real phone call it shows up a lot more, and it's one of the reasons an agent that felt great in testing starts "randomly" cutting itself off once it has a real number.
Two launches in early November 2026 went right at this layer, so it's worth walking through what's actually happening.
What shipped
Soniox STT v6 (Nov 3). The headline feature is that the speech model now knows what your agent is saying. You stream the agent's words into the same WebSocket as the caller's audio, and v6 uses them two ways: to recognise the caller's short replies better (it knows what question they are answering), and to filter out any of the agent's own voice that leaks back into the caller's audio. It keeps listening while the agent talks, so a real interruption still gets through. (Soniox launch post, agent text input docs)
Hume Audio Decision API (Nov 2). A different angle on the same gap: transcripts only carry words. Hume's API takes raw audio plus a question you write ("does the caller sound frustrated?", "are two people talking over each other?") and returns an answer with a confidence score. In an initial evaluation, Hume reports 70.7% agreement with a human panel's average judgment, compared with 56.5% for the individual-rater baseline. (Hume launch post)
And on the money side, Guava announced a $12.5M seed on Nov 2 (led by Acrew Capital) with a pitch literally about voice AI's "real-world growing pains". (BusinessWire)
The common thread: the models are fast enough now. The work is moving to what happens to audio between the model and a real person.
Why phone calls make self-interruption worse
On a call, your agent never hears the caller's room directly. It hears the caller's leg of the call: audio that went through their handset, their carrier, and whatever sits between there and your server. Your agent's own voice can come back on that leg in a few ways:
- Speakerphones and car kits. The caller's phone plays your agent out loud and its microphone picks it up again. Phone echo cancellers are good, not perfect, and cheap handsets and noisy cars push them hard.
- The network. Older parts of the phone network convert between two-wire and four-wire circuits, and that creates line echo. Carriers run network echo cancellers (the ITU-T G.168 kind) to remove it, and most of it goes. Some doesn't, especially right as speech starts.
- Narrowband audio. Most phone legs are 8 kHz G.711. Your STT is working with less information than in a browser, so a faint echo of the agent and a soft "uh-huh" from the caller look more alike.
Now add the usual voice agent loop: VAD or STT hears "speech" on the caller's leg while the agent is talking, the framework treats it as barge-in, playback stops, the agent apologizes for interrupting. The caller hears a bot that can't finish a sentence.
The tempting fix is to make barge-in less sensitive. That trades one bad experience for another: now the caller says "no, wait" and the agent keeps reading the whole policy.
What v6's echo filtering does is let you keep barge-in sensitive, because the STT can tell "this is the agent's own sentence coming back" from "this is the caller".
Soniox's docs are explicit that this sits on top of normal echo cancellation, not instead of it: "Keep your echo cancellation on. Agent text input handles what it leaves behind; it doesn't replace it." On a phone call that maps nicely: the handset and the network do their echo cancelling, v6 cleans up the remainder.
The part that's easy to get wrong: timing
Agent text messages don't carry timestamps. Soniox places each piece of text at the point in the audio stream where it arrives, so you're meant to send each piece about half a second to a second before the caller hears it. The docs say to follow playback, not the LLM and not the TTS, because both produce text and audio faster than real time.
In a browser app, the client can report playback position. On a phone call there's no client to ask. But there's a nice property here: on a phone call your server is the player. It writes audio frames into the carrier's media stream. So how much the caller has heard is roughly the smaller of two numbers: the time since the reply started, and the length of audio you've actually sent.
For 8 kHz mu-law audio, that length is simply the bytes sent divided by 8,000, because each sample is one byte. Taking the smaller number matters: if TTS stalls, the wall clock keeps running but the caller hasn't heard anything new, and you don't want the text to race ahead of the audio. (Soniox's docs suggest the same estimate for server-side agents.)
In practice that's a small pacer with one job per turn. After each chunk of caller audio (every 20 ms on most phone streams), it works out how far playback has got, adds a lead of about 0.7 seconds, and sends Soniox the agent's words whose TTS timestamps fall inside that window. When the TTS has finished and every word has gone out, it marks the end of the segment. It doesn't depend on any framework, so it fits wherever your agent writes audio to the phone leg.
Turning the feature on is one setting in the start request. For a raw phone stream you also declare the audio format: Soniox's own Twilio example uses mu-law at 8 kHz and sends both call tracks as two channels, where here it's just the caller leg. Check the current WebSocket API reference for the exact format names before you ship, because the docs' own example lets the service detect the format automatically.
Barge-in on a phone call has two halves
When the caller really does interrupt, the Soniox docs say: send any text that was already played, mark the end of the segment, and don't send the rest. So the pacer has a second job: on a real interruption, send only the words the caller has already heard, close the turn and drop everything after it.
The phone-specific half: the audio you've already handed to the carrier is still sitting in a buffer somewhere. If you only stop your TTS, the caller keeps hearing the agent for a moment after they started talking, which feels like being ignored. Most media-stream APIs have a way to flush that (Twilio Media Streams has a clear message, for example; SIP-based stacks like LiveKit and Pipecat handle it through their interruption logic). Do both at once: flush the phone leg, close the STT segment.
One honest caveat about the clock: the smaller-of-two-numbers estimate assumes the carrier plays what you send right away. If your provider buffers a lot, the caller hears things later than you think. The fix is the same as everywhere in voice AI: measure on a real call, not a browser tab.
Where Hume's API fits
Hume's Audio Decision API is aimed at a different moment. Echo filtering keeps the live turn-taking clean. Audio questions help you find out where calls went wrong: "were the agent and caller talking over each other?", "is there loud background noise?", "does the caller sound frustrated?". Hume says it's fast enough to ask dozens of questions every few seconds on a live call, and also describes live use as a longer-term opportunity. I'd start with recordings: run a week of recent phone calls through a couple of questions and see whether the "agent interrupted itself" calls cluster on speakerphone or noisy lines. That tells you whether this is a model problem or a phone-line problem before you change anything.
What I'd do next
- Pull 20 recordings where the agent stopped mid-sentence with no real caller speech. Listen to the caller leg only. You'll hear the echo.
- If you're on Soniox already, the new real-time model is stt-rt-v6. Agent text input is one setting in the start request, plus a pacer like the one above.
- Pace the agent's text from playback (audio sent to the phone leg), never from the LLM stream.
- On barge-in, flush the phone leg and close the STT segment together.
- Keep echo cancellation on everywhere it already exists. v6 handles what's left, it doesn't replace it.
Voice agents got fast. Getting them to finish a sentence on a real phone line is the next bit.
I work on the phone-line side at Vobiz, telephony for voice agents, so the phone leg is the part I stare at.
About the author
Piyush Sahoo, Founding Member, Vobiz, writes about where AI voice agents meet the phone network. More at piyushsahoo.me.
Top comments (0)