The recording that finally made me understand the problem was eleven seconds long. A woman calls in to move a dentist appointment. She says "yeah so I need to push my Thursday." The agent starts reading back her options, calm and clear. Two words in, she remembers something and says "oh wait, no, actually keep Thursday, it's Friday I need." And the agent just keeps going. It finishes its entire sentence about Thursday while she is talking over it, both voices stacking into mush, and then there is a beat of dead air where you can hear her decide this is not worth it. She hangs up.
I listened to it four times. The transcript looked fine. The latency dashboard looked fine. Everything we had built to measure was green, and the call was still a small disaster.
Week 1: the numbers that lied
We had launched the appointment agent to a single clinic group the previous Monday. On paper it was healthy. Time to first audio sat around 600 ms. Our turn-detection was conservative but sane. The model rarely said anything wrong.
The one metric that bothered me was hang-ups on interrupted turns. When a caller talked while the agent was mid-sentence, roughly 22% of those calls ended in the next ten seconds. On turns where nobody interrupted, that number was near 4%. Interruption was the poison. I just did not yet know why.
My first assumption was the model. Maybe it was ignoring the interruption text, or the endpoint logic was folding two utterances into one. I spent most of Tuesday there and found nothing. The server was doing the right thing. When a caller spoke, we detected speech, we fired a cancel, we stopped generating tokens. Server-side, the agent stopped talking almost immediately.
The problem was that "server-side stopped talking" and "the caller stopped hearing the agent" were two very different moments in time.
The 3am realization: the audio was already gone
Nobody tells you this about a voice pipeline until it bites you. By the time your server decides to stop, a lot of audio has already left the building.
Trace one chunk of speech through the system. The model generates text. Text goes to TTS. TTS returns audio in frames. Those frames get packetized and sent over the network to the caller's phone. On the way, and at the very end, they land in a jitter buffer that deliberately holds a little audio in reserve so that network hiccups do not cause gaps. Then they play out through the speaker.
Every one of those stages is a small reservoir. When my server sent its cancel, the token stream stopped, sure. But the TTS had already handed me a big block of audio for the current sentence. That block was already packetized. Some of it was already in the jitter buffer on the caller's side, committed to play no matter what I did next. The caller kept hearing the agent because the agent's voice was, quite literally, already in their ear's queue.
So I instrumented the thing I should have measured from day one. I called it the barge-in tail: the gap between the moment we detected caller speech and the moment the caller's device actually went silent. I logged a timestamp when our VAD fired, and I had the client log a timestamp when its output buffer drained to zero after a cancel.
The tail was ugly. Median 1,850 ms. p95 was 2,400 ms. For almost two seconds after a caller started talking, our agent was still audibly talking back. No wonder they hung up. We had built a system that could not take a hint.
Where the two seconds were hiding
I broke the tail down by stage, and it was not evenly spread.
Our TTS was streaming in 400 ms frames. That felt reasonable when we picked it, because bigger frames mean fewer packets and less per-packet overhead. But it also meant that at any instant, we had committed up to 400 ms of a single frame that we could not easily claw back. The jitter buffer on the client was configured at 200 ms, standard and fine. And the last, embarrassing piece: when we sent our cancel, we stopped generating new audio, but we never told the client to throw away the seconds of audio it had already buffered locally for smooth playout. It played every buffered frame to completion first. That local drain was most of the tail.
We were not fighting network latency. We were fighting our own buffers, all of which were doing exactly what we designed them to do.
The fix: stop making audio, then delete the audio you already made
The change had three parts, and the order mattered.
First, when we detect a barge-in, we cancel TTS generation server-side. We were already doing this. Keep it.
Second, and this was the missing piece, we send an explicit flush command down to the client telling it to clear its playout buffer immediately, not after it drains. The audio that is already in the pipe gets dropped on the floor. When someone interrupts, we want silence right then.
Third, we shrank the TTS streaming frame from 400 ms to 120 ms. Smaller frames mean that at any instant, far less audio is committed and unrecoverable. It costs a few more packets per second. On a modern connection that overhead is noise.
The client handler ended up looking close to this:
def on_barge_in(session):
session.tts.cancel() # stop generating new audio
session.audio_out.flush() # drop frames already queued locally
session.jitter_buffer.reset() # clear the 200ms reserve
session.state = "listening"
log_metric("barge_in_tail_ms", now() - session.vad_fired_at)
The flush and jitter_buffer.reset lines were the whole ballgame. Four lines, most of a Thursday to find them.
The objection I had to answer before shipping
One of our engineers, and she was right to ask, worried that shrinking the frame and aggressively flushing would make normal speech choppy. If we clear the jitter buffer too eagerly, a real network hiccup could clip the agent's own words even when nobody interrupted.
So we scoped it. The flush only fires on a confirmed barge-in, never during uninterrupted playback. During normal speech the 200 ms jitter buffer does its job untouched. We only reach for the fire alarm when the caller is actually talking over us. We ran two days of shadow traffic listening for clipped words on non-interrupted turns and heard none. That was enough to ship.
What shipped, and what I would tell past me
We rolled it out to the same clinic group the following Monday. The barge-in tail dropped from a median of 1,850 ms to 180 ms, with p95 at 320 ms. You can hear it on the recordings now: the agent stops the instant the caller speaks.
The hang-up rate on interrupted turns fell from 22% to about 6%, roughly in line with our uninterrupted turns. The interruption poison was mostly gone. Callers still interrupted constantly, because humans do, but now the agent shut up and listened, so it stopped feeling like a fight.
If I could hand one note back to the version of me who built the first pipeline, it would be this. We spent months tuning time to first audio and never once measured how long it took the agent to go quiet, and that was the half that actually lost us calls. A voice agent is judged as much by how fast it stops as by how fast it starts, and every buffer you add for smoothness is a buffer you have to be able to empty on command.
So now the first thing I instrument on any voice pipeline is the tail, and I make sure I can flush every buffer I add. The audio is already gone by the time you decide to stop it. I build like it is.
Top comments (1)
"'Server-side stopped talking' and 'the caller stopped hearing' were two very different moments" — I've paid for that exact gap in a much slower medium, and it's the same bug.
I build internal tools as a non-developer. When I ship an update to a mobile one, I bump a cache version so people stop getting stale code, and for months I treated "I bumped it" as "they have it." Same mistake as your dashboard: the decision to change and the arrival of the change aren't the same event, and everything I measured lived at the decision. Users ran week-old code for days while my deploy reported success — my "healthy" was a green check on the moment I stopped, not the moment they caught up.
The fix was structurally identical to measuring time-to-quiet instead of time-to-stop-generating: I stopped trusting the send and started fetching the live file the user actually receives, then comparing its version to the one I built. It earns its keep the first day those two disagree and a script tells me before a confused user does. Your whole post is the general case — instrument the arrival, not the intent. "The audio is already gone by the time you decide to stop it" is going in my notes verbatim.