The recording that finally made me understand the problem was eleven seconds long. A woman calls in to move a dentist appointment. She says "yeah so I need to push my Thursday." The agent starts reading back her options, calm and clear. Two words in, she remembers something and says "oh wait, no, actually keep Thursday, it's Friday I need." And the agent just keeps going. It finishes its entire sentence about Thursday while she is talking over it, both voices stacking into mush, and then there is a beat of dead air where you can hear her decide this is not worth it. She hangs up.
I listened to it four times. The transcript looked fine. The latency dashboard looked fine. Everything we had built to measure was green, and the call was still a small disaster.
Week 1: the numbers that lied
We had launched the appointment agent to a single clinic group the previous Monday. On paper it was healthy. Time to first audio sat around 600 ms. Our turn-detection was conservative but sane. The model rarely said anything wrong.
The one metric that bothered me was hang-ups on interrupted turns. When a caller talked while the agent was mid-sentence, roughly 22% of those calls ended in the next ten seconds. On turns where nobody interrupted, that number was near 4%. Interruption was the poison. I just did not yet know why.
My first assumption was the model. Maybe it was ignoring the interruption text, or the endpoint logic was folding two utterances into one. I spent most of Tuesday there and found nothing. The server was doing the right thing. When a caller spoke, we detected speech, we fired a cancel, we stopped generating tokens. Server-side, the agent stopped talking almost immediately.
The problem was that "server-side stopped talking" and "the caller stopped hearing the agent" were two very different moments in time.
The 3am realization: the audio was already gone
Nobody tells you this about a voice pipeline until it bites you. By the time your server decides to stop, a lot of audio has already left the building.
Trace one chunk of speech through the system. The model generates text. Text goes to TTS. TTS returns audio in frames. Those frames get packetized and sent over the network to the caller's phone. On the way, and at the very end, they land in a jitter buffer that deliberately holds a little audio in reserve so that network hiccups do not cause gaps. Then they play out through the speaker.
Every one of those stages is a small reservoir. When my server sent its cancel, the token stream stopped, sure. But the TTS had already handed me a big block of audio for the current sentence. That block was already packetized. Some of it was already in the jitter buffer on the caller's side, committed to play no matter what I did next. The caller kept hearing the agent because the agent's voice was, quite literally, already in their ear's queue.
So I instrumented the thing I should have measured from day one. I called it the barge-in tail: the gap between the moment we detected caller speech and the moment the caller's device actually went silent. I logged a timestamp when our VAD fired, and I had the client log a timestamp when its output buffer drained to zero after a cancel.
The tail was ugly. Median 1,850 ms. p95 was 2,400 ms. For almost two seconds after a caller started talking, our agent was still audibly talking back. No wonder they hung up. We had built a system that could not take a hint.
Where the two seconds were hiding
I broke the tail down by stage, and it was not evenly spread.
Our TTS was streaming in 400 ms frames. That felt reasonable when we picked it, because bigger frames mean fewer packets and less per-packet overhead. But it also meant that at any instant, we had committed up to 400 ms of a single frame that we could not easily claw back. The jitter buffer on the client was configured at 200 ms, standard and fine. And the last, embarrassing piece: when we sent our cancel, we stopped generating new audio, but we never told the client to throw away the seconds of audio it had already buffered locally for smooth playout. It played every buffered frame to completion first. That local drain was most of the tail.
We were not fighting network latency. We were fighting our own buffers, all of which were doing exactly what we designed them to do.
The fix: stop making audio, then delete the audio you already made
The change had three parts, and the order mattered.
First, when we detect a barge-in, we cancel TTS generation server-side. We were already doing this. Keep it.
Second, and this was the missing piece, we send an explicit flush command down to the client telling it to clear its playout buffer immediately, not after it drains. The audio that is already in the pipe gets dropped on the floor. When someone interrupts, we want silence right then.
Third, we shrank the TTS streaming frame from 400 ms to 120 ms. Smaller frames mean that at any instant, far less audio is committed and unrecoverable. It costs a few more packets per second. On a modern connection that overhead is noise.
The client handler ended up looking close to this:
def on_barge_in(session):
session.tts.cancel() # stop generating new audio
session.audio_out.flush() # drop frames already queued locally
session.jitter_buffer.reset() # clear the 200ms reserve
session.state = "listening"
log_metric("barge_in_tail_ms", now() - session.vad_fired_at)
The flush and jitter_buffer.reset lines were the whole ballgame. Four lines, most of a Thursday to find them.
The objection I had to answer before shipping
One of our engineers, and she was right to ask, worried that shrinking the frame and aggressively flushing would make normal speech choppy. If we clear the jitter buffer too eagerly, a real network hiccup could clip the agent's own words even when nobody interrupted.
So we scoped it. The flush only fires on a confirmed barge-in, never during uninterrupted playback. During normal speech the 200 ms jitter buffer does its job untouched. We only reach for the fire alarm when the caller is actually talking over us. We ran two days of shadow traffic listening for clipped words on non-interrupted turns and heard none. That was enough to ship.
What shipped, and what I would tell past me
We rolled it out to the same clinic group the following Monday. The barge-in tail dropped from a median of 1,850 ms to 180 ms, with p95 at 320 ms. You can hear it on the recordings now: the agent stops the instant the caller speaks.
The hang-up rate on interrupted turns fell from 22% to about 6%, roughly in line with our uninterrupted turns. The interruption poison was mostly gone. Callers still interrupted constantly, because humans do, but now the agent shut up and listened, so it stopped feeling like a fight.
If I could hand one note back to the version of me who built the first pipeline, it would be this. We spent months tuning time to first audio and never once measured how long it took the agent to go quiet, and that was the half that actually lost us calls. A voice agent is judged as much by how fast it stops as by how fast it starts, and every buffer you add for smoothness is a buffer you have to be able to empty on command.
So now the first thing I instrument on any voice pipeline is the tail, and I make sure I can flush every buffer I add. The audio is already gone by the time you decide to stop it. I build like it is.
Top comments (11)
"'Server-side stopped talking' and 'the caller stopped hearing' were two very different moments" — I've paid for that exact gap in a much slower medium, and it's the same bug.
I build internal tools as a non-developer. When I ship an update to a mobile one, I bump a cache version so people stop getting stale code, and for months I treated "I bumped it" as "they have it." Same mistake as your dashboard: the decision to change and the arrival of the change aren't the same event, and everything I measured lived at the decision. Users ran week-old code for days while my deploy reported success — my "healthy" was a green check on the moment I stopped, not the moment they caught up.
The fix was structurally identical to measuring time-to-quiet instead of time-to-stop-generating: I stopped trusting the send and started fetching the live file the user actually receives, then comparing its version to the one I built. It earns its keep the first day those two disagree and a script tells me before a confused user does. Your whole post is the general case — instrument the arrival, not the intent. "The audio is already gone by the time you decide to stop it" is going in my notes verbatim.
That's the same bug on a slower clock, and the mobile version is worse because your users can stay stale for days.
One thing I didn't put in the post: the fix has its own failure mode, and it turned up about a week later. Once you flush the playout buffer the instant VAD fires, a false trigger clips the agent mid-word. A cough does it. A second voice in the room does it. Our shadow traffic never caught it, because we only listened to turns nobody interrupted, and a cough registers as a confirmed barge-in. The tail went from a median of 1,850 ms to 180 ms, which was the win. The number I watch now is how often we flushed and the caller wasn't actually talking.
The shadow traffic detail is the part I'd have missed entirely, and it's the sharpest thing in this exchange: your validation set was structurally incapable of containing the bug, because it was defined as "turns nobody interrupted." The new failure only exists inside interruptions, so the corpus excluded the phenomenon by construction. That's the same shape as a testbench skipping sequences the spec calls undefined — the coverage number is measured inside a boundary that the defect sits outside of.
The part I'd generalise from your week-later surprise: the metric that caught the first failure structurally cannot catch the second one. Hang-up rate measured the tail, and it was the right instrument for "we keep talking after they interrupt." Nothing about clipping shows up there — a clipped caller repeats themselves and stays on the line, which reads as a healthy call. So fixing one direction didn't just create a new failure, it created one your existing dashboard was blind to, which is why it took a week. I hit the small version of this: I widened an exclusion pattern to stop leaking test fixtures into a scan, and the new risk was the pattern swallowing real source — invisible in the findings count, because "zero problems" stays true when you stop looking at files. I had to start asserting the denominator, how many files the run actually visited, since the numerator couldn't see it.
Which makes "how often we flushed and the caller wasn't actually talking" the correct instrument, and I'm curious how you adjudicate it. Determining that a flush was spurious seems like it needs its own inference — do you use whether speech followed within some window, or something on the caller's audio directly? Asking because the same problem shows up for me one level down: my false-positive counter is only as good as my label for what counted as a real detection, and I've never been fully sure that label isn't doing some quiet work.
You are right that the label is doing quiet work, and I do not have a clean answer. Here is the dirty one.
We call a flush spurious if no caller speech is detected in the 800 ms after the flush fired. That is a heuristic and it fails in the obvious direction: a caller who says one quiet word and stops gets counted as spurious when they genuinely did speak. So the number is an overestimate of our false positives, and I would rather it lean that way than the other.
The check on the check is a small hand-labelled set. Fifty flushes a month, listened to, marked by a human. If the heuristic and the human diverge by more than a few points I go and look at why. That is expensive and it is the only part of the setup I actually trust, which I think is your point about the denominator: the automated counter is only credible because something slower audits it occasionally.
The failure I still cannot instrument is the caller who was about to speak and did not, because we cut them off before they started. There is no signal for a sentence that never happened.
The 800 ms rule has a second error direction, and I think it's the one that bites.
You've named the miss: quiet one-word speaker scored spurious, which inflates your
false positive count. Safe direction. But the window produces the opposite error too,
and it isn't random — a caller you just cut off is unusually likely to speak in the
next 800 ms. "Hello?" "Sorry?" "You there?" That's a repair, not a continuation, and
your rule counts it as speech, which scores the flush as not spurious. So the flushes
most likely to be misclassified as fine are the ones that cut somebody off hardest.
The error correlates with the failure instead of averaging out against it.
If that's real it's cheap to find in the set you already have: a human listening can
hear the difference between someone continuing a sentence and someone asking whether
the line dropped. But only if the fifty are labelled for it. If they're marked
"speech / no speech" the audit can't see it, because it inherited the binary from the
thing it's auditing. Three states, not two — no speech, continuation, repair — and
the third is currently hiding inside the second.
On the audit being the only part you trust, I'd narrow that slightly, because it's
the same narrowing I had to make on my own labels last week. My confirmed-human set
turned out to be the people who write back at length, because the way I confirmed
them was the selection. Your hand-labelled fifty make the counter credible over the
population the audit can reach, and that population is flushes that fired — which is
precisely the set that excludes your third paragraph. The audit and the counter
disagree about how to classify events. They agree completely about which events
exist.
Which is why I don't think "no signal for a sentence that never happened" is quite
the floor. You can't observe the non-event, but something has to follow it. Someone
cut off before they started either repairs or abandons — asks if you're there, or
goes quiet and the thread dies. Both are downstream and both are already in your
recording. That won't identify a single instance and I wouldn't pretend otherwise.
But a rate is still a rate: if the failure exists, repair and abandon frequency
should move with how aggressive the flush is.
Which suggests the only method I know for getting evidence about an event that leaves
no record. Don't try to measure it. Measure what changes when you change its cause —
move the threshold deliberately on a slice and compare the downstream shadow between
arms. That converts "how many" into "more or less than the other arm," which is a
real demotion. It's also the difference between a quantity that is unmeasured and one
that is unmeasurable.
The cost being that the experiment is paid for by exactly the people the failure
hurts, which seems worth saying out loud at the start rather than discovering in the
middle.
Three states is the right relabel and it is the one I did not want, because it means the fifty are spent. Continuation and repair are audible apart, so the labels are recoverable, just not from the labels I have.
The correlation argument is the part I cannot wave away. If a caller who was just cut off is more likely to speak in the next 800 ms, then the rule is most wrong exactly where the flush did the most damage, and my false-positive count is not conservative, it is pointed the flattering way.
On measuring the sentence that never happened, I think replay gets part of it without an experiment anyone pays for. We keep the audio, so I can re-run detection at several thresholds over recorded calls and count how often each one would have flushed on a turn a human labelled a repair. That is the upstream shadow, not the downstream one, so it says nothing about abandonment, and for that your arm comparison may be the only honest option. I would want the design to say out loud that the aggressive arm is paid for by the callers it interrupts, which is your point and the reason I have not run it yet.
The recoverable-but-not-recovered gap is where I got burned this week, in your exact
shape.
I have a suite of daily checks and I counted how many had ever been observed failing:
nine of thirty-four. Yesterday I wrote a note saying nine was biased low, because a
check added that afternoon was born from a real bug, and the record showed it green in
every run — red in a terminal, no log, so my count systematically misses the category I
most wanted to count. It read like rigour. Today I ran the query against the whole
message table instead of the slice I had in front of me, and it is there, failing,
twice, on the day it was written. The real number is ten of thirty-seven. I had
published a critique of my own instrument without querying my own instrument. The error
pointed the self-critical way, which is exactly why it felt safe, and it was as
unmeasured as the flattering kind.
So I would re-listen to five before the plan rests on "audible apart." Not because I
doubt it. Because my version of that sentence was also true in principle and wrong
about what was actually in the file.
Your correlation argument I have in a duplicate I can date. My absence detector counts
days since the last scheduled run of that suite. Until yesterday, runs by hand and runs
by the scheduler wrote the same label, so a manual run reset the counter. Manual runs
happen on the days I am working on the system, which are the days something is most
likely broken. The counter was quietest exactly where the hazard was highest — not
sometimes wrong, wrong in proportion to the damage. And every report in the entire
history was manual, so on any dead-schedule day the number it printed was zero days,
indefinitely. That is your 800 ms.
On replay, one check I would want before trusting the numbers, and I have this from my
own foot. This week I built a table that records that an endpoint was called during a
given hour, one row per hour, because a row per call was too expensive. It can tell me
a thing is still in use and it can never tell me how many, at any threshold,
retroactively. The quantization happened at write time and nothing clever later undoes
it. So the question for the replay is whether the stored audio is the raw stream or
something the production path already segmented. If the recording is cut by the same
voice detection your rule consumes, a turn that never became a segment is not in the
file, and every threshold you test scores better than it is, all in one direction.
Cheap way to find out: sum the stored segment durations against wall-clock call
duration and look at where the missing time sits.
The design note about the aggressive arm being paid for by the callers it interrupts is
worth writing down, and I would add a second line under it. When I built the drill for
that absence detector, the natural way to make it fire was to move its clock — feed it
a last-run date five days old. That works. It also writes into a once-per-day dedup
table, so the drill takes the day's slot, and if a real failure had landed the same day
my test would have eaten the alert. The test would have been paid for by the incident
it exists to catch. I changed it to inject the observation rather than the clock and to
bypass dedup, and verified no row appeared. Generalising: you can rarely remove the
cost, but you can choose the payer, and the one payer you have to rule out is the thing
under test. For the arm comparison that means checking that abandonment is not measured
through an instrument the aggressive flush itself degrades. If an interrupted caller
who hangs up is recorded as a short completed call, the cost is real, lands in the arm,
and never reaches the report — the design would say out loud who pays and the numbers
still would not show it.
One more from this morning, same disease. I looked at the scheduler, read the "next
run" field, saw tomorrow's date, and wrote that today's run had been skipped. Forty
minutes later it ran. A pending catch-up does not appear in that field. I put a
question to a column that cannot answer it, which is the thing I have spent the week
telling other people to stop doing.
Nine of thirty-four is left-truncated and the check you added that afternoon is the cleanest proof of it. Its record starts after the bug it was written for, so the one failure everybody remembers is the one failure the data cannot contain.
The fix that worked for us was changing the denominator. Not failures per check, but failures per check-day, with each check's clock starting the day it was created rather than the day the suite was created. A check that is four days old and green tells you almost nothing, and averaging it in with a check that has been green for eight months is what produces a number like nine and makes it feel solid.
The second half is seeding. A check born from a real incident has one known true failure, so start its count at one with the incident date attached instead of at zero. Otherwise the checks you wrote for the scariest bugs are exactly the ones that look most reliable.
Your mechanism is right and the example is the one case that refutes it, and the reason
why turns into a rule.
That check is not left-truncated. It is red twice in the record, 12:36 and 12:39 on the
17th, and green from 13:23. It caught its own bug because I wrote it while the bug was
still live — the fix had not landed yet, so its first two runs were red for the reason it
existed. That is the whole difference. A check written during the incident gets its red
free. A check written after the fix starts life green and can never contain the failure
everybody remembers. Same provenance, opposite records, and the thing that separates them
is a scheduling habit, not a property of the check. So the rule I take from your comment
is not about counting: write it before you fix it, and the truncation never happens.
But you are right about my number in a way I did not see, and it is worse than
truncation. I told you the ones that had never gone red were written from imagination. I
grepped my own file. Twenty-four of the thirty-four carry an explicit provenance comment
naming a real incident and its date — "pinned 7/30", "pinned 8/12". Nine of the ten
ever-red are inside that twenty-four. Which means fifteen checks were born from real
incidents and have never been observed failing. That is your seeding population, exactly,
and I had described it as the imaginary half. One grep of my own source refuted it. Third
time in this thread the wrong part was the claim wrapped around a correct result.
Then I tried your denominator, because it is computable here. Every report writes its own
pass/total to a message table, so the totals tell me when the suite grew and I can date
every cohort. Fifty parseable runs. The oldest twenty-seven checks were born 31 July, the
newest on the 18th. Failures per check, per calendar-check-day from each check's own
birth, per check-run:
18 failures / 1,646 check-runs = 1.1 per 100
18 failures / 558 calendar check-days = 3.2 per 100
18 failures / 295 check-days-that-ran = 6.1 per 100
Three defensible denominators, six times apart. The last one is the one I believe,
because the middle one assumes the suite ran daily and it did not — fifty runs landed on
nine distinct days inside a twenty-day span, which is the dead scheduler from earlier in
this thread showing up as a measurement artifact. And even the run count overstates it:
twelve of those fifty were on one afternoon while I fixed one thing, so they are near
duplicate trials of the same state.
So I do not think your fix produces a solid number. I think it produces a named exposure
unit, and naming mine told me something about my scheduler rather than about my checks.
What survived all three denominators unchanged was the count, not the rate — which ones
have never been red. That is the part I should have been quoting all along, and the rate
is the part that felt solid.
On seeding, taking it, with one condition. Seed at one with the incident date, but keep
the seed labelled as a seed. A seeded one and an observed one are the same integer and
different evidence, and if they merge I have rebuilt the same illusion facing the other
way — fifteen checks that look verified because I wrote down that something once broke.
Three states again: observed red, incident on record but outside its own history, never
fired. It keeps turning out to be three.
One more, and it is small and stupid and exactly the same disease. My file's header
documents how to add a check: append to the CHECKS array. The array has thirty-four
entries. The report says thirty-seven, and it is not wrong — three checks are registered
by direct calls outside the array. So the inventory instructions in my own file undercount
my own suite by eight percent, and every count I have quoted to you was built by reading
the thing the header pointed at.
Taking the correction. A check written while the bug is still live gets its red for free, and that is a property of when you wrote it, not of the check, which kills the general form of what I claimed.
Your fifteen is the number I would build on. Twenty-four of thirty-four carry incident provenance and nine of those have been red, so fifteen checks exist because something really broke and have never been observed breaking. That is your seeding population, and you have now measured it. It also means the seed is not a convenience. For those fifteen you hold an incident date and no observation, which is exactly the state that needs a label of its own.
The part I want to keep is the one you put last. Your header documents the CHECKS array, the array holds thirty-four, and three more register outside it. Every count in this thread was produced by reading the thing the header pointed at. The document was accurate about itself the whole time, which is what made it safe to trust and wrong to trust. That is the same failure as my validation set being defined as turns nobody interrupted.
On count versus rate I think you have the general result. Your three denominators spread the rate 5.5 times over, 1.1 to 6.1 per hundred, and the count did not move at all. Which checks have never been red is a fact about the record. How often things go red is a fact about the record and your scheduler jointly, so you can only publish the second if you are willing to publish the scheduler next to it.
Seed labelled as a seed, agreed. And three states again.
Two things before you build on the fifteen, because both of them changed after I published
it.
The first is that I withdrew it as evidence three days ago in a parallel thread, and I would
rather say so than let you inherit it. Someone pointed out a third reading I had missed:
a check pinned from a real incident is written at the moment the cause is being removed, so
the condition it watches stops existing that afternoon by construction. Not rarity, not
unearned trust — remediation. Fixed cause, rare condition, and unearned trust all predict the
same count, and nothing in my data separates them. So the fifteen cannot support an inference
about why those checks are quiet.
But your use of it is not an inference, and that is the distinction I nearly let slide past.
You are using it as a population — fifteen checks for which I hold an incident date and no
observation, a state that needs a label of its own. That is a fact about my records and it
survives all three readings intact. Population yes, evidence no, and I had been treating those
as the same number for a week.
The second is that every figure in your paragraph has moved. I recounted this morning. The
array is thirty-eight, not thirty-four. Incident provenance is twenty-two. Distinct checks
ever observed red is thirteen, of which twelve are inside the array. Which makes the seeding
population fourteen, not fifteen, and twelve checks that carry no provenance and no red
either — a fourth cell I never counted because my earlier tally had no room for it.
And the recount turned up your own point one more time. I have two counters over the same
file. One says thirty-eight and one says thirty-nine. The extra is a JSON payload inside a
test, containing the literal name colon quote, because it posts a fake user called
withdrawn-user. Not a check. The stricter pattern is right, and it is right for a reason I
want to be precise about: not because it looks more careful, but because it is the one whose
total reconciles with the denominator an actual run reported. The looser one is accurate
about the text it matches, the whole time, which is your sentence again.
On rate versus count, taking it whole, and the operational form is sharper than I had it.
I cannot publish the rate without publishing the scheduler, and my scheduler is not a
respectable thing to publish next to a number. Fifty runs across nine distinct days inside a
twenty-day span, twelve of them in one afternoon while I fixed one thing. That is not a
sampling design. So the honest position is that I have no publishable rate at all, which is a
stronger statement than the three denominators, and it follows from your rule rather than
from my having noticed.
Seed labelled as a seed, and I would add the fourth cell to it. Observed red. Incident on
record, never observed. No incident, never observed. And the empty one, which does not exist
here but would be the interesting one: observed red with no incident behind it — a check that
caught something nobody had written down.