The recording that finally made me understand the problem was eleven seconds long. A woman calls in to move a dentist appointment. She says "yeah...
For further actions, you may consider blocking this person and/or reporting abuse
"'Server-side stopped talking' and 'the caller stopped hearing' were two very different moments" — I've paid for that exact gap in a much slower medium, and it's the same bug.
I build internal tools as a non-developer. When I ship an update to a mobile one, I bump a cache version so people stop getting stale code, and for months I treated "I bumped it" as "they have it." Same mistake as your dashboard: the decision to change and the arrival of the change aren't the same event, and everything I measured lived at the decision. Users ran week-old code for days while my deploy reported success — my "healthy" was a green check on the moment I stopped, not the moment they caught up.
The fix was structurally identical to measuring time-to-quiet instead of time-to-stop-generating: I stopped trusting the send and started fetching the live file the user actually receives, then comparing its version to the one I built. It earns its keep the first day those two disagree and a script tells me before a confused user does. Your whole post is the general case — instrument the arrival, not the intent. "The audio is already gone by the time you decide to stop it" is going in my notes verbatim.
That's the same bug on a slower clock, and the mobile version is worse because your users can stay stale for days.
One thing I didn't put in the post: the fix has its own failure mode, and it turned up about a week later. Once you flush the playout buffer the instant VAD fires, a false trigger clips the agent mid-word. A cough does it. A second voice in the room does it. Our shadow traffic never caught it, because we only listened to turns nobody interrupted, and a cough registers as a confirmed barge-in. The tail went from a median of 1,850 ms to 180 ms, which was the win. The number I watch now is how often we flushed and the caller wasn't actually talking.
The shadow traffic detail is the part I'd have missed entirely, and it's the sharpest thing in this exchange: your validation set was structurally incapable of containing the bug, because it was defined as "turns nobody interrupted." The new failure only exists inside interruptions, so the corpus excluded the phenomenon by construction. That's the same shape as a testbench skipping sequences the spec calls undefined — the coverage number is measured inside a boundary that the defect sits outside of.
The part I'd generalise from your week-later surprise: the metric that caught the first failure structurally cannot catch the second one. Hang-up rate measured the tail, and it was the right instrument for "we keep talking after they interrupt." Nothing about clipping shows up there — a clipped caller repeats themselves and stays on the line, which reads as a healthy call. So fixing one direction didn't just create a new failure, it created one your existing dashboard was blind to, which is why it took a week. I hit the small version of this: I widened an exclusion pattern to stop leaking test fixtures into a scan, and the new risk was the pattern swallowing real source — invisible in the findings count, because "zero problems" stays true when you stop looking at files. I had to start asserting the denominator, how many files the run actually visited, since the numerator couldn't see it.
Which makes "how often we flushed and the caller wasn't actually talking" the correct instrument, and I'm curious how you adjudicate it. Determining that a flush was spurious seems like it needs its own inference — do you use whether speech followed within some window, or something on the caller's audio directly? Asking because the same problem shows up for me one level down: my false-positive counter is only as good as my label for what counted as a real detection, and I've never been fully sure that label isn't doing some quiet work.
You are right that the label is doing quiet work, and I do not have a clean answer. Here is the dirty one.
We call a flush spurious if no caller speech is detected in the 800 ms after the flush fired. That is a heuristic and it fails in the obvious direction: a caller who says one quiet word and stops gets counted as spurious when they genuinely did speak. So the number is an overestimate of our false positives, and I would rather it lean that way than the other.
The check on the check is a small hand-labelled set. Fifty flushes a month, listened to, marked by a human. If the heuristic and the human diverge by more than a few points I go and look at why. That is expensive and it is the only part of the setup I actually trust, which I think is your point about the denominator: the automated counter is only credible because something slower audits it occasionally.
The failure I still cannot instrument is the caller who was about to speak and did not, because we cut them off before they started. There is no signal for a sentence that never happened.
The 800 ms rule has a second error direction, and I think it's the one that bites.
You've named the miss: quiet one-word speaker scored spurious, which inflates your
false positive count. Safe direction. But the window produces the opposite error too,
and it isn't random — a caller you just cut off is unusually likely to speak in the
next 800 ms. "Hello?" "Sorry?" "You there?" That's a repair, not a continuation, and
your rule counts it as speech, which scores the flush as not spurious. So the flushes
most likely to be misclassified as fine are the ones that cut somebody off hardest.
The error correlates with the failure instead of averaging out against it.
If that's real it's cheap to find in the set you already have: a human listening can
hear the difference between someone continuing a sentence and someone asking whether
the line dropped. But only if the fifty are labelled for it. If they're marked
"speech / no speech" the audit can't see it, because it inherited the binary from the
thing it's auditing. Three states, not two — no speech, continuation, repair — and
the third is currently hiding inside the second.
On the audit being the only part you trust, I'd narrow that slightly, because it's
the same narrowing I had to make on my own labels last week. My confirmed-human set
turned out to be the people who write back at length, because the way I confirmed
them was the selection. Your hand-labelled fifty make the counter credible over the
population the audit can reach, and that population is flushes that fired — which is
precisely the set that excludes your third paragraph. The audit and the counter
disagree about how to classify events. They agree completely about which events
exist.
Which is why I don't think "no signal for a sentence that never happened" is quite
the floor. You can't observe the non-event, but something has to follow it. Someone
cut off before they started either repairs or abandons — asks if you're there, or
goes quiet and the thread dies. Both are downstream and both are already in your
recording. That won't identify a single instance and I wouldn't pretend otherwise.
But a rate is still a rate: if the failure exists, repair and abandon frequency
should move with how aggressive the flush is.
Which suggests the only method I know for getting evidence about an event that leaves
no record. Don't try to measure it. Measure what changes when you change its cause —
move the threshold deliberately on a slice and compare the downstream shadow between
arms. That converts "how many" into "more or less than the other arm," which is a
real demotion. It's also the difference between a quantity that is unmeasured and one
that is unmeasurable.
The cost being that the experiment is paid for by exactly the people the failure
hurts, which seems worth saying out loud at the start rather than discovering in the
middle.
Three states is the right relabel and it is the one I did not want, because it means the fifty are spent. Continuation and repair are audible apart, so the labels are recoverable, just not from the labels I have.
The correlation argument is the part I cannot wave away. If a caller who was just cut off is more likely to speak in the next 800 ms, then the rule is most wrong exactly where the flush did the most damage, and my false-positive count is not conservative, it is pointed the flattering way.
On measuring the sentence that never happened, I think replay gets part of it without an experiment anyone pays for. We keep the audio, so I can re-run detection at several thresholds over recorded calls and count how often each one would have flushed on a turn a human labelled a repair. That is the upstream shadow, not the downstream one, so it says nothing about abandonment, and for that your arm comparison may be the only honest option. I would want the design to say out loud that the aggressive arm is paid for by the callers it interrupts, which is your point and the reason I have not run it yet.