DEV Community: Marcus Chen

Turn thirty-one, and the agent asked for the account number the caller gave on turn four

Marcus Chen — Tue, 25 Aug 2026 05:09:58 +0000

Look: the call was going fine. That is what made it annoying.

A caller with a billing dispute. Long call, the kind we are actually proud of, because a year ago the agent would have dumped her to a queue in ninety seconds. She gave her account number on turn four. She explained the charge, we pulled the invoice, she disputed two lines, we walked the two lines, she asked about the credit timing, we answered, she asked whether it would hit before her next statement.

Turn thirty-one, the agent said: "Sure, can I get your account number?"

She hung up.

Week one: the wrong suspect

My first guess was the ASR. It usually is. I pulled the audio, the transcript was clean, turn four had the digits, and the digits were right.

Second guess was the tool layer. Also wrong. The account number had been passed to three separate tools during the call and all three had it.

The number was in the transcript and in the tool arguments. It was not in the prompt.

What our context policy actually was

Nobody wrote it down, which is the first thing I would tell past me. It had accreted. Read out of the code, it was:

keep the last 6 turns verbatim
everything older gets folded into a running summary
every 5 turns the summarizer is handed the previous summary plus whatever just fell out of the window, and writes a new one. It never sees the original transcript again. That last clause is the whole post.
prepend the summary, then the 6 verbatim turns

That is a completely ordinary policy. I have seen close variants in four codebases. It is also the thing that ate the account number, and the reason took me embarrassingly long to see, because I kept thinking about it as one summarization step instead of a chain of them.

The arithmetic nobody had done

A fact stated on turn 4 and needed on turn 31 is 27 turns old. It is out of the 6-turn verbatim window, so it lives in the summary. And the summary does not get summarized once. It gets regenerated every 5 turns, and each regeneration is a fresh chance to drop the fact.

So how many times does it get rewritten? I reached for a formula, ceil((d - K) / S), which gives five. Then I enumerated the actual schedule and it is four: our summarizer regenerates on multiples of five, turn 4 falls out of the verbatim window at turn 10, and it gets rewritten at 15, 20, 25 and 30.

The formula is an upper bound over where the fact happens to land. Holding the distance at 27 turns and sliding the statement turn, the true count is four in eight of the ten possible alignments and five in two. So the question is not "does my summarizer keep account numbers." It is "does my summarizer keep account numbers four times in a row," and the four is a small integer nobody on the team could have named.

We measured the per-pass number, which turned out to be the only measurement worth taking. Take a sample of real calls, take the facts you care about, and for each regeneration check whether the fact went into the new summary. Ours came out at about 0.90 for structured identifiers. It was worse for constraints stated in prose ("she said she is travelling until the 14th"), which I will come back to.

Then it is just powers:

1 pass: 90.0 percent survival
2 passes: 81.0 percent
3 passes: 72.9 percent
4 passes: 65.6 percent
5 passes: 59.0 percent
7 passes: 47.8 percent

Our case is four, so 65.6 percent. A summarizer I would have described to you as good, because 90 percent sounds good, is closer to two-in-three a half hour into a call, and it is one boundary alignment away from 59.

I have written about compounding before, a couple of weeks ago, about per-turn error rates across a call, and I read that as a fact about the model. This is a different exponent. This one counts the number of times we rewrote the fact ourselves, and that count is not in any config file.

And a real call does not carry one fact. Ours carry about five that matter: the identifier, the specific charge in dispute, the resolution the caller asked for, one constraint on timing, and whatever the caller said they had already tried. Treat those five as independent and the picture at turn 31 is:

per-fact survival 65.6 percent
expected facts still present: 3.28 of 5
all five present: 12.2 percent of calls, under independence

I said something close to the opposite of what follows a couple of weeks ago, so let me correct myself. Positive association between the drops makes the all-clean number BETTER than the independent estimate, not worse, so 12.2 percent is a floor rather than an estimate. What association makes worse is how it feels, because the calls that lose one fact are the same calls that lose three.

How far above 12.2 percent the truth sits depends on how much the drops share a cause, and I cannot measure that from a few hundred calls. The number I trust is the expected count, because expectation does not care about correlation at all: 3.28 facts of 5 still present at turn 31, however the drops are coupled.

The failure you can compute, and the one you cannot

The comparison that changed how I think about this. Same policy family, same K, and the difference is what happens past the window edge.

inside the 6-turn window: verbatim window keeps it, rolling summary keeps it
1 rewrite later: window 0 percent, summary at r=0.90 90.0 percent
2 rewrites: window 0 percent, summary 81.0 percent
4 rewrites, which is where our turn-4 fact lands at turn 31: window 0 percent, summary 65.6 percent
7 rewrites: window 0 percent, summary 47.8 percent

I am counting rewrites rather than turns on purpose. Turns are what you can see and rewrites are what actually happens to the fact, and the map between them depends on where the fact falls relative to your regeneration boundary. On our policy, roughly one rewrite per five turns once the fact is out of the window.

The plain truncation window is honest. It fails completely and it fails at a turn number you can compute in your head. You lose everything past turn K and you know it, so you build around it.

The rolling summary is the one that hurts, because it works. It works at 10 turns, it mostly works at 20, and it degrades smoothly enough that nothing in your testing is going to catch the edge. There is no turn number where it breaks. There is a probability that slides, and what a slide produces is not a bug report. It is a slow drip of calls that went weird late.

Raising r does not save you either, it just moves the wall. At r=0.95, four rewrites leaves you at 81.5 percent. At r=0.98, which is better than anything I have measured, four rewrites is 92.2 percent and seven is 86.8. Nothing you can do to r buys you a long call.

Week two: what we changed

Three things, in the order they mattered.

First, and this is most of the fix, we stopped putting facts that must not be lost into a lossy channel. This is the same shape as the handoff card we built for the human escalation path, and I did not notice that for a week. That card was for a seam between two systems. This one is for a seam inside one system, between the agent and its own past, and nobody had drawn that seam on any diagram. There is now a slots object that lives outside the summary entirely:

from dataclasses import dataclass, field, fields

@dataclass
class CallSlots:
    account_id: str | None = None
    disputed_items: list[str] = field(default_factory=list)
    requested_resolution: str | None = None
    timing_constraint: str | None = None
    already_attempted: list[str] = field(default_factory=list)

    def render(self) -> str:
        lines = [f"{f.name}: {v}" for f in fields(self)
                 if (v := getattr(self, f.name))]
        return "KNOWN FACTS (verbatim, do not paraphrase):\n" + "\n".join(lines)

It is prepended whole on every turn, it is never summarized, and it is populated by the tool layer rather than by the model, because the tool layer is where the account number was already correct. Cost is under a hundred tokens on a call that was already spending thousands. Survival is 100 percent by construction, and for the account number that is the only rate I am willing to design around.

Yes, this is a slot-filling dialogue manager, the thing everyone spent 2023 declaring obsolete. I was one of the people declaring it. What I had wrong was the reason it existed. I assumed it was a workaround for models that could not follow a conversation. The models follow the conversation fine. What they cannot do is recover a string that four rewrites ago decided was not important enough to carry.

Second, we made the summarizer's job smaller and told it so. It no longer has to remember the account number, because the account number is not its problem any more. Its prompt now says which categories are handled elsewhere. Retention on what remains went up, which is unsurprising, and I am not going to publish a number for it because I only have a few hundred calls and the confidence interval is wider than the effect.

Third, we log the passes. Every regeneration writes a span attribute with the fact keys going in and the fact keys coming out. The diff is the drop. That took an afternoon and it converted an invisible failure into a line on a dashboard, and I would do that one first if I were starting over, because we spent a week on suspects we could have eliminated in an hour.

The prose facts are worse and I have no clean fix

The identifiers survive because they are short, structured, and obviously important. The constraint stated in prose is the one that dies quietly.

"She is travelling until the 14th" is the fact that makes the whole call correct, and it has no field to live in. Our slots object has a timing_constraint string, which helps when the extractor notices it, and the extractor notices it maybe most of the time. That is a smaller unsolved problem rather than a solved one. Anyone selling you a summarizer that keeps every soft constraint through a forty-turn call is selling you something they have not measured.

What I do now is cheap and blunt: before the agent takes any action with a date in it, it re-reads the full transcript for date-shaped constraints, once, at that moment. One extra call on the small number of turns where being wrong is expensive. It is not elegant. It has caught three of these since we shipped it.

What shipped, and what I would tell past me

Shipped: a slots object outside the summary, a smaller summarizer with an explicit list of what it does not own, per-regeneration drop logging, and a targeted re-read before date-bearing actions.

What I would tell past me is not any of that. It is: write down your context policy, then compute how many times a fact from turn 4 gets rewritten before turn 31. The answer was five, and five was the whole bug. Nobody on the team could have told you that number, and it was three lines of arithmetic away the entire time.

The caller who hung up was right to. From where she sat, she had told us her account number, and then a machine that had been doing quite well suddenly had not been listening. That is exactly what happened. We just had it filed under "summarization quality" instead of under "we ask the model to remember something twenty-seven turns after we threw it away."

What shipping a voice-agent feature taught me about real testing

Marcus Chen — Wed, 19 Aug 2026 04:56:44 +0000

They all claim to "simulate real calls." The difference that mattered was whether the simulation could reproduce the failure I actually saw in production.

TL;DR: Over a few months I tried five tools for testing a production voice agent before shipping changes. They sort into two camps: voice-testing specialists (Hamming, Coval, Cekura) and broader platforms that added voice simulation (Future AGI, plus Vapi's built-in testing if you build there). The deciding question was not the feature list, it was whether I could take a real failed call and replay it as a regression test. Here is the rundown as of June 2026.

Why "simulate a call" is not one feature

Every tool says it simulates calls, but that covers a wide range: a scripted turn-by-turn script, a synthetic persona holding a free-form conversation, or replaying a real transcript. And voice adds failure modes a text agent never has: barge-in, end-of-turn detection, ASR errors, latency under load. A tool that simulates text dialogue well can miss every one of those, which is why "it has simulation" tells you almost nothing on its own.

The five, and what each actually does

Hamming: the scale and breadth pick. Auto-generated test cases, simulation of 1000+ concurrent calls, 50+ built-in metrics, and CI/CD integration. If you need load testing and a wide metric set out of the box, this was the most complete of the five.
Coval: the simulation-first pick. It treats voice QA the way self-driving teams treat simulation, you define scenarios and it runs many variations of each. Fits cleanly into CI if you already think in scenarios.
Cekura: the least-setup pick. It reads your agent config and auto-generates test scenarios and edge cases, so you are not hand-writing a suite from zero. The right call when you are early and have no tests yet.
Future AGI Simulation: the platform pick. You define synthetic personas with a simulation type, voice or text, and run them through your agent before prod. The draw is that it sits in the same place as the eval and observability, so a simulated failure and a real prod trace live together instead of in two tools; their eval SDK is Apache-2.0 on github.com/future-agi, and voice simulation is part of the platform as of June 2026. The synthetic-data generation behind the personas is on their paid tier.
Vapi: the build-here pick. If you already build your agent on Vapi, its test and simulation tooling is right there with no second integration. Much less compelling if you do not.

The broader landscape has more (Bluejay, Cyara, Roark, others), but these five are the ones I actually ran. I am not crowning one: for raw scale and metrics it was Hamming, for scenario-style CI it was Coval, for zero-setup it was Cekura. They genuinely serve different shapes of team.

The question that decided it

Can I take a real failed call and turn it into a permanent regression test? Most of these generate NEW synthetic calls, which is great for coverage but does nothing about the specific bug that just bit you. The tools that let me import a real transcript or trace and replay it as a fixed case are the ones that actually stopped repeat incidents. That capability, not the metric count, is what I would evaluate on first.

What shipped, and what I would tell past me

We kept two of the five running: one for raw scale, one for scenario-style CI. If voice is your whole product, a specialist goes deeper on voice-specific failure modes. If voice is one surface among several, a platform that also does eval and observability keeps it in one place. Manual QA did not go away, because simulation catches regressions and not taste, and whether it sounds natural and whether the persona is right are still human calls.

What I would tell past me is to check barge-in and end-of-turn timing explicitly before picking anything. Not all of these model interruption or turn boundaries, and that is exactly where a voice agent feels broken to the person on the phone.

The thing I still have not solved: every one of these generates or replays calls, and none of them told me which new failure modes I had not thought to test for. Coverage is bounded by my own imagination of how the agent breaks. If you have a tool that surfaces the unknown-unknowns in a voice agent, rather than testing the ones you already named, that is the one I want to hear about.

Your ASR confidence score is a number you can act on. We were throwing it away.

Marcus Chen — Tue, 18 Aug 2026 05:58:11 +0000

Voice agents treat the transcript as ground truth. The speech recognizer often tells you it is not sure, and we were ignoring it.

TL;DR: When I went back through a month of "the agent did the wrong thing" incidents on our voice agent, close to a third of them started with a transcription the speech recognizer had already flagged as low confidence. The agent acted on it anyway, because nothing downstream of the recognizer looked at the confidence score. We started routing low-confidence turns to a one-line confirmation, and that class of failure mostly went away.

The transcript is not ground truth, and the recognizer knows it

A voice agent is a pipeline, and the first stage hands the rest of the system a string. Everything after that, intent classification, tool calls, the whole agent, treats that string as what the user said. But the recognizer almost always hands you a confidence score alongside the words, and a low score is the recognizer telling you, in advance, that it is guessing. We were dropping that score on the floor and treating a guess exactly like a certainty.

What the incidents actually looked like

The failures were rarely the dramatic kind. They were a caller saying an order number that came through with two digits wrong, or a yes that the recognizer scored as a coin flip because of background noise, and the agent confidently proceeding to act on the misheard version. To the dashboards everything looked fine: the agent did exactly what its input said. The input was wrong, and the one part of the system that suspected the input was wrong had already said so and been ignored.

The fix was a confirmation turn, not a better model

We did not swap recognizers or fine-tune anything. We put a threshold on the confidence score for turns that lead to a state change. Below it, the agent does not act, it reflects back what it heard and asks the caller to confirm. "I have order four-four-one-two, is that right." Above it, it proceeds as before. The cost is one extra turn on the uncertain calls, which is exactly the calls where an extra turn is worth it. The wrong-action incidents that traced back to a low-confidence transcript dropped by most of their volume.

Why this is easy to miss

The confidence score lives at the bottom of the stack and the failures show up at the top, several components away, so nobody connects them. The recognizer's own metrics looked healthy, word error rate was fine on average. Averages were never the problem. The problem was the specific turns where the recognizer was uncertain and we acted as if it were certain, and you only see those if you carry the confidence score forward to the moment of action and log it there.

The Remaining question

The threshold is the hard part and I do not have a principled way to set it. Too high and you confirm everything and the agent feels slow and patronizing. Too low and you let through the misses you were trying to catch. We set ours empirically, just below the confidence level where our incidents clustered, which is reactive in exactly the way I keep complaining about. If anyone has a calibrated way to choose a confirmation threshold per intent rather than one global number, that is the comment I want to read.

The demo was flawless. The first real call had three people talking.

Marcus Chen — Thu, 13 Aug 2026 22:10:47 +0000

A scripted voice-agent demo works with one clean speaker. The first production call had crosstalk, and turn-taking fell apart. This is how I fixed it.

The demo went perfectly. It always does. One person, one microphone, a quiet room, and a script we had rehearsed maybe forty times. The agent listened, waited its turn, answered in about 800ms, and everyone in the room nodded. We shipped it to a pilot customer that Friday.

The following Monday, 9:14am, the first real call came in. A support line for a property-management company. The caller was in a car. Her husband was in the passenger seat. Their kid was in the back. Three humans, one phone, all talking at once, and my careful little agent sat there and did the worst possible thing: it started answering the kid.

Week 1: the demo lie

Our stack was ordinary. WebRTC brought audio in from the browser and the phone bridge, a voice-activity detector decided when someone was speaking, and when the VAD said "silence for 700ms" we treated that as end-of-turn and fired the transcript at the LLM.

That endpointing rule is the whole problem, and I did not see it for two days.

With one speaker, a 700ms silence gap almost always means "I finished my sentence, your turn." The rule works. It works in every demo you will ever give, because demos have one cooperative speaker who pauses politely.

Real calls do not pause politely. People talk over each other. They finish each other's sentences. A gap in speaker A is not a gap in the conversation, it is speaker B leaning in. My VAD saw energy, saw a dip, saw energy again, and interpreted the dip as a turn boundary. So the agent barged in on the mother mid-thought to answer a question the four-year-old had half-asked.

Week 1, later: reading the receipts

I pulled the raw audio for that 9:14am call and looked at it in Audacity like it owed me money. Then I ran our VAD offline, frame by frame, and logged every speech/no-speech flip with a timestamp.

Here is roughly what the first 6 seconds looked like once I lined it up:

0.00s  speech    (mother: "hi I'm calling about the")
1.42s  speech    (kid, overlapping: "MOM can we")
1.80s  silence    <- 240ms dip. NOT a turn end.
2.05s  speech    (mother continues: "about the deposit on")
3.10s  silence    <- 90ms. breath.
3.20s  speech    (father, low: "the Oakwood place")
4.60s  silence    <- 810ms. agent fires here. too late, wrong context.

The agent had already committed to a response at the 1.80s mark internally, buffered it, and then a second endpoint at 4.60s made it dump the whole thing. It answered "the deposit" question using audio that had three speakers braided together. The transcript it sent to the LLM was word salad, because our ASR was single-channel and had no idea two mouths were fighting for the same 8kHz of bandwidth.

Two problems, not one. First, I was detecting speech but not detecting who. Second, my endpointing logic assumed silence meant "conversation turn over" when it often just meant "this one speaker took a breath."

Week 2: VAD is necessary, not sufficient

First fix was the easy one. I had been using the VAD that shipped with WebRTC (the old GMM-based one). It is fast and it is fine for gross energy gating, but it flaps a lot on overlapped speech and car noise. I swapped the gate for Silero VAD, which is a small neural model and much steadier on noisy input.

One thing that bit me: Silero VAD (v4 and v5) wants exactly 512 samples per chunk at 16kHz. That is 32ms. Not 30, not 480 samples. If you feed it the wrong window it silently gives you garbage probabilities. Ask past-me how he knows.

import torch
import numpy as np

model, utils = torch.hub.load(
    repo_or_dir="snakers4/silero-vad",
    model="silero_vad",
    trust_repo=True,
)

SAMPLE_RATE = 16000
CHUNK = 512  # Silero requires exactly this at 16kHz. 32ms.

def speech_probs(pcm_f32: np.ndarray):
    """Yield (t_seconds, prob) for each 32ms frame."""
    for i in range(0, len(pcm_f32) - CHUNK, CHUNK):
        frame = torch.from_numpy(pcm_f32[i : i + CHUNK])
        prob = model(frame, SAMPLE_RATE).item()
        yield (i / SAMPLE_RATE, prob)

Cleaner probabilities helped. The agent stopped triggering on tire noise. But it still could not tell the mother from the kid, so it still answered the wrong person. VAD tells you that someone is speaking. It never tells you who.

Week 2, the 11pm session: diarization

For "who," I reached for pyannote.audio. It does speaker diarization: given a chunk of audio, it returns time-stamped segments each labeled with a speaker id (SPEAKER_00, SPEAKER_01, and so on). It is not magic and it is not free (you run it as a heavier model, and on a live call you run it on a rolling window, not the whole call), but it was the piece I was missing.

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token=HF_TOKEN,
)

# rolling window of the last ~8s of the call
diarization = pipeline({"waveform": window_tensor, "sample_rate": 16000})

for turn, _, speaker in diarization.itertracks(yield_label=True):
    print(f"{turn.start:.2f}-{turn.end:.2f}  {speaker}")
    # 0.00-1.60  SPEAKER_00   (mother)
    # 1.42-1.95  SPEAKER_01   (kid, overlaps SPEAKER_00)
    # 3.20-4.55  SPEAKER_02   (father)

Now I could see the overlap explicitly. SPEAKER_01 starts at 1.42s while SPEAKER_00 is still going until 1.60s. That 180ms of true overlap is exactly what the naive endpointer had misread as a turn boundary.

Week 3: turn-taking that respects overlap

The real fix was not any single model. It was rewriting the endpointing logic to combine three signals instead of one:

Is anyone speaking right now (Silero VAD probability over a short window).
Who is the primary speaker (the diarization label with the most energy in the current window).
Has the primary speaker actually yielded (silence from that specific speaker past a threshold, while no new speaker has taken the floor).

The rule that shipped, in plain words: only treat a gap as end-of-turn if the person we are tracking as the primary speaker has been silent for more than 600ms and no other speaker has started in that gap. If a new speaker starts, we do not barge in, we re-anchor to whoever now holds the floor and keep listening.

class TurnTaker:
    def __init__(self, silence_ms=600):
        self.silence_ms = silence_ms
        self.primary = None
        self.last_primary_speech_t = None

    def update(self, t, speaking, primary_speaker):
        if speaking and primary_speaker is not None:
            if primary_speaker != self.primary:
                # floor changed. someone new is talking. do NOT interrupt.
                self.primary = primary_speaker
            self.last_primary_speech_t = t
            return "listening"

        if self.last_primary_speech_t is None:
            return "listening"

        gap_ms = (t - self.last_primary_speech_t) * 1000
        if gap_ms > self.silence_ms:
            return "end_of_turn"   # safe to respond now
        return "listening"

It is not sophisticated. It is a state machine that refuses to speak until one specific human has clearly stopped and no one else has jumped in. That single change took the "agent talks over the caller" complaints from most calls in the pilot to roughly one in a hundred over the next two weeks on our deployment. Not zero. One in a hundred. Overlap is genuinely hard and I stopped pretending I would solve it completely.

What shipped, and what I would tell past me

What shipped: WebRTC for transport, Silero VAD as the fast speech gate, pyannote.audio for diarization on a rolling 8-second window, and a turn-taking state machine that anchors on the primary speaker and waits for a per-speaker 600ms silence before responding. Diarization runs slightly behind real time, so I let it correct the primary-speaker label a beat late rather than blocking on it. Good enough.

What I would tell the version of me giving that flawless Friday demo:

The demo is a lie you tell yourself. One clean speaker in a quiet room is not your product, it is your best case, and your best case will never call the support line. Real audio arrives with three people in a moving car and a codec that already mangled it.

Silence is not a turn. A dip in energy means one mouth paused, nothing more. Do not let your agent treat a breath as an invitation.

And measure the thing that actually hurts. I spent two days optimizing response latency (the 800ms everyone loved in the demo) when the real defect was that the agent was fast at answering the wrong person. Fast and wrong is worse than slow and right on a phone call. Slow the agent down until it is sure whose turn it is, then make it fast.

The 9:14am call is still in my logs. I keep it around. It is the most honest test case I have.

The call failed on turn nine. My eval gave me one number for the whole call.

Marcus Chen — Thu, 13 Aug 2026 21:54:35 +0000

The transcript was fourteen turns long and the score was 0.62.

That is the entire output. One float, one call, and a rubric that said something like "did the agent resolve the customer's issue." It did not. Score 0.62, below our 0.7 bar, test red, and I am supposed to go fix it.

Fix what? The call opened fine. The agent got the account number right, pulled the right policy, answered two questions correctly. Somewhere in the middle it went sideways, and by turn fourteen it was confidently offering a refund on a plan that does not have refunds. A single number for a fourteen-turn conversation tells you the call was bad. It does not tell you when it became bad, and "when" is the only thing that maps to a code change.

Week 1: reading transcripts like a chump

The first week I did what everyone does. I read them.

Forty-one failed calls, top to bottom, with a notepad. It works, in the sense that a human reading a conversation can usually spot the moment it turns. It took me somewhere between four and nine minutes per call depending on length, and by call twenty I was skimming, which is the point where the method quietly stops working and you do not notice.

Worse, my judgements were not stable. I re-read six calls I had already annotated, blind, three days later. On four of them I picked the same turn. On two I picked a different one, and in both cases the two candidate turns were three apart. Four out of six is not a rate I would put in a report, and with six calls it is barely a number at all. It was enough to stop me trusting the notepad.

The six-hour regression that we fixed by reverting everything

The thing that changed my approach was an on-call page that had nothing to do with evals.

We had a regression, calls degrading in production, and the only signal was that the mean conversation score had dropped from 0.81 to 0.74 over about six hours. Seven points, across every call. Nobody could say which part of the conversation got worse, so nobody could say which of the four changes that shipped that day did it. We reverted all four. It worked, and it taught me nothing, and I spent the next morning re-landing three of them one at a time.

That is when I wrote down the actual requirement: I need a score that is attached to a turn index, not to a call. Everything else is downstream of that.

The trick: score the prefixes, not the call

The method that ended up working is embarrassingly simple, and it is the one part of this post I would actually defend.

You already have a scorer that takes a conversation and returns a number. Do not write a new one. Run the one you have against every prefix of the conversation: turns 1 through 1, turns 1 through 2, turns 1 through 3, and so on. You get a curve instead of a point. The turn where the curve falls off is the turn that broke the call.

def prefix_scores(turns, score_conversation):
    """score_conversation(list_of_turns) -> float in [0,1], grading the LAST
    turn it is given in the context of the ones before it.
    Returns [(k, score_of_turn_k_given_turns_1_to_k), ...] for k = 1..len(turns)."""
    return [(k, score_conversation(turns[:k])) for k in range(1, len(turns) + 1)]


def biggest_drop(curve, min_drop=0.15):
    """The turn index with the largest single-step decline in score."""
    drops = [(curve[i][0], curve[i - 1][1] - curve[i][1])
             for i in range(1, len(curve))]
    turn, drop = max(drops, key=lambda t: t[1])
    drop = round(drop, 3)          # 0.70 - 0.55 is 0.1499... in binary floating point
    return (turn, drop) if drop >= min_drop else (None, drop)


# the fourteen-turn call from the top of this post
curve = prefix_scores(call.turns, rubric_scorer)
print(biggest_drop(curve))     # (9, 0.31)

Turn nine. The agent had been asked whether the customer could cancel and get money back, and it answered from the wrong policy document. Every turn after nine is built on that mistake, which is exactly why the whole-call verdict was bad and exactly why it could not tell me anything: an outcome rubric grades the destination, and once the conversation is pointed somewhere wrong at turn nine, the destination is wrong no matter which turn did the pointing.

The prefix curve for that call, rounded:

Turns 1 to 4: 0.91, 0.89, 0.90, 0.88
Turns 5 to 8: 0.86, 0.85, 0.87, 0.84
Turn 9: 0.53
Turns 10 to 14: 0.51, 0.49, 0.47, 0.44, 0.58

These are turn-local scores, not the gate's number, and the distinction matters for reading the graph. The gate's whole-call verdict on this conversation was 0.62. No point on the curve is that number and none of them should be, because they answer a different question: each one asks whether the agent's most recent turn was right given everything said so far.

Turns 10 through 13 do not just stay bad, they get slightly worse each time, and the slope is worth a caveat. My reading is escalating commitment: each of those turns is graded on its own merits, and on its own merits each is a bigger claim than the one before it. Turn 10 asserts the refund, turn 11 quotes an amount, turn 12 promises a timeline, turn 13 reads out a confirmation number. Nothing is carried forward by the scorer; the agent is simply wrong about more, more specifically, each time it opens its mouth.

I should be honest that this is a reading of four points from one call and not a result. It could as easily have gone the other way: a rubric asking whether the latest turn was correct and appropriate might reasonably treat "here is your confirmation number" for a refund that does not exist as a second cliff rather than three points worse than the turn before, since inventing a confirmation number is a different severity class from repeating a wrong policy. I got the gentle ramp and I do not have a mechanism that predicts gentle over cliff. The test is sitting there in the other 40 calls, which should show a ramp where the agent escalates and a plateau where it just repeats itself, and I have not run it.

Then look at turn 14, which goes back up 14 points against turn 13. That is the closing turn, and my rubric scores a turn partly on whether it is well formed: acknowledges the customer, summarises, offers a next step. The agent did all three, on top of a wrong answer, and got paid for it. Some fraction of what my scorer measures is how gracefully the agent delivers bad information, and I would not have found that without the curve.

The fortnight I spent not trusting it

Cost first, because this is the objection I would raise.

Prefix scoring is O(n) calls to your scorer for an n-turn conversation, so a fourteen-turn call costs fourteen judge invocations instead of one. Across the 41 failures that was turn for turn about 470 extra judge calls. At the model we use for grading that was small money and roughly nine minutes of wall clock, run in parallel. On our full nightly suite it would not be small, which is why we do not run it there: prefix scoring is a debugging tool that runs on failures, not a gate that runs on everything. The gate still emits one number per call. When the gate goes red, the debugger goes and finds the turn.

You can also do it in log(n) instead of n if you bisect: score the first half, and if it is already bad recurse left, otherwise recurse right. I tried it. It found the same turn on 34 of the 41 calls and a different one on 7, and every one of the 7 was a call with two separate problems, where bisection commits to a side early and never sees the other one. Full scan for debugging, bisection if you are impatient and know your calls fail once.

Now the part that lies to you, and it took me a fortnight to see it.

A prefix is not a conversation. When you score turns 1 through 5 in isolation you are asking your rubric to grade a call that appears to end at turn 5, and most rubrics have opinions about endings. Mine did. "Did the agent resolve the issue" scores an unfinished conversation harshly for the simple reason that nothing has been resolved yet, so every early prefix carried a penalty that had nothing to do with quality. My first version of this curve sloped downward everywhere and I nearly threw the method out.

The fix was to grade prefixes against a rubric that asks a turn-local question instead of an outcome question. Not "was the issue resolved," which only makes sense at the end. Something closer to "given everything said so far, was the agent's last turn correct and appropriate." Same scorer, different prompt, and the curve went flat-then-cliff instead of monotonically down. The rubric you use for the gate is very likely the wrong rubric for the curve, and reusing it is what makes the method look broken.

Worth being explicit here, because I have argued something that sounds like the opposite. A few weeks ago I wrote about a seven-turn call where every turn graded in isolation was correct and the call still failed, and I used it to argue against turn-level grading. I still think that is right about grading turns in isolation, which is what that system did: it handed the judge one turn with no history. On turn four the agent confirmed a Tuesday to a caller who had said earlier in that same call that she could not do Tuesdays, and turn four read as a perfectly good confirmation to anything that could not see the turn where she said it. The rubric here is different. It grades the latest turn conditioned on the whole prefix, which is exactly the information the isolated version was throwing away, so it should have caught that one. I have not gone back and run it on that call, and I should. What I got wrong in July was blaming the granularity when the problem was the missing context.

Three more places it misleads. Turns where the agent says almost nothing ("sure, one moment") score noisily because there is very little to grade, and I now skip any agent turn under about five words rather than trust its number.

The min_drop threshold has a blind spot I should name, since it is the same shape as the bug that started all this. A call that degrades gradually, 0.84 to 0.71 to 0.58, has no single step reaching 0.15, so the function returns nothing at all and reports the largest drop it saw, 0.13, even though the call lost 26 points end to end. A slow slide is invisible to a detector that only looks one step at a time. Looking at the curve rather than the returned index catches it, which is an argument for plotting the thing rather than trusting the number that comes out of it.

And a conversation that fails because of something the agent never said, an omission rather than an error, does not produce a cliff at all. The curve just sits slightly low the whole way. I have not solved that one. Omissions remain the failure class I still find by reading.

What shipped, and what I'd tell past me

What shipped: prefix scoring as a debug command, run on demand against failed calls, with a turn-local rubric that is versioned separately from the gate rubric. The output is a turn index and a drop magnitude. It goes in the incident notes. Time from "this call failed" to "this turn, this cause" went from four to nine minutes of reading down to well under a minute.

I owe you a number on its reliability, because I spent a whole section above complaining that my own labels did not reproduce and it would be cheap to skip the same test on the tool. Temperature 0 does not buy you determinism here, incidentally. It makes the sampler greedy, which removes the sampling noise and nothing else. Two things still move a score between replays: floating-point reduction in the serving stack is not associative, so a change in how your request gets batched with other people's can shift the logits enough to flip an argmax at a near-tie, and the provider can move the model under a stable name. Both are outside your process. So it has to be measured rather than assumed.

I replayed all 41 calls three times at temperature 0. The identified turn was stable on 39 and moved on 2. Both of the unstable ones had their two largest candidate drops within about 0.04 of each other, so the detector was picking between near-ties rather than the judge being wildly inconsistent, and both of those calls show two visible steps on the curve rather than one cliff. That is a failure mode you can see, which is the property I actually wanted.

Second thing that shipped, and honestly the bigger win: when the mean score moves in production, we now re-run prefix scoring across a sample of the affected calls and look at the distribution of drop-turns. A regression concentrated at turn 2 and a regression spread evenly across turns 4 to 12 are different bugs with different suspects. I have not been able to go back and test that against the six-hour incident, because the affected calls aged out of our retention before I built any of this. It is the first thing I will run the next time the mean moves.

What I would tell past me: the granularity of your score is a design decision, and defaulting to one score per conversation is one of the choices, however little it feels like choosing. I spent a week reading transcripts because my tooling handed me a float and I assumed that was the shape the answer came in. It was just the shape my scorer happened to emit. The conversation was always a sequence and the failure was always at an index, and I could have asked for the index at any point in that week.

The other thing I would tell him is that the number going back up at turn fourteen was the tell. A score that improves at the end of a call that failed is measuring the shape of the answer as much as its content. I looked at that number for a week and read it as noise.

Two weeks before launch, every turn was green and the call still died

Marcus Chen — Mon, 10 Aug 2026 22:27:06 +0000

The dashboard was a wall of green. Word error rate under 5 percent. Intent classification at 94 percent on our eval set. Response appropriateness, graded by a rubric we trusted, sitting comfortably in the high 80s. By every number we tracked, the scheduling agent was ready to ship.

Then I sat in on the recordings.

A woman called to reschedule a dentist appointment. The agent transcribed her perfectly. It caught the intent (reschedule) on the first try. It offered times. Every single turn, if you froze it and graded it in isolation, was correct. On turn four it confirmed "Tuesday the 14th" when she had asked for the 14th but had earlier said she could not do Tuesdays. Small slip. The agent did not catch it. She did, sort of, and got confused, and re-explained, and the agent, now anchored on the 14th, kept steering back to it. Turn seven, she said "you know what, I'll just call the front desk." Click.

Every turn passed. The call failed. And nothing in my green dashboard knew it had happened.

The number that was lying to me

Here is what I had gotten wrong, and I think a lot of voice teams get it wrong the same way. I was measuring quality at the turn level and quietly assuming it would add up to quality at the call level. It does not. Turn-level metrics and outcome-level success are different quantities, and treating one as a proxy for the other is the bug.

The assumption hiding underneath a per-turn average is independence. When you report "94 percent turn accuracy," you are implicitly treating each turn as its own little exam. But a conversation is not a set of independent exams. It is a chain. The user's turn 5 depends on your turn 4. If turn 4 quietly plants a wrong assumption, turn 5 is now operating on bad state, and no amount of local correctness on turn 5 saves the call. Errors do not average. They compound.

Watch what that does to the math. Suppose, generously, that every turn is 95 percent correct and, even more generously, that the turns really were independent. The probability that a whole conversation of n turns is clean is 0.95^n, not 0.95.

def all_turns_correct(per_turn_accuracy: float, num_turns: int) -> float:
    """Probability every turn in a session is correct,
    under the (false but instructive) independence assumption."""
    return per_turn_accuracy ** num_turns

for n in (1, 5, 10, 20):
    p = all_turns_correct(0.95, n)
    print(f"{n:>2} turns: {p:.2%} of sessions fully clean")

Run it:

 1 turns: 95.00% of sessions fully clean
 5 turns: 77.38% of sessions fully clean
10 turns: 59.87% of sessions fully clean
20 turns: 35.85% of sessions fully clean

A 95-percent-per-turn agent has roughly a 60 percent chance of getting through a 10-turn call without a single slip. My real agent was worse than 95 on the turns that mattered, and calls routinely ran past 10 turns. The green dashboard and the dead call were both telling the truth. They were just measuring different things, and I had confused one for the other.

And the independence assumption makes that estimate optimistic, not pessimistic. Real errors are correlated in the worst direction. One wrong slot value does not just cost you that turn, it poisons the turns downstream that build on it. So 0.95^n is a ceiling on how well things go, not a floor.

The other half: patience is a budget

The compounding math explains why clean calls are rarer than turn accuracy suggests. It does not fully explain why calls fail, because most failed calls do not end in some dramatic model breakdown. They end the way the dentist call ended: the human runs out of patience and leaves.

A user does not have infinite turns in them. Every repeated question, every "sorry, I didn't catch that," every loop back to a thing they already said, spends down a budget. The task can be technically still-recoverable at turn 7 and still be over, because the person on the other end has decided you are not worth turn 8. Your agent never registered a failure. The transcript just stops.

This is why I stopped trusting any metric that could not see the whole call. The unit of success for a voice agent is not the turn. It is the session, judged against what the caller actually called to do.

Measure the thing the caller wanted

Task-oriented dialogue research has worked at this altitude for years, and it is worth borrowing the vocabulary. The MultiWOZ line of work evaluates dialogue systems against the user's goal, not the utterance: a task-success notion of whether the system actually provided the entity and information the user asked for, with the attributes they requested. Correctness is defined at the level of the goal. The dataset and its task-oriented evaluation are described in Budzianowski et al., "MultiWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling" (https://arxiv.org/abs/1810.00278).

You do not need their dataset. You need their altitude. For a production voice agent, define, per call, a binary (or small-ordinal) outcome that answers: did the caller accomplish what they called to do?

For our scheduler that meant: was an appointment actually booked, moved, or cancelled in the backing system, matching the constraints the caller stated, without a human agent picking up the pieces afterward? That is checkable. The booking system knows. The handoff log knows.

Then instrument it. The point is to log a session-level outcome alongside the turns, and to log where calls die, not just whether they die.

from dataclasses import dataclass, field
from enum import Enum

class Outcome(str, Enum):
    COMPLETED = "completed"        # caller's goal achieved in the system of record
    ABANDONED = "abandoned"        # caller hung up before resolution
    HANDOFF   = "handoff"          # escalated to a human
    FAILED    = "failed"           # ended without the goal met

@dataclass
class SessionTrace:
    session_id: str
    intent: str                    # what they called to do
    turns: list = field(default_factory=list)
    outcome: Outcome = Outcome.FAILED
    last_state: str = "greeting"   # dialogue state when the call ended

    def log_turn(self, state: str):
        self.turns.append(state)
        self.last_state = state

def conversation_success_rate(traces):
    done = sum(t.outcome == Outcome.COMPLETED for t in traces)
    return done / len(traces) if traces else 0.0

def abandonment_by_state(traces):
    """Where do dying calls die? Group abandons by last dialogue state."""
    counts = {}
    for t in traces:
        if t.outcome == Outcome.ABANDONED:
            counts[t.last_state] = counts.get(t.last_state, 0) + 1
    return dict(sorted(counts.items(), key=lambda kv: -kv[1]))

Two numbers fall out, and they are the two I actually steer by now. Conversation success rate is the headline: of everyone who called to do X, what fraction left having done X. Abandonment-by-state is the diagnostic: it points a finger at the exact dialogue state where people give up. When we ran it, the abandons piled up on one state, the confirmation step, which is exactly where the Tuesday slip lived. The turn metrics had been averaging that pain into invisibility.

None of this replaces turn-level metrics. Word error rate still matters. Intent accuracy still matters. They are how you debug why a session failed once you know it did. What they cannot do is tell you whether the call was a success, because you cannot read call success off a single turn. You can only read it off the whole call.

What shipped, and what I'd tell past me

We slipped the launch by a week. We wired the booking system's ground truth back into our eval as the session outcome, replayed a few hundred recorded calls against it, and watched conversation success rate come in well below what the turn dashboard had implied. That gap was the whole story. We fixed the confirmation state (make the agent re-check stated constraints before locking a slot, not after), and the abandonment cluster on that state shrank.

If I could hand one note back to the version of me staring at the wall of green, it would be this: a per-turn average is a measurement of your model's reflexes, not of your user's success. They are correlated, but the correlation gets weaker with every turn, because errors compound and patience runs out. Pick the outcome the caller actually wanted, make it checkable against a system of record, and measure at the level of the whole call. Log where calls die, not just that they scored well while dying.

The dentist call still bothers me. Every turn was correct and the woman still hung up and drove to a phone. The agent never knew it lost. Now it would.

The guardrail fired at 1.4 seconds. The caller had heard the sentence at 1.1.

Marcus Chen — Mon, 10 Aug 2026 22:23:56 +0000

Two weeks ago I wrote about putting a guardrail in front of our voice agent, on the input, where a caller had talked the model out of its own refund policy. This is the other half of that job, and it is the harder half. Everything below is about the output side, and about one number I had never measured.

Here is the shape of it. Our output rail worked. It fired, it logged, it named the rule, and the log has a timestamp on it. Then I put that timestamp next to the rest of the turn. The sentence went to TTS at 900 milliseconds, the caller's handset started playing it at 1,100, and the rail fired at 1,400. Three hundred milliseconds behind the ear it was supposed to protect.

Two days arguing with a trace

I spent most of a Wednesday convinced I had a bug in the rail. The rule was correct, the scanner was correct, and the block was in the log where a block should be. What I could not explain was why the call recording had the agent saying the thing anyway.

The recording is the part that settles arguments. You can read a trace ten times and talk yourself into a story. Then you play sixteen seconds of audio and hear your agent say a sentence, and hear the caller react to it, and the story stops working.

The rail had not failed. It had run late, and late on a phone call is a different failure from the one I was looking for. I had been using "blocked" to mean two things for months.

Input rails have time, output rails do not

The asymmetry took me a while to state cleanly, so let me state it cleanly here.

When you scan an input, you have the whole duration of the caller's utterance to work with. They are still talking. Every millisecond they spend finishing their sentence is a millisecond your scanner spends for free. Input safety is a scheduling problem with a generous budget.

Output is the opposite. The model produces text, the text becomes audio, the audio plays, and every one of those steps is moving away from you. There is no point after which you get to reconsider, because audio is irreversible. Once a sample has played there is no call you can make that unsends it, and the correction you play afterwards is a second thing the caller hears rather than a replacement for the first. The best your rail can do, once it is late, is apologise on your behalf.

Which reframes the question I should have been asking all along: how much did the caller hear before the rail was allowed to have an opinion?

Where a voice pipeline actually commits

Voice stacks commit earlier than most people picture, and they commit somewhere upstream of the speaker.

The usual arrangement: the model streams tokens, an aggregator buffers them until it has something worth speaking, and in every implementation I have worked on that unit is a sentence. That sentence goes to TTS. TTS returns audio. The audio goes out.

The commitment happens at the aggregator. The moment a sentence is handed to the synthesiser you have spent it, because everything downstream is a pipeline you can stop but not rewind. You can cut the audio mid-word, and we do, and stopping halfway through "your balance is forty-two thousand" is not a save.

Which means there is a number sitting in every voice stack that nobody I have asked has measured: the size of the text block your output rail waits for, compared against one sentence. If the first is larger than the second, the first sentence goes out unchecked. That is not an occasional failure. It happens on every turn, by construction.

And the first sentence is where a voice agent puts the answer. It is where it confirms the appointment, states the balance, or repeats the thing from the record. Our incident was not in some rambling fourth paragraph.

The number I should have been logging

Here is the instrumentation, because this is the part I would actually hand someone.

We already had a timestamp for when the rail fired, because the rail wrote one. We had nothing for when the audio reached the caller, and you cannot get that from the server. Server-side, everything looks fine: we stopped generating, we cancelled, the log is clean. The event I needed was on the far end.

So we made the client emit two things per turn: the moment its playout buffer started on a given sentence, and the moment it drained. Then one derived field per rail trigger:

fired_minus_played_ms = rail_fired_at - audio_started_playing_at

Positive means the caller heard it first. That is the whole metric. It took an afternoon.

Our first week of data was not comfortable reading. A meaningful share of triggers came back positive, which is to say a meaningful share of the blocks on my guardrail dashboard had prevented nothing at all. Before that field existed, every one of them had been counted as the system working.

I would take a dashboard with a smaller, honest block count over one that quietly counts arrivals.

What the tooling does and does not decide for you

I went back through the options I weighed in the earlier post, this time reading their streaming behavior rather than their feature lists. Capabilities below are as of July 2026, read from source or from the vendor's own docs.

NVIDIA NeMo Guardrails applies output rails over token chunks, defaulting to 200 tokens with 50 carried for context. Two details matter more than the size: streaming output rails are off unless you enable them, and stream_first defaults to true, meaning chunks are streamed before the rails are applied. Guardrails AI takes a sentence-shaped approach instead, accumulating text in validate_stream and validating once more than one sentence has arrived. Future AGI's gateway checks accumulated text every 100 characters and can either stop the stream or append a disclaimer; like NeMo's it is opt-in, and in its case at two levels, since both the guardrail engine and the streaming checker default to disabled in the gateway config. Llama Guard 4 is a model rather than a policy, a fine-tuned Llama 4 that scores input and output against MLCommons categories, so the granularity is whatever you hand it. Meta ships the orchestration separately, in LlamaFirewall, which describes itself as a policy engine that coordinates several scanners and is built for low-latency environments.

All of that is readable in about ten minutes if you want to check me: OutputRailsStreamingConfig in rails/llm/config.py in github.com/NVIDIA/NeMo-Guardrails, validate_stream in validator_base.py in github.com/guardrails-ai/guardrails, stream_checker.go alongside DefaultConfig() in github.com/future-agi/future-agi, and the LlamaFirewall README in github.com/meta-llama/PurpleLlama.

Lakera is the one that made me feel slow. Their Guard docs have a section on screening streamed output that recommends sentence-level chunking for accuracy, a ten-token minimum for incremental snapshots, and a delay buffer that screens a chunk before showing it, which they say costs latency and is the right default when safety outranks speed. That is the conclusion I arrived at over an incident and two days of trace-reading, and they had already written it down. The only thing voice adds is that the display in "screen before display" is a speaker, so the deadline is harder and the buffer costs you barge-in budget rather than a flicker.

None of that picks your block size for you. The tools give you a dial and a default. The default assumes a user who is reading. Only you know whether yours is listening.

What shipped

We moved the rail in front of the TTS handoff and paid the latency, which on our stack ran 120 to 300 milliseconds depending on which scanner was in the path. We covered most of that with a fixed, hardcoded opener while the first real sentence gets checked, which is the same filler trick voice teams already use for model latency, pointed at a safety budget instead. The coarse end-of-stream check stayed, because it catches things a sentence-at-a-time view misses, but it now writes to the incident log rather than to the prevention count. And the fired-minus-played field ships on every trigger.

The engineer who reviewed that guardrail config and closed the ticket was me, and here is what he had wrong. He read the config as a promise about what the caller would hear. It was a promise about what the model would finish generating. In every system I had built before this one those were the same sentence, so I never learned to tell them apart. On a phone call they come apart by about three hundred milliseconds, and that gap is the only part of the conversation the caller remembers.

If you run a voice agent, go and find out how much of its first sentence has ever been checked. I was six months in before I asked, and the honest answer was none of it.

Four minutes with the bot, and the human opened with "How can I help you today?"

Marcus Chen — Mon, 10 Aug 2026 06:58:06 +0000

We were proud of the transfer. It worked on the first try, the call reconnected cleanly, nothing dropped, and the queue wait that quarter was under ten seconds. We had spent a sprint on it.

Then I listened to one.

The caller had spent just over four minutes with the voice agent. She had given her account number, confirmed her address, described a duplicate charge, and read out the last four digits of the card it hit. The agent could not issue the refund, which was correct, that path needs a human. So it transferred her.

The human picked up and said "Hi, thanks for holding, how can I help you today?"

She said all of it again. The account number, the address, the duplicate charge, the last four digits. Four minutes of work, done twice, and the second time by a person who costs us money per minute.

What actually crosses the transfer

The thing I had not understood is that a warm transfer moves the call, not the conversation.

The call is a SIP leg. Moving it is a solved problem and that is the part we had spent the sprint on. The conversation lived somewhere else: in the agent's session state, in a service the contact-centre desktop had never heard of. What the human's screen showed when the call arrived was what it had always shown, a caller ID and a queue name.

So the human was not being lazy. They opened with a wide-open prompt because that is the only safe move when your screen tells you nothing. Anything more specific risks guessing wrong at a caller who is already annoyed.

I should be clear that the mechanism for carrying context across a transfer is not something anybody needs to invent. Screen-pop, attached data on the call, user-to-user information on the transfer itself: contact centres have had these for decades, and every platform I have worked with exposes some version of them. We had simply never wired the voice agent into any of it. The agent had been built to handle calls, and the transfer got treated as its exit door.

The channel existed. What follows is about what turned out to be worth putting in it, which was much less than I expected.

Week one, and the fix that changed nothing

We did the obvious thing, which was to put the transcript on the screen.

The whole four-minute transcript, in a panel, on screen-pop. It was live within a few days. Average handle time on the human leg did not move, and when I sat with the support team I understood why in about a minute.

An agent has a beat of about two seconds between the call arriving on their headset and having to speak. Nobody reads four minutes of dialogue in two seconds. They also cannot skim it, because the useful facts are scattered through it in whatever order the caller happened to say them. Two of the people I watched had already closed the panel by the time they said hello. One told me she had stopped opening it in the first week, because reading it while listening to a live caller made her lose the thread of what the caller was saying now.

We had moved the data and left the work where it was.

The number that finally made the case

I could not get anything else prioritised on the strength of one recording, so we built a measurement.

We called it the re-ask rate, and the definition took three tries. It ended up on the caller's channel: the share of transferred calls where, in the first sixty seconds of the human leg, the caller re-states information the agent had already captured. That is the same side of the call I count repair on, scoped per call and across the transfer instead of within a single leg.

Putting it on the human's channel was the first two tries, and it fails twice over. It misses the commonest case, because in our opening recording the human asks for nothing specific: they say "how can I help you today" and the caller volunteers everything unprompted. It also punishes the eventual fix, because a human holding a card starts saying the account number out loud to confirm it.

The caller channel has a hole of its own, which took the third try to close. When the human reads a fact back and the caller says "yes, 4471", that is an exact entity match on the caller's side, and it is a confirmation rather than a re-statement. So a caller-side match is excluded when the same entity appeared on the human channel in the immediately preceding turn. Without that clause the metric gets worse exactly as the experience gets better.

It needs dual-channel audio, which we already had for quality monitoring, and is otherwise cheap. We had the entities the agent extracted, account number, address, the disputed amount. Matching those against the caller-side transcript of the first minute is mostly string comparison. Where a match was genuinely ambiguous we excluded it from the numerator and hand-reviewed a sample each week, to check the exclusions were not hiding a pattern.

The first run came back at 62 percent, with the transcript panel already live. Almost two thirds of transferred calls had the caller repeating something the system already knew, and the median call had two separate facts in it.

That number did what the recording could not. Nobody argues with 62 percent.

Why it stayed invisible is more specific than "we had no metrics", and I want to be accurate, because we did have one that crossed the transfer. Our session-outcome metric knew perfectly well when a call had handed off. Escalation was one of its outcome values and it counted against the agent. What it recorded was that the handoff happened. Nothing looked at what happened inside the human leg afterwards, so the agent's numbers ended at the transfer, the human leg's handle time started at it and was benchmarked against other transferred calls carrying the same defect, and the waste sat in the join.

That join had no owner. The voice team's dashboard was accurate, the contact-centre team's dashboard was accurate, and the broken thing was on neither.

What we shipped

Not the transcript. A handoff card, three lines, rendered before the human's phone rings.

The first line is what the caller wants, in the agent's words, one sentence. The second is the facts already confirmed, labelled, so the human can open with "I have your account here". The third is why the call transferred, which is usually the one thing the agent could not do, and this mattered more than I expected: knowing the agent had already failed at something tells the human where not to start.

We also stopped auto-populating anything the agent had captured with low confidence. Handing a human a wrong address confidently is worse than handing them nothing, because they will read it back and be wrong in front of the caller. Below the confidence threshold the field is simply absent from the card, with no caveat, because a caveat is one more thing to read inside that same two-second beat.

Re-ask rate went from 62 percent to 18. Most of what is left is one case: transfers that touch payment details, where the human has to re-verify identity from scratch whatever the screen says. Those run at about one call in seven of our transfers, which puts a floor somewhere near 15 percent. The remainder is a small tail, and part of that tail we inflicted on ourselves with the omission rule above, since a field the card leaves out looks identical to a field nobody captured, so the human asks. I took that trade. Eighteen is close to our floor, and I stopped pushing.

Average handle time on the human leg came down by 47 seconds, which is roughly what the arithmetic predicts and the main reason I believe it: a 44-point drop in calls that were re-asking a median of two facts, at a bit under two minutes to ask for two facts, wait while the caller finds them, read them back and confirm them.

Which brings me back to the caller in the opening. Her transfer was a payment dispute, so she sits in exactly the class the compliance rule covers. The card would not have saved her the identity check, and that check takes the account number and the address, so she would have given those again either way. What it would have saved her is the dispute: the duplicate charge, the card digits, the whole explanation she had already given once to a machine that understood it perfectly.

The part I keep thinking about

The voice agent was never the problem in this story. It captured everything correctly, it made the right call about what it could not do, and it transferred cleanly. Every metric pointing at it was green and every one of them was honest.

We had built the agent to handle calls. The business needed something that handed calls over well, and those have different success conditions, only one of which was on anybody's dashboard.

If you are running a voice agent in front of humans, the handover is a product surface with its own failure modes, and in most shops nobody has been asked to own it.

Three things I'd say to the guy who was proud of the transfer

Find out who owns the join. No amount of instrumentation fixes that until someone's name is on the seam.

The transcript panel shipped on time, did what the ticket said, and did not move average handle time by a second. It handed a human four minutes of reading and a two-second beat to do it in.

And listen to a transferred call before you design the transfer. I keep relearning this one, which is why it goes at the end where I will see it again, and the thing I needed has been audible inside a minute every time.

A month of failed calls, and my eval had the same name for all of them

Marcus Chen — Thu, 06 Aug 2026 14:30:23 +0000

I spent a Monday morning sorting a spreadsheet that could not be sorted.

Four hundred and eleven calls from the previous month had come back below our threshold. I wanted them grouped, because I had one sprint and I wanted to spend it on whatever was biting the most callers. So I opened the export and looked for the column that says what went wrong.

There is no such column. There is a score. Every one of those 411 calls carried a number under 0.7 and nothing else, and a number under 0.7 does not tell you whether the agent talked over the caller or invented a policy.

I tried the obvious substitutes before admitting that. Sorting by score just puts the worst calls on top, and the worst calls are a mix of everything. Sorting by duration finds the ones that dragged, which is one failure mode out of a dozen. Sorting by which intent the caller came in with tells you where the failures land, not what they are, and by Wednesday I had three tabs that each answered a question I had not asked.

What I wanted was a count per reason. Twelve rows, sorted descending, so I could point at the top one on Monday and be done arguing about it.

What I had already fixed, and what it did not fix

Two weeks earlier I wrote about replacing task-success rate with repair rate: counting how often the caller has to restate themselves because the agent misheard or barrelled ahead. That change was worth making. Repair rate moves when the call is bad in the way callers care about, and task-success does not.

It also did not help me that Monday. A better number is still a number. Repair rate told me which calls were bad and roughly how bad. It had nothing to say about which of them were bad for the same reason.

That is the gap I had been calling a metrics problem for about six months. It is a vocabulary problem. Until your failures have names, you cannot count them by name, and if you cannot count them by name you cannot pick the biggest one.

The question I ended up asking five tools

So I went and read the trees. One question, asked the same way of each:

When an eval marks a case bad, what comes back, and who wrote the list of things it is allowed to say?

That second half is the one that matters and the one nobody advertises. A vocabulary you write yourself starts empty and fits your product. A vocabulary the vendor ships saves you the blank page and constrains you to their idea of failure. Both are defensible. They are very different purchases.

Everything below is from the repositories as of 4 August 2026, ordered by GitHub stars purely because that is a neutral ordering and not a ranking of fitness. File paths are there so you can check me rather than believe me.

Tool	What comes back on a failure	Who writes the label space	Nearest thing to a voice failure
Langfuse (32,498 stars, MIT core; ee/ is commercial)	A score attached to a trace or an observation, typed CATEGORICAL, NUMERIC, BOOLEAN or TEXT (packages/shared/prisma/schema.prisma:465)	You do. model ScoreConfig keeps your category names as a reusable, project-scoped object (schema.prisma:441)	Nothing prewritten. The categories column ships empty and you fill it
Promptfoo (23,920 stars, MIT)	The name of the assertion that failed, drawn from a 66-entry enum (src/types/index.ts:595)	Promptfoo writes the catalogue, you pick per test case	The closest of the five. latency, trace-span-duration and conversation-relevance are all in that same enum
DeepEval (17,398 stars, Apache-2.0)	A per-metric score plus the judge's reason string	DeepEval, as named metric modules you import	The richest multi-turn set: turn_relevancy, role_adherence, conversation_completeness, knowledge_retention under deepeval/metrics/. Conversational, not spoken
Arize Phoenix (10,896 stars, Elastic 2.0, so not an OSI licence)	A Score carrying a validated label; a label outside the declared set raises rather than passing through (packages/phoenix-evals/src/phoenix/evals/evaluators.py:766)	You declare the choices, Phoenix enforces them. Fourteen metrics ship under .../phoenix/evals/metrics/	user_friction.py, which is the only name in any of the five that is about the caller's experience of the exchange
Future AGI (1,586 stars, Apache-2.0)	A classified error with a category path, evidence spans and a suggested fix (futureagi/tracer/models/trace_error_analysis.py:91)	Future AGI, and the list is not in the repo: category is a 200-character string, not an enum (same file, line 112)	Nothing voice-shaped in the one readable taxonomy (31 subcategories, futureagi/model_hub/utils/evals.py:3066)

The thing none of them have a word for

Read down that last column. Five tools, and the two nearest hits are a latency assertion and a metric called user friction.

Neither of those is what I need. A voice agent fails by starting its sentence 300 milliseconds into the caller's. It fails by going quiet for two seconds while a tool call resolves, which on a phone line reads as a dropped call. It fails by reading a confirmation number at conversational speed to someone holding a pen. It fails by acknowledging with the same four words eleven times.

None of those are hallucinations. None are wrong tool arguments. They are the entire content of my last three post-mortems, and there is not a name for any of them in any vocabulary I read, including the two vendors that ship a prewritten failure list rather than an empty one.

I nearly wrote the wrong conclusion here. The label spaces were written for agents that type, which is what almost every agent still does. That is not the tools being bad at voice. Voice is the minority case, and the vocabularies reflect that honestly.

There are voice-native vendors in this space. Coval, Hamming and Cekura all sell testing for spoken agents, and any of them may already have solved this. All three are closed source, I could not open the tree, and I am not putting a capability claim in a table on the strength of a landing page. They are worth a demo. They are not worth a row I cannot check.

Two shapes of vocabulary, and what each one costs

The five split cleanly once you stop reading them as competitors and start reading them as two designs.

Langfuse and Phoenix hand you the primitive. Langfuse gives you a named categorical score config that lives at the project level, so agent_talked_over_caller becomes a real object other people on your team can attach to a turn. Phoenix goes one step further and refuses labels outside your declared set, which sounds pedantic until a judge invents a category at 2am and quietly splits your counts in half.

DeepEval and Future AGI hand you a filled list. DeepEval's is readable and importable, which is the version of this I would push people toward first: you can see exactly what role_adherence means before you depend on it.

Future AGI sits at the far end. Its cloud platform clusters production failures and returns a root cause with a suggested fix (futureagi.com/platform/evaluate/error-feeds). The open-source UI gates that behind a "Cloud feature" screen (frontend/src/components/oss-upgrade-gate/oss-upgrade-gate.jsx:17). As of August 2026 you cannot read the category list before you send traces.

If I were choosing today for the voice half specifically, I would take the primitive over the filled list, and Langfuse's score config is the cleanest primitive of the five. Not because it does more. Because the twelve names I actually need do not exist yet in anyone's list, so the thing I am buying is somewhere to put them.

Week two: where the twelve names attach

We wrote our own. Twelve categories, all voice, all lifted from post-mortems we had already written: talked-over-caller, dead-air-over-1.5s, confirmation-read-too-fast, acknowledgement-loop, and eight more that are specific enough to be embarrassing.

Declaring them is one call. The config is the vocabulary, and the twelve labels live inside it:

curl -X POST https://cloud.langfuse.com/api/public/score-configs \
  -u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "voice_failure_mode",
    "dataType": "CATEGORICAL",
    "description": "Turn-level voice failure taxonomy. One label per agent turn.",
    "categories": [
      {"label": "talked-over-caller",         "value": 1},
      {"label": "dead-air-over-1.5s",         "value": 2},
      {"label": "confirmation-read-too-fast", "value": 3},
      {"label": "acknowledgement-loop",       "value": 4}
    ]
  }'

Two constraints worth knowing before you name anything: the config name is capped at 35 characters, and each category carries a numeric value alongside the label, which is what you end up grouping on.

The mechanical detail that took me two tries to get right is where the label attaches. It goes on the turn, not the call. The naming was the easy half.

A call scored talked-over-caller tells you the problem happened somewhere in four minutes of audio. A turn scored talked-over-caller tells you which turn, which means you can pull the 400 milliseconds around it and listen to it. We spent the first week attaching per call and produced a leaderboard nobody could act on.

One label per turn, not a set. We tried multi-label for three days and stopped, because a turn tagged both dead-air and acknowledgement-loop makes the counts ambiguous exactly when you are trying to rank them, and ranking them is the entire point. If a turn genuinely has two, we take the one the caller reacted to.

A rough judge assigns the label on every turn and I re-label the disagreements by hand on Friday mornings. It runs about forty minutes and it is the most useful forty minutes in my week, because the disagreements are where the vocabulary is still wrong.

What shipped, and what I would tell the version of me sorting that spreadsheet

The counts were not what I expected. Dead air came third. Acknowledgement-loop, the one I would have sworn was cosmetic, came first by a distance, and it traced back to a single retry path that had been in production since May. Nine lines. It had been sitting there the whole time I was tuning thresholds.

What I would tell the guy with the spreadsheet is narrower than "go build a taxonomy". It is that your sprint goes to whatever you can count, so what you can count is the thing to fix first. I had spent six months getting better at saying how bad a call was. The change that moved what we shipped was smaller than that: I stopped grading calls and started labelling turns.

The spreadsheet still has 411 rows. It sorts now, and the top row is a retry path from May.

The voice A/B test that picked the worse agent, and won by 4 points

Marcus Chen — Wed, 05 Aug 2026 05:56:10 +0000

We ran a clean A/B test between two versions of a phone agent. Variant B won by 4 points on our success metric. We shipped B. Two weeks later the escalation rate to human agents had gone up, and the "won by 4 points" version was the reason. The test wasn't rigged. It was just the wrong shape for voice, and I'd built it out of chatbot habits.

Here's what I got wrong, in order.

Week 0: the setup that felt correct

For a chatbot A/B test the recipe is boring and reliable. Split traffic, hold everything constant except the one change, define a success metric (task completion, thumbs up, whatever), run until you have significance, ship the winner. I've done it dozens of times and it works, because a text turn is atomic. The user sends a message, the bot sends a message, and nothing happens in between, because there is no "in between."

A voice turn has an in-between. That gap is where this whole story happens, and I designed the test as if the gap didn't exist.

Week 1: what the 4 points actually measured

Here is the detail I glossed over when I set it up. Variant A and Variant B were two different agent builds, and they did not carry the same turn-detection config. B's build had a shorter endpointing threshold: it decided the caller was done talking after about 500ms of silence, where A waited around 800ms. I thought of that as a latency tweak. It is not. It changes who the agent is.

The shorter threshold did two things from one cause. It made B start answering sooner after the caller stopped, which felt snappy. It also made B treat a mid-sentence pause, the breath someone takes in the middle of "I want to cancel my... order from last week," as the end of the turn. So B interrupted people. It answered a question the caller hadn't finished asking.

And our metric couldn't see it. The callers who got cut off but whose intent was already clear still had the task marked complete, so they scored as wins. The callers who got cut off, had to repeat themselves, got annoyed, and asked for a human? A lot of those escalations happened after the task field had already flipped to done, so the metric never counted them. B scored higher on the number while quietly losing more callers, and the gap between those two facts was invisible in the dashboard.

Endpointing is a real variable in a voice test. A text A/B never has to think about it, because text turns have no silences to measure. My A/B test held the prompt constant and let endpointing float between the two builds, so I was changing two things and crediting the result to one.

Week 2: the confounds text doesn't have

Once I started pulling call recordings instead of trusting the scalar, the list of voice-only confounds got long:

Endpointing, the one that bit us. A 500ms silence threshold and an 800ms threshold are two different agents even with an identical prompt. If it differs between variants, it is part of your experiment whether you meant it to be.
Barge-in. What happens when the human talks over the agent? Cut off cleanly, keep going, or the two talk over each other for a beat. None of that shows up in a text metric.
Latency distribution, not the average. A small mean difference can hide a tail: some responses took 1.5s, and on a live call 1.5s of silence feels like the line dropped. People start saying "hello? are you there?" and the transcript fills with noise that then confuses the agent.
When you score the call. A text conversation ends and then you score it. With voice, the "task done" moment and the "caller gave up" moment can be seconds apart, and the bad part usually comes second.

None of these are prompt content. All of them can differ between two builds without anyone deciding they should.

What I'd measure instead

The fix isn't a better single number. It is treating a voice interaction as a timed, two-party process and measuring it like one.

Start by pinning the turn-taking config across variants the same way you pin the prompt. Endpoint threshold, barge-in policy, VAD settings: fix them, or you are A/B testing them by accident, which is exactly what I did. Then add interruption rate as a first-class metric, because task completion alone told me B was better and interruption rate would have told me the truth: count how often the agent started speaking while the caller was still talking. Report the latency distribution (p50, p95, p99) rather than the mean, since the tail is what makes a call feel broken. Score the call from the recording after the last turn, not at the instant a task field flips true, so the escalation eight seconds later is part of the result. And profile the timing behavior offline before you split live traffic: frameworks like Pipecat and LiveKit let you replay recorded audio through the pipeline, which is the closest thing voice has to a fixed test fixture.

That last one is the chatbot habit I miss most. In text you can freeze the input and get a deterministic comparison for free. In voice you have to manufacture that determinism on purpose, and if you skip it, the timing noise picks your winner for you.

What shipped, and what I'd tell past me

We rolled B back, pinned the endpointing config so both variants waited the same 800ms, and re-ran with interruption rate and tail latency as gates alongside task completion. The winner flipped. A modest version A that waited a beat longer and interrupted less kept more callers to the end.

What I'd tell the version of me who set up that first test: the 4-point win was real, it just measured a different agent than the one I thought I was comparing, because the two builds disagreed about when a caller was finished talking. In text, holding the prompt constant is enough to hold the experiment constant. In voice, the silences between words are part of the agent's behavior, so if you don't pin the timing, it varies on its own and takes your result with it.

Still open for me: I don't have a clean way to put "that interruption felt rude" on a scale. Task completion and interruption count are proxies for it, not the thing itself. If you've found a measurable stand-in for how an interruption actually lands with a caller, I'd genuinely like to hear it.

Our voice agent scored 91 percent. The callers still hung up angry.

Marcus Chen — Sun, 26 Jul 2026 21:15:10 +0000

Two weeks after we launched the support line, the dashboard was the color you want. Task-success rate: 91 percent. The agent booked the appointment, reset the password, quoted the balance. Green across the board. We had a wall of it.

The support queue told a different story. People were escalating to humans anyway, and when I pulled the recordings to find out why, almost none of them had failed. The agent got the job done in nearly every call I listened to. It just made the caller work for it.

That gap, between "the task completed" and "the call was good," is the thing I had measured wrong. And I had measured it wrong because I was grading a voice agent with a text agent's ruler.

What task-success hides

Task-success rate asks one question: did the agent reach the goal state. It is a transcript metric. You can compute it from the words alone. That is what makes it comforting, and it is why it misses most of what goes wrong on a voice call.

Here is a call that scores a perfect 1.0. The caller says their account number. The agent mishears one digit, reads it back, the caller says "no, seven, not eleven," the agent tries again, mishears the next field, the caller repeats the whole thing slower, and eventually they get there. Appointment booked. Task complete. From the transcript, a clean success.

From the caller's chair, that was ninety seconds of repeating themselves to a machine that would not listen. They will not call back. The transcript scored the destination. Nobody scored the road.

The metric that actually tracks "was this call good"

The thing I should have been counting has a name, and it is not mine. Conversation analysts have studied it since the 1970s. The canonical reference is Schegloff, Jefferson, and Sacks, "The preference for self-correction in the organization of repair in conversation" (1977). Repair is what people do when something in the talk goes wrong: they restate, they correct, they say "no, I meant," they slow down and try the same thing again.

Human conversations have repair too. The difference is rate and who initiates it. When a caller has to initiate repair over and over because the agent misheard, cut them off, or answered a question they did not ask, the call is bad no matter what the final state says.

So the metric I care about now is simple to define and annoyingly revealing:

repair_rate = (turns where the caller re-states, corrects, or says "no / I said")
              / (total caller turns)

You count it per call and you watch the distribution, not the average. A mean of "0.12 repairs per turn" sounds fine. The tail is where your angriest callers live: the six percent of calls where the caller had to repair four or five times before the agent caught up. Those are the ones churning, and task-success rate cannot see them because every one of them ends in success.

Two cheaper cousins are worth logging next to it. Turns-to-completion, because a booking that takes eleven turns is a worse booking than one that takes four. And interruption rate, how often the agent starts talking over the caller, which on our traffic correlated with repair more than any single ASR number did. All three are conversational, not transcript-level. All three need the audio and the timing, not just the words.

Wiring the metric in

Counting repair by hand during a post-mortem tells you what went wrong last week. To change what ships, the same count has to run against every candidate build, on calls that resemble your real traffic: the frustrated repeat-caller, the fast talker, the one with background noise. Practically that means generating those calls and scoring the audio and transcript against metrics you define, repair rate among them. Several tools now cover that ground for voice agents and are worth knowing before you build it yourself. Capabilities below are as of July 2026.

Coval builds simulation-first QA and borrows its framing from self-driving-car testing. Hamming calls itself a flight simulator for voice agents and pairs automated call generation with production monitoring. Future AGI's agent-simulate is open source (Apache-2.0): it drives a simulated caller through your agent in a LiveKit room and scores the result in its ai-evaluation library against built-in or custom metrics (github.com/future-agi). Cekura auto-generates test cases so your QA set is not just the ten calls you thought of. Maxim AI spans the wider loop, experimentation through production observability, for voice and multimodal agents.

Pick by your constraints, not by the feature grid, because none of these will tell you which metric matters for your callers. Run as many simulated calls as you like scored on task-success and you get back the same green wall I started with. Choosing the number is the part that stays yours.

What shipped, and what I would tell the version of me with the green dashboard

We kept task-success on the board, because it is a real floor and a regression in it is a real fire. We just stopped treating it as the headline. Repair rate in that tail, the calls where someone had to say it four or five times, is the number I look at first now. When a build lowers it, the calls sound better and the escalations drop, and those two things move together in a way task-success never did.

If I could go back to the engineer staring at 91 percent and feeling done, I would tell him one thing. The dashboard is green because you asked it the question a chatbot answers. Voice agents fail in the parts a transcript throws away: the timing, the talking-over, the third time the caller had to say their own name. Go count those. The color will change, and so will the thing your callers actually feel.

The 1.8 seconds after "wait": the week our voice agent refused to stop talking

Marcus Chen — Fri, 24 Jul 2026 08:01:30 +0000

The recording that finally made me understand the problem was eleven seconds long. A woman calls in to move a dentist appointment. She says "yeah so I need to push my Thursday." The agent starts reading back her options, calm and clear. Two words in, she remembers something and says "oh wait, no, actually keep Thursday, it's Friday I need." And the agent just keeps going. It finishes its entire sentence about Thursday while she is talking over it, both voices stacking into mush, and then there is a beat of dead air where you can hear her decide this is not worth it. She hangs up.

I listened to it four times. The transcript looked fine. The latency dashboard looked fine. Everything we had built to measure was green, and the call was still a small disaster.

Week 1: the numbers that lied

We had launched the appointment agent to a single clinic group the previous Monday. On paper it was healthy. Time to first audio sat around 600 ms. Our turn-detection was conservative but sane. The model rarely said anything wrong.

The one metric that bothered me was hang-ups on interrupted turns. When a caller talked while the agent was mid-sentence, roughly 22% of those calls ended in the next ten seconds. On turns where nobody interrupted, that number was near 4%. Interruption was the poison. I just did not yet know why.

My first assumption was the model. Maybe it was ignoring the interruption text, or the endpoint logic was folding two utterances into one. I spent most of Tuesday there and found nothing. The server was doing the right thing. When a caller spoke, we detected speech, we fired a cancel, we stopped generating tokens. Server-side, the agent stopped talking almost immediately.

The problem was that "server-side stopped talking" and "the caller stopped hearing the agent" were two very different moments in time.

The 3am realization: the audio was already gone

Nobody tells you this about a voice pipeline until it bites you. By the time your server decides to stop, a lot of audio has already left the building.

Trace one chunk of speech through the system. The model generates text. Text goes to TTS. TTS returns audio in frames. Those frames get packetized and sent over the network to the caller's phone. On the way, and at the very end, they land in a jitter buffer that deliberately holds a little audio in reserve so that network hiccups do not cause gaps. Then they play out through the speaker.

Every one of those stages is a small reservoir. When my server sent its cancel, the token stream stopped, sure. But the TTS had already handed me a big block of audio for the current sentence. That block was already packetized. Some of it was already in the jitter buffer on the caller's side, committed to play no matter what I did next. The caller kept hearing the agent because the agent's voice was, quite literally, already in their ear's queue.

So I instrumented the thing I should have measured from day one. I called it the barge-in tail: the gap between the moment we detected caller speech and the moment the caller's device actually went silent. I logged a timestamp when our VAD fired, and I had the client log a timestamp when its output buffer drained to zero after a cancel.

The tail was ugly. Median 1,850 ms. p95 was 2,400 ms. For almost two seconds after a caller started talking, our agent was still audibly talking back. No wonder they hung up. We had built a system that could not take a hint.

Where the two seconds were hiding

I broke the tail down by stage, and it was not evenly spread.

Our TTS was streaming in 400 ms frames. That felt reasonable when we picked it, because bigger frames mean fewer packets and less per-packet overhead. But it also meant that at any instant, we had committed up to 400 ms of a single frame that we could not easily claw back. The jitter buffer on the client was configured at 200 ms, standard and fine. And the last, embarrassing piece: when we sent our cancel, we stopped generating new audio, but we never told the client to throw away the seconds of audio it had already buffered locally for smooth playout. It played every buffered frame to completion first. That local drain was most of the tail.

We were not fighting network latency. We were fighting our own buffers, all of which were doing exactly what we designed them to do.

The fix: stop making audio, then delete the audio you already made

The change had three parts, and the order mattered.

First, when we detect a barge-in, we cancel TTS generation server-side. We were already doing this. Keep it.

Second, and this was the missing piece, we send an explicit flush command down to the client telling it to clear its playout buffer immediately, not after it drains. The audio that is already in the pipe gets dropped on the floor. When someone interrupts, we want silence right then.

Third, we shrank the TTS streaming frame from 400 ms to 120 ms. Smaller frames mean that at any instant, far less audio is committed and unrecoverable. It costs a few more packets per second. On a modern connection that overhead is noise.

The client handler ended up looking close to this:

def on_barge_in(session):
    session.tts.cancel()            # stop generating new audio
    session.audio_out.flush()       # drop frames already queued locally
    session.jitter_buffer.reset()   # clear the 200ms reserve
    session.state = "listening"
    log_metric("barge_in_tail_ms", now() - session.vad_fired_at)

The flush and jitter_buffer.reset lines were the whole ballgame. Four lines, most of a Thursday to find them.

The objection I had to answer before shipping

One of our engineers, and she was right to ask, worried that shrinking the frame and aggressively flushing would make normal speech choppy. If we clear the jitter buffer too eagerly, a real network hiccup could clip the agent's own words even when nobody interrupted.

So we scoped it. The flush only fires on a confirmed barge-in, never during uninterrupted playback. During normal speech the 200 ms jitter buffer does its job untouched. We only reach for the fire alarm when the caller is actually talking over us. We ran two days of shadow traffic listening for clipped words on non-interrupted turns and heard none. That was enough to ship.

What shipped, and what I would tell past me

We rolled it out to the same clinic group the following Monday. The barge-in tail dropped from a median of 1,850 ms to 180 ms, with p95 at 320 ms. You can hear it on the recordings now: the agent stops the instant the caller speaks.

The hang-up rate on interrupted turns fell from 22% to about 6%, roughly in line with our uninterrupted turns. The interruption poison was mostly gone. Callers still interrupted constantly, because humans do, but now the agent shut up and listened, so it stopped feeling like a fight.

If I could hand one note back to the version of me who built the first pipeline, it would be this. We spent months tuning time to first audio and never once measured how long it took the agent to go quiet, and that was the half that actually lost us calls. A voice agent is judged as much by how fast it stops as by how fast it starts, and every buffer you add for smoothness is a buffer you have to be able to empty on command.

So now the first thing I instrument on any voice pipeline is the tail, and I make sure I can flush every buffer I add. The audio is already gone by the time you decide to stop it. I build like it is.