Two weeks before launch, every turn was green and the call still died

The dashboard was a wall of green. Word error rate under 5 percent. Intent classification at 94 percent on our eval set. Response appropriateness, graded by a rubric we trusted, sitting comfortably in the high 80s. By every number we tracked, the scheduling agent was ready to ship.

Then I sat in on the recordings.

A woman called to reschedule a dentist appointment. The agent transcribed her perfectly. It caught the intent (reschedule) on the first try. It offered times. Every single turn, if you froze it and graded it in isolation, was correct. On turn four it confirmed "Tuesday the 14th" when she had asked for the 14th but had earlier said she could not do Tuesdays. Small slip. The agent did not catch it. She did, sort of, and got confused, and re-explained, and the agent, now anchored on the 14th, kept steering back to it. Turn seven, she said "you know what, I'll just call the front desk." Click.

Every turn passed. The call failed. And nothing in my green dashboard knew it had happened.

The number that was lying to me

Here is what I had gotten wrong, and I think a lot of voice teams get it wrong the same way. I was measuring quality at the turn level and quietly assuming it would add up to quality at the call level. It does not. Turn-level metrics and outcome-level success are different quantities, and treating one as a proxy for the other is the bug.

The assumption hiding underneath a per-turn average is independence. When you report "94 percent turn accuracy," you are implicitly treating each turn as its own little exam. But a conversation is not a set of independent exams. It is a chain. The user's turn 5 depends on your turn 4. If turn 4 quietly plants a wrong assumption, turn 5 is now operating on bad state, and no amount of local correctness on turn 5 saves the call. Errors do not average. They compound.

Watch what that does to the math. Suppose, generously, that every turn is 95 percent correct and, even more generously, that the turns really were independent. The probability that a whole conversation of n turns is clean is 0.95^n, not 0.95.

def all_turns_correct(per_turn_accuracy: float, num_turns: int) -> float:
    """Probability every turn in a session is correct,
    under the (false but instructive) independence assumption."""
    return per_turn_accuracy ** num_turns

for n in (1, 5, 10, 20):
    p = all_turns_correct(0.95, n)
    print(f"{n:>2} turns: {p:.2%} of sessions fully clean")

Run it:

 1 turns: 95.00% of sessions fully clean
 5 turns: 77.38% of sessions fully clean
10 turns: 59.87% of sessions fully clean
20 turns: 35.85% of sessions fully clean

A 95-percent-per-turn agent has roughly a 60 percent chance of getting through a 10-turn call without a single slip. My real agent was worse than 95 on the turns that mattered, and calls routinely ran past 10 turns. The green dashboard and the dead call were both telling the truth. They were just measuring different things, and I had confused one for the other.

And the independence assumption makes that estimate optimistic, not pessimistic. Real errors are correlated in the worst direction. One wrong slot value does not just cost you that turn, it poisons the turns downstream that build on it. So 0.95^n is a ceiling on how well things go, not a floor.

The other half: patience is a budget

The compounding math explains why clean calls are rarer than turn accuracy suggests. It does not fully explain why calls fail, because most failed calls do not end in some dramatic model breakdown. They end the way the dentist call ended: the human runs out of patience and leaves.

A user does not have infinite turns in them. Every repeated question, every "sorry, I didn't catch that," every loop back to a thing they already said, spends down a budget. The task can be technically still-recoverable at turn 7 and still be over, because the person on the other end has decided you are not worth turn 8. Your agent never registered a failure. The transcript just stops.

This is why I stopped trusting any metric that could not see the whole call. The unit of success for a voice agent is not the turn. It is the session, judged against what the caller actually called to do.

Measure the thing the caller wanted

Task-oriented dialogue research has worked at this altitude for years, and it is worth borrowing the vocabulary. The MultiWOZ line of work evaluates dialogue systems against the user's goal, not the utterance: a task-success notion of whether the system actually provided the entity and information the user asked for, with the attributes they requested. Correctness is defined at the level of the goal. The dataset and its task-oriented evaluation are described in Budzianowski et al., "MultiWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling" (https://arxiv.org/abs/1810.00278).

You do not need their dataset. You need their altitude. For a production voice agent, define, per call, a binary (or small-ordinal) outcome that answers: did the caller accomplish what they called to do?

For our scheduler that meant: was an appointment actually booked, moved, or cancelled in the backing system, matching the constraints the caller stated, without a human agent picking up the pieces afterward? That is checkable. The booking system knows. The handoff log knows.

Then instrument it. The point is to log a session-level outcome alongside the turns, and to log where calls die, not just whether they die.

from dataclasses import dataclass, field
from enum import Enum

class Outcome(str, Enum):
    COMPLETED = "completed"        # caller's goal achieved in the system of record
    ABANDONED = "abandoned"        # caller hung up before resolution
    HANDOFF   = "handoff"          # escalated to a human
    FAILED    = "failed"           # ended without the goal met

@dataclass
class SessionTrace:
    session_id: str
    intent: str                    # what they called to do
    turns: list = field(default_factory=list)
    outcome: Outcome = Outcome.FAILED
    last_state: str = "greeting"   # dialogue state when the call ended

    def log_turn(self, state: str):
        self.turns.append(state)
        self.last_state = state

def conversation_success_rate(traces):
    done = sum(t.outcome == Outcome.COMPLETED for t in traces)
    return done / len(traces) if traces else 0.0

def abandonment_by_state(traces):
    """Where do dying calls die? Group abandons by last dialogue state."""
    counts = {}
    for t in traces:
        if t.outcome == Outcome.ABANDONED:
            counts[t.last_state] = counts.get(t.last_state, 0) + 1
    return dict(sorted(counts.items(), key=lambda kv: -kv[1]))

Two numbers fall out, and they are the two I actually steer by now. Conversation success rate is the headline: of everyone who called to do X, what fraction left having done X. Abandonment-by-state is the diagnostic: it points a finger at the exact dialogue state where people give up. When we ran it, the abandons piled up on one state, the confirmation step, which is exactly where the Tuesday slip lived. The turn metrics had been averaging that pain into invisibility.

None of this replaces turn-level metrics. Word error rate still matters. Intent accuracy still matters. They are how you debug why a session failed once you know it did. What they cannot do is tell you whether the call was a success, because you cannot read call success off a single turn. You can only read it off the whole call.

What shipped, and what I'd tell past me

We slipped the launch by a week. We wired the booking system's ground truth back into our eval as the session outcome, replayed a few hundred recorded calls against it, and watched conversation success rate come in well below what the turn dashboard had implied. That gap was the whole story. We fixed the confirmation state (make the agent re-check stated constraints before locking a slot, not after), and the abandonment cluster on that state shrank.

If I could hand one note back to the version of me staring at the wall of green, it would be this: a per-turn average is a measurement of your model's reflexes, not of your user's success. They are correlated, but the correlation gets weaker with every turn, because errors compound and patience runs out. Pick the outcome the caller actually wanted, make it checkable against a system of record, and measure at the level of the whole call. Log where calls die, not just that they scored well while dying.

The dentist call still bothers me. Every turn was correct and the woman still hung up and drove to a phone. The agent never knew it lost. Now it would.