Two weeks after we launched the support line, the dashboard was the color you want. Task-success rate: 91 percent. The agent booked the appointment, reset the password, quoted the balance. Green across the board. We had a wall of it.
The support queue told a different story. People were escalating to humans anyway, and when I pulled the recordings to find out why, almost none of them had failed. The agent got the job done in nearly every call I listened to. It just made the caller work for it.
That gap, between "the task completed" and "the call was good," is the thing I had measured wrong. And I had measured it wrong because I was grading a voice agent with a text agent's ruler.
What task-success hides
Task-success rate asks one question: did the agent reach the goal state. It is a transcript metric. You can compute it from the words alone. That is what makes it comforting, and it is why it misses most of what goes wrong on a voice call.
Here is a call that scores a perfect 1.0. The caller says their account number. The agent mishears one digit, reads it back, the caller says "no, seven, not eleven," the agent tries again, mishears the next field, the caller repeats the whole thing slower, and eventually they get there. Appointment booked. Task complete. From the transcript, a clean success.
From the caller's chair, that was ninety seconds of repeating themselves to a machine that would not listen. They will not call back. The transcript scored the destination. Nobody scored the road.
The metric that actually tracks "was this call good"
The thing I should have been counting has a name, and it is not mine. Conversation analysts have studied it since the 1970s. The canonical reference is Schegloff, Jefferson, and Sacks, "The preference for self-correction in the organization of repair in conversation" (1977). Repair is what people do when something in the talk goes wrong: they restate, they correct, they say "no, I meant," they slow down and try the same thing again.
Human conversations have repair too. The difference is rate and who initiates it. When a caller has to initiate repair over and over because the agent misheard, cut them off, or answered a question they did not ask, the call is bad no matter what the final state says.
So the metric I care about now is simple to define and annoyingly revealing:
repair_rate = (turns where the caller re-states, corrects, or says "no / I said")
/ (total caller turns)
You count it per call and you watch the distribution, not the average. A mean of "0.12 repairs per turn" sounds fine. The tail is where your angriest callers live: the six percent of calls where the caller had to repair four or five times before the agent caught up. Those are the ones churning, and task-success rate cannot see them because every one of them ends in success.
Two cheaper cousins are worth logging next to it. Turns-to-completion, because a booking that takes eleven turns is a worse booking than one that takes four. And interruption rate, how often the agent starts talking over the caller, which on our traffic correlated with repair more than any single ASR number did. All three are conversational, not transcript-level. All three need the audio and the timing, not just the words.
Wiring the metric in
Counting repair by hand during a post-mortem tells you what went wrong last week. To change what ships, the same count has to run against every candidate build, on calls that resemble your real traffic: the frustrated repeat-caller, the fast talker, the one with background noise. Practically that means generating those calls and scoring the audio and transcript against metrics you define, repair rate among them. Several tools now cover that ground for voice agents and are worth knowing before you build it yourself. Capabilities below are as of July 2026.
Coval builds simulation-first QA and borrows its framing from self-driving-car testing. Hamming calls itself a flight simulator for voice agents and pairs automated call generation with production monitoring. Future AGI's agent-simulate is open source (Apache-2.0): it drives a simulated caller through your agent in a LiveKit room and scores the result in its ai-evaluation library against built-in or custom metrics (github.com/future-agi). Cekura auto-generates test cases so your QA set is not just the ten calls you thought of. Maxim AI spans the wider loop, experimentation through production observability, for voice and multimodal agents.
Pick by your constraints, not by the feature grid, because none of these will tell you which metric matters for your callers. Run as many simulated calls as you like scored on task-success and you get back the same green wall I started with. Choosing the number is the part that stays yours.
What shipped, and what I would tell the version of me with the green dashboard
We kept task-success on the board, because it is a real floor and a regression in it is a real fire. We just stopped treating it as the headline. Repair rate in that tail, the calls where someone had to say it four or five times, is the number I look at first now. When a build lowers it, the calls sound better and the escalations drop, and those two things move together in a way task-success never did.
If I could go back to the engineer staring at 91 percent and feeling done, I would tell him one thing. The dashboard is green because you asked it the question a chatbot answers. Voice agents fail in the parts a transcript throws away: the timing, the talking-over, the third time the caller had to say their own name. Go count those. The color will change, and so will the thing your callers actually feel.
Top comments (0)