I spent a Monday morning sorting a spreadsheet that could not be sorted.
Four hundred and eleven calls from the previous month had come back below our threshold. I wanted them grouped, because I had one sprint and I wanted to spend it on whatever was biting the most callers. So I opened the export and looked for the column that says what went wrong.
There is no such column. There is a score. Every one of those 411 calls carried a number under 0.7 and nothing else, and a number under 0.7 does not tell you whether the agent talked over the caller or invented a policy.
I tried the obvious substitutes before admitting that. Sorting by score just puts the worst calls on top, and the worst calls are a mix of everything. Sorting by duration finds the ones that dragged, which is one failure mode out of a dozen. Sorting by which intent the caller came in with tells you where the failures land, not what they are, and by Wednesday I had three tabs that each answered a question I had not asked.
What I wanted was a count per reason. Twelve rows, sorted descending, so I could point at the top one on Monday and be done arguing about it.
What I had already fixed, and what it did not fix
Two weeks earlier I wrote about replacing task-success rate with repair rate: counting how often the caller has to restate themselves because the agent misheard or barrelled ahead. That change was worth making. Repair rate moves when the call is bad in the way callers care about, and task-success does not.
It also did not help me that Monday. A better number is still a number. Repair rate told me which calls were bad and roughly how bad. It had nothing to say about which of them were bad for the same reason.
That is the gap I had been calling a metrics problem for about six months. It is a vocabulary problem. Until your failures have names, you cannot count them by name, and if you cannot count them by name you cannot pick the biggest one.
The question I ended up asking five tools
So I went and read the trees. One question, asked the same way of each:
When an eval marks a case bad, what comes back, and who wrote the list of things it is allowed to say?
That second half is the one that matters and the one nobody advertises. A vocabulary you write yourself starts empty and fits your product. A vocabulary the vendor ships saves you the blank page and constrains you to their idea of failure. Both are defensible. They are very different purchases.
Everything below is from the repositories as of 4 August 2026, ordered by GitHub stars purely because that is a neutral ordering and not a ranking of fitness. File paths are there so you can check me rather than believe me.
| Tool | What comes back on a failure | Who writes the label space | Nearest thing to a voice failure |
|---|---|---|---|
| Langfuse (32,498 stars, MIT core; ee/ is commercial) | A score attached to a trace or an observation, typed CATEGORICAL, NUMERIC, BOOLEAN or TEXT (packages/shared/prisma/schema.prisma:465) | You do. model ScoreConfig keeps your category names as a reusable, project-scoped object (schema.prisma:441) | Nothing prewritten. The categories column ships empty and you fill it |
| Promptfoo (23,920 stars, MIT) | The name of the assertion that failed, drawn from a 66-entry enum (src/types/index.ts:595) | Promptfoo writes the catalogue, you pick per test case | The closest of the five. latency, trace-span-duration and conversation-relevance are all in that same enum |
| DeepEval (17,398 stars, Apache-2.0) | A per-metric score plus the judge's reason string | DeepEval, as named metric modules you import | The richest multi-turn set: turn_relevancy, role_adherence, conversation_completeness, knowledge_retention under deepeval/metrics/. Conversational, not spoken |
| Arize Phoenix (10,896 stars, Elastic 2.0, so not an OSI licence) | A Score carrying a validated label; a label outside the declared set raises rather than passing through (packages/phoenix-evals/src/phoenix/evals/evaluators.py:766) | You declare the choices, Phoenix enforces them. Fourteen metrics ship under .../phoenix/evals/metrics/ | user_friction.py, which is the only name in any of the five that is about the caller's experience of the exchange |
| Future AGI (1,586 stars, Apache-2.0) | A classified error with a category path, evidence spans and a suggested fix (futureagi/tracer/models/trace_error_analysis.py:91) | Future AGI, and the list is not in the repo: category is a 200-character string, not an enum (same file, line 112) | Nothing voice-shaped in the one readable taxonomy (31 subcategories, futureagi/model_hub/utils/evals.py:3066) |
The thing none of them have a word for
Read down that last column. Five tools, and the two nearest hits are a latency assertion and a metric called user friction.
Neither of those is what I need. A voice agent fails by starting its sentence 300 milliseconds into the caller's. It fails by going quiet for two seconds while a tool call resolves, which on a phone line reads as a dropped call. It fails by reading a confirmation number at conversational speed to someone holding a pen. It fails by acknowledging with the same four words eleven times.
None of those are hallucinations. None are wrong tool arguments. They are the entire content of my last three post-mortems, and there is not a name for any of them in any vocabulary I read, including the two vendors that ship a prewritten failure list rather than an empty one.
I nearly wrote the wrong conclusion here. The label spaces were written for agents that type, which is what almost every agent still does. That is not the tools being bad at voice. Voice is the minority case, and the vocabularies reflect that honestly.
There are voice-native vendors in this space. Coval, Hamming and Cekura all sell testing for spoken agents, and any of them may already have solved this. All three are closed source, I could not open the tree, and I am not putting a capability claim in a table on the strength of a landing page. They are worth a demo. They are not worth a row I cannot check.
Two shapes of vocabulary, and what each one costs
The five split cleanly once you stop reading them as competitors and start reading them as two designs.
Langfuse and Phoenix hand you the primitive. Langfuse gives you a named categorical score config that lives at the project level, so agent_talked_over_caller becomes a real object other people on your team can attach to a turn. Phoenix goes one step further and refuses labels outside your declared set, which sounds pedantic until a judge invents a category at 2am and quietly splits your counts in half.
DeepEval and Future AGI hand you a filled list. DeepEval's is readable and importable, which is the version of this I would push people toward first: you can see exactly what role_adherence means before you depend on it.
Future AGI sits at the far end. Its cloud platform clusters production failures and returns a root cause with a suggested fix (futureagi.com/platform/evaluate/error-feeds). The open-source UI gates that behind a "Cloud feature" screen (frontend/src/components/oss-upgrade-gate/oss-upgrade-gate.jsx:17). As of August 2026 you cannot read the category list before you send traces.
If I were choosing today for the voice half specifically, I would take the primitive over the filled list, and Langfuse's score config is the cleanest primitive of the five. Not because it does more. Because the twelve names I actually need do not exist yet in anyone's list, so the thing I am buying is somewhere to put them.
Week two: where the twelve names attach
We wrote our own. Twelve categories, all voice, all lifted from post-mortems we had already written: talked-over-caller, dead-air-over-1.5s, confirmation-read-too-fast, acknowledgement-loop, and eight more that are specific enough to be embarrassing.
Declaring them is one call. The config is the vocabulary, and the twelve labels live inside it:
curl -X POST https://cloud.langfuse.com/api/public/score-configs \
-u "$LF_PUBLIC_KEY:$LF_SECRET_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "voice_failure_mode",
"dataType": "CATEGORICAL",
"description": "Turn-level voice failure taxonomy. One label per agent turn.",
"categories": [
{"label": "talked-over-caller", "value": 1},
{"label": "dead-air-over-1.5s", "value": 2},
{"label": "confirmation-read-too-fast", "value": 3},
{"label": "acknowledgement-loop", "value": 4}
]
}'
Two constraints worth knowing before you name anything: the config name is capped at 35 characters, and each category carries a numeric value alongside the label, which is what you end up grouping on.
The mechanical detail that took me two tries to get right is where the label attaches. It goes on the turn, not the call. The naming was the easy half.
A call scored talked-over-caller tells you the problem happened somewhere in four minutes of audio. A turn scored talked-over-caller tells you which turn, which means you can pull the 400 milliseconds around it and listen to it. We spent the first week attaching per call and produced a leaderboard nobody could act on.
One label per turn, not a set. We tried multi-label for three days and stopped, because a turn tagged both dead-air and acknowledgement-loop makes the counts ambiguous exactly when you are trying to rank them, and ranking them is the entire point. If a turn genuinely has two, we take the one the caller reacted to.
A rough judge assigns the label on every turn and I re-label the disagreements by hand on Friday mornings. It runs about forty minutes and it is the most useful forty minutes in my week, because the disagreements are where the vocabulary is still wrong.
What shipped, and what I would tell the version of me sorting that spreadsheet
The counts were not what I expected. Dead air came third. Acknowledgement-loop, the one I would have sworn was cosmetic, came first by a distance, and it traced back to a single retry path that had been in production since May. Nine lines. It had been sitting there the whole time I was tuning thresholds.
What I would tell the guy with the spreadsheet is narrower than "go build a taxonomy". It is that your sprint goes to whatever you can count, so what you can count is the thing to fix first. I had spent six months getting better at saying how bad a call was. The change that moved what we shipped was smaller than that: I stopped grading calls and started labelling turns.
The spreadsheet still has 411 rows. It sorts now, and the top row is a retry path from May.
Top comments (0)