Most production voice agents do not fail by falling over. They fail by being confidently wrong in a way that looks exactly like being right — and then reporting success.
That is a different problem from a crash, and it needs a different answer. A crash tells you about itself. A wrong disposition does not.
I write this up because it is the part of a build that almost never appears in a scope of work, and it is the part that decides whether the system can be trusted six weeks after launch.
The failure that makes verification necessary
Take an agent that qualifies outbound calls and pushes interested leads into a CRM.
An answering machine produces a transcript that structurally resembles a conversation. Greeting, pause, speech. A classifier reading that transcript has no strong signal separating a person said hello from a recording said hello — they look the same in the only data it has.
So a share of "qualified leads" are voicemail greetings. Every one of them burns a human's dial on somebody who never spoke to the agent. The system built to relieve a sales team starts manufacturing work for it.
The instinct is to tighten the classification prompt. That fights the symptom. You are asking a model to distinguish two inputs that genuinely resemble each other, and no amount of instruction creates a signal that is not in the transcript.
The better move is structural: stop trying to make the first pass smarter, and check its output afterwards. Take the transcript plus the disposition the first pass assigned, and have a second pass ask whether that disposition was correct. Post-hoc adjudication rather than better prediction.
The trap in that fix
I published that recommendation, and a reader called Alexander Rådahl Benz pushed back on exactly that section. His objection, and he is right about the risk:
"One thing I'd change in the verifier section: don't trust a second LLM just because it's separate. If both see the same transcript and rubric, they can share the same blind spot."
This is the part worth slowing down on, because "add a verification step" sounds like diligence and can easily be theatre.
A second model instance is not a second opinion. If the second pass reads the same transcript, against the same rubric, asking the same question, it is not independent in any sense that matters — it is the same judgement executed twice. When it agrees, you have learned nothing. You have bought latency and a feeling of rigour.
Independence is a design choice, not a property of the model
Here is where I would put the fix somewhere different from don't trust it.
Nobody said the second pass has to see the same evidence and ask the same question. The correlation between two passes is something you engineer. There are three levers:
Change the question. The first pass asks is this qualified? Have the second ask what would disqualify this? Same model, opposite burden of proof. Make it argue the disposition is wrong, and only let the record through if that argument fails.
Change the evidence. The first pass reads the transcript. Give the second the audio, or the call metadata — duration, answer latency, DTMF events, the shape of the turn-taking. A voicemail and a live answer look alike in a transcript and quite different in call metadata. Two passes over the same evidence share its blind spots by construction; two passes over different evidence do not.
Change the threshold. An adjudicator that has to be convinced is not the same instrument as one that has to be persuaded otherwise.
Same model. Different burden of proof. That is what independence is made of.
Then make the independence measurable
Alexander Rådahl Benz's follow-up is the part I would keep above everything else here, because it converts a design claim into something you can check:
"I'd make the independence measurable, though: track how often the verifier flips the first disposition, broken down by failure class and evidence source. If it never disagrees, the burden of proof is different only on paper; if it disagrees randomly, the second path is noise. The design earns its keep when it catches a repeatable class the first pass misses."
That is the test. A verifier that never overturns anything is decorative. A verifier that overturns things at random is noise wearing a lab coat. His last sentence there is the whole test: the design earns its keep when it catches a repeatable class the first pass misses — voicemails, say, or a specific kind of ambiguous yes.
So the number to instrument is not accuracy. It is the flip rate, split by failure class and by which evidence source produced the flip. That tells you whether the second path is doing independent work or rubber-stamping.
Add a small human-labelled set, re-scored after every prompt or model change, and you have a verifier you can actually reason about.
The same failure, well outside voice
I hit this exact shape in my own tooling, and it is worth including because it generalises past AI entirely.
I had written a checker whose job was to confirm that a set of published articles carried a real link back to my site. It reported everything healthy. The articles carried the site as plain text — not a link at all, no signal passed. The checker counted a bare-text mention as a link, which is to say it accepted a weaker form of the thing it was built to verify, and so it certified the precise failure it existed to catch.
That is the same defect as the correlated verifier. Not "the check was wrong" — the check was structurally incapable of failing on that input.
The rule I took from it: if a check has never once failed, suspect that it cannot. Feed it a known-bad case deliberately and confirm it screams. A check you have never seen fail is a check you have never tested.
Two more disciplines that belong in the same conversation
Fail toward doubt. When a check cannot read the thing it is checking, it must return unknown, never fine. A health indicator that defaults to reassurance lets a broken system run for weeks while everyone reads the green light. Absence of a confirmed failure is not the same as a confirmed pass, and only one of those is safe to build on.
Sample what the system discards. The errors are asymmetric. False positives are loud — a rep dials a dead number and complains that day. False negatives are silent: a verifier quietly binning good leads looks exactly like a slightly quieter week. Nothing surfaces it, ever, unless somebody deliberately reads the discarded pile.
That failure is easy to produce. Ask a model is this a qualified lead? and it will tend to demand an unambiguous yes; anything with a complicating clause reads as not-a-clear-yes. A lead who answers yes to every question and mentions in passing that they are away until next week is a perfectly workable lead — and to a literal-minded adjudicator, a no.
What to ask before you hire someone to build this
Five questions. The answers cost nothing at design time and a great deal afterwards.
- How does the system verify its own dispositions? If the answer is the prompt is good, there is no verification layer.
- What does the second pass see that the first one didn't? If the answer is the same transcript, the two passes will share their blind spots.
- How will we measure that the verifier is independent? Flip rate, broken down by failure class and evidence source — not overall accuracy.
- How do we sample the records the system throws away? No answer here means the expensive failure is structurally invisible.
- What does the system report when a check cannot run? If the answer is anything other than unknown, the monitoring will eventually lie to you.
None of these are exotic. They are the difference between a voice agent that works in a demo and one that is still trustworthy when nobody is watching it closely.
If you are evaluating a builder and they have thought about the discarded pile before you asked, that tells you most of what you need to know.
Questions I get asked about this
Does a verification pass mean the agent needs a second model licence?
No. Independence comes from the question, the evidence and the burden of proof — not from a different vendor. Ussama Assad builds these with the same model on both passes and varies what each one sees.
Is this worth it on a small deployment?
The flip-rate instrumentation scales down fine; the thing that does not scale down is skipping it. A small deployment producing silent false negatives is just a small deployment nobody can audit.
Credit where it is owed: the independence-is-measurable framing above came out of a public exchange with Alexander Rådahl Benz in the comments of an earlier piece of mine — it sharpened the argument, and it belongs to him.
I'm Ussama Assad — I build and debug production outreach AI: voice agents, cold-email systems, lead-gen pipelines. Everything I write here is a real failure I traced to its cause and fixed. https://ussama.dev
Top comments (0)