Every voice agent I have shipped demoed perfectly. Then a real caller talked over it, changed their mind twice, asked something off-script, and the agent confidently booked the wrong appointment. None of that showed up in testing, because testing was a handful of clean, single-turn prompts. The conversation is where voice agents break, and it is the exact thing most test setups never touch.
Whatever you use to test a voice agent, whether you buy it or build it, its job is to drive the agent through realistic, messy, multi-turn conversations before a customer does. Anything less is shipping on vibes. This is based on a longer piece on the Future AGI blog; here is the checklist I actually run, and why each item earns its place.
Why picking a voice test setup is harder than it looks
A voice agent holds state across turns, handles interruptions, recovers from its own mistakes, and is supposed to stay on persona for the whole call. A test that sends one prompt and checks one response measures none of that. So the thing I want is not a recorder that replays a golden script. It is a simulator that generates realistic users, pushes the agent the way real callers do, and scores the whole trajectory. The checklist below is really one idea broken into pieces: test the conversation, not the script.
The checklist I run before I trust one
Does it simulate real users, or just replay a script? Real callers are not deterministic. A script runner only proves the agent handles the script. I want synthetic personas with traits that drive behaviour across the whole call, not a fixed list of recorded prompts dressed up as testing.
Does it test multi-turn conversations, not single turns? Voice agents fail on turn four, not turn one. State, memory, and recovery only show up across turns, so the unit of testing has to be the whole conversation with goals and expectations per turn.
Does it cover adversarial and edge-case users? The happy path always passes. Production breaks on the caller who interrupts, pushes back, and drops off-script. If the test users are all cooperative, the setup is lying to me.
Can it auto-generate diverse scenarios at scale? Hand-writing two hundred scenarios never actually happens, so coverage stays thin and the same three cases get tested forever. I want to generate diverse, realistic scenarios from a seed description instead of writing each one by hand.
Does it work with the framework I built the agent on? A tester that supports one framework forces a rewrite or gets abandoned. I look for adapters across the major agent frameworks so I can point the simulator at the agent I already have.
Does it give a pass/fail verdict and a transcript per conversation? "The agent seems fine" is not a test result. I need a verdict I can gate a release on, and the full transcript to debug a failure, not an aggregate feeling with no per-conversation detail.
Does it assert expected behaviour turn by turn? A conversation can reach the goal while doing something wrong on the way, like leaking data or breaking a policy. I want per-turn assertions that flag the exact turn it went wrong, not just a final outcome check.
Does it score conversation quality, not just task completion? An agent can finish the task rudely, off-brand, or with a hallucinated detail. Completion is not quality, so I want coherence, persona consistency, and groundedness scored across the call on top of task success.
Does it score tone directly, and have a path to audio? A voice agent can hit the goal and still sound clipped, robotic, or tonally wrong for a caller who is already angry. I want tone scored as its own signal I can assert on per turn, not averaged into one blended quality number, plus a way to score audio quality like naturalness, pacing, and latency once real speech is in play.
Does it scale to many persona and scenario combinations? Real coverage is personas times scenarios. A setup that runs them one at a time is not a release gate. I want a runner that executes every combination and aggregates the results into one number.
Does it integrate with evaluation and tracing? A failed conversation is a starting point, not an answer. I need to jump from the verdict to the scores and the trace to find the cause, which does not happen if the tool's results live nowhere near my observability.
Does it turn failed conversations into grouped, root-caused issues? Eighty failed transcripts is noise. What I actually need is the three underlying causes behind them, ranked, so I fix the bug and not the symptom. A tool that stops at "here are the failures" leaves the hardest part on my desk.
Does it have a path to real voice testing when I need it? Most failures are conversational logic I can catch in simulation, but real speech adds latency and audio-quality failure modes that text cannot surface. I want an honest account of how mature that speech path is, not a text-only tool sold as voice testing.
Does it run before production, in CI, not just after? A tool I run manually after an incident is a post-mortem, not a test. The point is to block the bad release, so I want a programmatic path that wires into CI and lets the pass rate gate the deploy the same way unit tests do.
How I weight these depending on what I am shipping
Not every item matters equally on every project. If the agent handles open-ended conversations with real users, items one, two, and three decide it. If I need real coverage rather than three hand-written cases, four and ten matter most. If "it finished the task" is not a high enough bar for the brand or a compliance team, eight, nine, and eleven carry the weight. If I need to close the loop on failures fast rather than just find them, twelve is the one. And if latency and audio quality are actual launch criteria, thirteen moves to the top, and I validate that speech path early because it tends to be the newest and least proven part of any setup.
The teams shipping reliable voice agents stopped testing clean single-turn prompts and started simulating messy, multi-turn callers. The whole checklist collapses back to that. Test the conversation, not the script, and gate the release on the number that comes out.
Top comments (0)