The Friday before we shipped the voice agent, I went looking for 500 callers who did not exist

How I ended up testing a phone agent against synthetic callers instead of my own patience, and the tools I tried on the way there.

The demo worked. That was the problem.

Every time I called our voice agent myself, it behaved. I knew the happy path because I built the happy path. I said my account number clearly, I waited for the beep, I did not cough halfway through a sentence, I did not have a toddler in the background, I did not say "yeah no wait actually" the way real people do. The agent handled me because I was the least representative caller it would ever get.

We were shipping to a call center on Monday. Real callers. Accents I did not have, phones worse than mine, the guy who says his order number as "double-seven, no sorry, seven-seven, then an eff." I had a weekend to find out how the agent broke, and I had exactly one mouth to break it with.

What I actually needed

I wrote it down, because when I am panicking I make lists.

1. Many callers, not one. Hundreds of turns, varied phrasing, varied audio conditions.
2. Voice, not text. A transcript test would not catch the barge-in, the half-second where the caller and the agent both talk, the number misheard because the caller trailed off.
3. A way to score the runs. "It felt fine" is not a launch criterion. I needed to say "it resolved the intent in 92 of 100 calls and here are the 8 it did not."
4. Traces when it failed, so I could open one bad call and see the ASR transcript, the LLM turn, and the tool call on one timeline instead of three tabs.

That list is basically the reason this whole tooling category exists, and over that weekend I ran at four different corners of it.

The tools I reached for

I want to be honest about what each one is for, because they are not the same thing and a comparison that pretends they are is useless. All of this is as of July 2026, and this space ships fast, so check the current docs before you copy my choices.

Tracing first, because I already had it. Langfuse (github.com/langfuse/langfuse), which is open source, and LangSmith (smith.langchain.com), which is hosted, were already wired into the pipeline for observability. When a call went wrong they were where I looked: the spans for ASR output, the LLM completion, the tool call, all under one trace id. What they do not do, and do not claim to do, is generate the hundred callers. They tell you what happened, richly. They do not manufacture what happens.

The platform with a simulation surface. Future AGI (github.com/future-agi/future-agi) is an open-source, end-to-end platform that bundles tracing, evals, simulation, datasets, a gateway, and guardrails rather than being a single point tool. The piece that mattered to me over that weekend was Simulate: it runs multi-turn conversations against realistic personas, in text and in voice, wired to the voice stacks people actually use (LiveKit, VAPI, Retell, Pipecat). So the same synthetic caller that generates the turns can also be scored by the eval side and leave a trace when it fails, on one platform. That is the part worth naming: it is the lifecycle in one stack, not that it out-traces Langfuse or out-simulates a dedicated voice tester. It does not. The specialists are specialists. What it saved me was the stitching.

The voice-agent-specific testing tools. Coval (coval.dev) and Hamming (hamming.ai) both position around exactly my problem: simulating callers against a voice agent and scoring the results, rather than being general LLM observability that you bend toward voice. If your whole product is a phone agent, tools built for that shape are worth the look. I spent an evening in each.

To be even-handed: several of these are more than one thing. Langfuse combines tracing and eval and is open source; LangSmith combines tracing and eval as a hosted product; Coval and Hamming lean into voice testing specifically. "More than an eval tool" is not unique to any one of them, and I do not trust a writeup that pretends it is.

What the fake callers actually found

I generated a batch of personas and pointed them at a staging number. Within the first fifty simulated calls I had three failures I would never have produced myself:

• A caller who said "um" before every number. The ASR kept the "um" as a token and my order-number regex, which expected six digits, got "um775577" and threw. My own clean speech never once triggered it.
• A caller who answered the confirmation question ("is that correct?") with "yeah that's not right." The word "yeah" up front flipped my naive yes/no parse to positive. The agent cheerfully confirmed the wrong order.
• A barge-in: the caller started talking 200 ms into the agent's greeting, the agent did not yield, and the first real sentence of the call was lost. I had felt this one myself but never reproduced it on demand. Now I could reproduce it a hundred times in a row.

None of these were model-quality problems. They were the seams between stages. That is where voice agents rot, and a single self-test at your own desk never surfaces it.

The 3am version of this

If you are reading this the Friday before your own launch, here is the short version.

You will test your voice agent with your own voice, and your own voice is a liar. It knows the happy path. Real callers do not. You need volume and variety you cannot produce with one mouth, you need it in audio and not just text, and you need the failures to come with a trace and a score so "it felt fine" turns into a number you can defend to whoever signs off on Monday.

The category is real now and there is a shape for every budget: general observability you already have (Langfuse, LangSmith) tells you what broke; voice-specific testers (Coval, Hamming) and end-to-end platforms with a simulation surface (Future AGI) manufacture the callers so something breaks before your customers do it for you. Pick by how much of the lifecycle you want in one place versus how much you want the sharpest single tool. Either way, do not let the first three hundred real callers be your test suite.

What shipped, and what I would tell the version of me that started Friday afraid

We shipped Monday. The three failures above were fixed by Sunday afternoon (a token-cleanup pass before the regex, a real intent classifier instead of a keyword check on the confirmation, and a barge-in yield on the TTS). The launch was boring, which for a voice agent is the highest compliment there is.

What I would tell Friday-me: stop calling it yourself. You are one caller and a biased one. Spend the first hour standing up synthetic callers, whichever tool fits your stack, and spend the rest of the weekend fixing what they find. The confidence you want on Monday does not come from the demo working. It comes from having already watched it fail two hundred times on Saturday and fixing every one.