Your Voice Agent Needs a Flight Simulator Before It Gets a Phone Number

Pilots spend hours in a simulator before they ever carry a passenger. Most voice agents go live after the founder makes five test calls from a quiet room.

That gap is turning into a product category. This week The Times of India profiled Cekura, a Y Combinator-backed startup that tests and monitors voice agents for banks and lenders, catching agents that mishear callers or make promises they are not allowed to make (Times of India). A week earlier, monday.com launched Harmony, a voice platform for sales calls that ships with tools to simulate changes and test alternatives before rollout (Unite.AI).

You can buy this, and at scale you probably should. But the core idea fits in a weekend, and building it once teaches you what to look for. Here is the simulator I would build.

Why voice is harder to test than chat

• Errors compound. Speech to text hears "fifty" as "fifteen", the model reasons perfectly on the wrong number, and the caller gets a confident wrong answer.
• Timing is part of correctness. Talking over the caller, or leaving four seconds of silence, fails the call even if every word was right.
• Calls are long and branchy. A bug on turn nine only shows up if the caller says the right thing on turn three.
• Nothing is deterministic. The same prompt can pass ten times and fail on the eleventh.

So a handful of manual calls tells you almost nothing. You need volume, variety and a score.

The simulator in four parts

1. A scenario matrix: who is calling, what they want, and what makes it hard.
2. A simulated caller: a model that plays the customer, speaking through real audio.
3. Scoring: hard checks first, then a judge model with a rubric.
4. A regression gate in CI, fed by real production calls.

1. The scenario matrix

Write scenarios as data, then multiply them. Ten intents, six personas and four conditions give you 240 distinct calls from a few dozen lines. Even the short lists below already produce 64.

from dataclasses import dataclass
from itertools import product

@dataclass
class Scenario:
    intent: str        # what the caller wants
    persona: str       # how they behave
    condition: str     # what makes it hard
    must: list[str]    # outcomes the agent must reach
    must_not: list[str]  # things the agent must never do

INTENTS = ["reschedule payment", "ask about late fee", "dispute a charge", "ask for a human"]
PERSONAS = ["calm and brief", "angry and interrupts", "elderly and slow", "non-native speaker"]
CONDITIONS = ["quiet line", "street noise", "says numbers fast", "changes mind halfway"]

def build_matrix():
    for intent, persona, condition in product(INTENTS, PERSONAS, CONDITIONS):
        yield Scenario(
            intent=intent, persona=persona, condition=condition,
            must=["confirm identity", "summarize next step"],
            must_not=["promise a fee waiver", "quote a rate not in policy"],
        )

The must_not list is where most real damage hides. Write it with whoever owns the business rules, not just the engineers.

2. The simulated caller

The caller is another model with a hidden goal and a personality. Its text goes through text to speech, gets noise mixed in for the condition, and reaches your agent over the same transport your real calls use: your WebSocket or SIP endpoint, not a shortcut into the prompt. If you skip the audio path, you skip half the bugs.

CALLER_PROMPT = """You are a customer phoning a lender.
Goal: {intent}. Personality: {persona}.
Stay in character. Reveal details only when asked.
If the agent asks something unclear, say so like a real person would.
End the call when your goal is met or after 12 turns."""

async def run_call(scenario, agent_endpoint):
    caller = LLM(system=CALLER_PROMPT.format(**vars(scenario)))
    call = await connect(agent_endpoint)            # same path as production
    transcript, timings = [], []
    agent_text = await call.listen()                # agent greets first
    for turn in range(12):
        transcript.append(("agent", agent_text))
        reply = await caller.respond(agent_text)
        audio = add_noise(tts(reply), scenario.condition)
        sent_at = now()
        agent_text = await call.say_and_listen(audio)
        timings.append(now() - sent_at)             # caller stops, agent starts
        transcript.append(("caller", reply))
        if caller.done:
            break
    return transcript, timings

3. Scoring: hard checks before the judge

Run cheap, deterministic checks first. They are fast, exact and impossible to argue with.

FORBIDDEN = ["waive", "guarantee", "definitely approved", "no fee"]

def hard_checks(scenario, transcript, timings):
    agent_lines = " ".join(t for who, t in transcript if who == "agent").lower()
    return {
        "no_forbidden_promises": not any(p in agent_lines for p in FORBIDDEN),
        "human_when_asked": scenario.intent != "ask for a human" or "transfer" in agent_lines,
        "p95_response_ms_ok": p95(timings) < 1200,
        "ended_cleanly": len(transcript) < 24,
    }

Then a judge model scores what string matching cannot: did the agent actually solve the caller's problem, did it confirm the right numbers, was it polite under pressure. Ask for a score and a quote from the transcript as evidence, so you can audit the judge.

JUDGE_RUBRIC = """Score the agent from 1 to 5 on each item and quote the line that justifies it:
- goal_met: did the caller get what they called for?
- facts_confirmed: were names, dates and amounts read back correctly?
- recovery: when misunderstood, did the agent recover gracefully?
- tone: calm, clear and respectful throughout?
Return JSON."""

A call passes only if every hard check passes and every judge score is 4 or above.

4. The regression gate

Every prompt edit, model swap or voice change is a deploy. Run the matrix on each pull request and block the merge if the pass rate drops.

# .github/workflows/voice-sim.yml
on: pull_request
jobs:
  simulate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python sim/run.py --scenarios 240 --concurrency 20 --out results.json
      - run: python sim/gate.py results.json --min-pass-rate 0.92 --max-drop 0.02 --critical no_forbidden_promises

Treat no_forbidden_promises as critical: a single failure fails the build, regardless of the overall pass rate.

Close the loop with real calls

Simulated callers are only as creative as you made them. Real callers are more creative. Sample a few percent of production calls, run the same hard checks and judge on their transcripts, and turn every failure into a new scenario. After a month, your matrix reflects what customers actually say, not what you guessed they would say.

What to watch

Metric	Why it matters	A sane starting target
Pass rate on the matrix	Overall health per release	92% or higher
Critical rule failures	Promises, wrong amounts	Zero
p95 time to first response	Feels human or robotic	Under 1.2 s
Talk-over events per call	Interrupting the caller	Under 1
Escaped failures	Bugs found in production first	Falling every week

The agents that survive real customers are not the ones with the cleverest prompt. They are the ones that have already taken ten thousand bad calls in a simulator before taking their first real one.

We build and test voice agents at Axionry, an AI product development studio. See how we approach voice AI development, or estimate what your agent will cost per minute with the voice AI cost calculator.