A perfect transcript can still be a wrong call: evaluating voice agents

I built a small tool called voiceeval because I kept hitting the same blind spot: a voice agent can produce a transcript that reads as flawless and still have failed the call.

The problem

Here is the call that started it. The caller asks for a refund. The agent refunds the amount and says so. Read the transcript and every line agrees with every other line. The caller asked for fifty dollars, the agent refunded fifty dollars, done.

The caller said fifteen.

"Fifteen" and "fifty" are one unstressed syllable apart. Speech-to-text picks one, and the agent acts on whichever it got with exactly the same confidence. By the time it reaches the transcript, the mistake is already baked in and internally consistent. Nothing downstream can see it, because the transcript has no memory of what was actually spoken and no clock on how long the caller waited.

That is the real issue. If you evaluate a voice agent by reading its transcript, you are evaluating a text agent that happens to have been spoken out loud. You will score a call as perfect when the caller hung up during a four-second silence, or when the agent confidently refunded the wrong amount and nobody ever found out.

Everyone can demo a voice agent. I wanted something that tells me whether mine is getting worse.

The core idea

voiceeval does not run calls. It judges them. You give it a call as timed turns, each turn carrying who spoke, what the STT heard, when they started and stopped, and for the caller's turns an optional truth field: what the caller actually said. It runs a set of checks that are each invisible to a text eval, and returns findings with a severity per case.

The input is deliberately dumb JSON so that whatever produced your call (LiveKit, Vapi, Twilio, a test script) can emit it with a few lines of glue:

{
  "id": "refund-happy-path",
  "policy": {"max_refund": 50},
  "turns": [
    {"speaker": "user", "text": "refund fifty dollars",
     "truth": "refund fifteen dollars",
     "start_s": 2.0, "end_s": 5.0},
    {"speaker": "agent", "text": "Refunding fifty now.", "start_s": 5.4, "end_s": 7.2,
     "actions": [{"name": "refund", "args": {"amount": 50}, "consequential": true}]}
  ]
}

How it works

Run the checker over a call and you get findings, ordered by severity:

$ voiceeval check fixtures/misheard_call.json

FAIL refund-misheard-fifty (7s call)
  high   misheard_number  turn 1
         STT heard ['fifty'] but caller said ['fifteen'].
  high   no_confirmation  turn 2
         Took consequential action (refund) without ever confirming.

Two separate failures, both invisible in text.

The first, misheard_number, only fires because the turn carried a truth field. The check normalises both strings, and if they differ it extracts the numbers from each side and compares the sets. When the heard numbers do not match the spoken numbers, that is a high-severity misheard number, because a wrong number the agent acts on is the most expensive failure in voice. If the words differ but the numbers match, it downgrades to a plain medium misheard.

The second, no_confirmation, is about the shape of the interaction rather than the words. The check walks the turns, finds any action marked consequential, and looks backward for an agent turn that actually confirmed, using phrases like "just to confirm", "did you say", or "shall I go ahead". A refund with no confirmation before it is flagged high. A read-only lookup is not, because demanding confirmation for every read would make the agent unusable. A lookup is not a refund.

Other checks cover the failures that live in the clock. Latency measures the gap between the caller finishing and the agent starting against a budget, defaulting to 1.5 seconds, and escalates to high past double that. Talking over the user looks for the agent's speech still running when the caller starts, but ignores overlaps under 300ms because humans interrupt each other constantly and flagging normal turn-taking is just noise. Dead air catches long silences where nobody is speaking, which is where callers hang up. Policy violation reads its limit from the interaction, not from the library, so a refund above max_refund is caught while what counts as allowed stays a business decision.

For regression work you run a whole suite, label it, then diff two runs:

voiceeval run calls/*.json --label v1 -o v1.json
# ... change the prompt ...
voiceeval run calls/*.json --label v2 -o v2.json
voiceeval diff v1.json v2.json

With --strict the diff exits non-zero on a regression, so a prompt change that quietly drops the pass rate fails CI instead of shipping.

One honest limitation

The misheard-number check is only as good as the truth field. Without ground truth for what the caller actually said, mis-hearing is undetectable by construction, and there is a test in the suite that documents exactly this. In production that failure is silent, and no tool can fix that for you. This is the argument for scripted test calls: you supply the truth once, in the fixture, and then the check can hold the STT accountable to it. If you only have raw production transcripts, this particular check has nothing to compare against.

I am also honest that the eval logic is the project and it is fully tested with no keys and no network, while the STT adapter that turns audio into timed turns is not exercised by those tests. If your platform already hands you a timed transcript, you never touch it.

Closing

voiceeval is small on purpose. It does one thing: it looks at a voice call the way the caller experienced it, with a clock and a record of what was really said, and it tells you which calls failed even when the transcript swears they passed. If you are shipping a voice agent and only reading transcripts, the fifteen-versus-fifty call is already somewhere in your logs, scored green.

Code and fixtures: https://github.com/royalpinto007/voiceeval