Originally published at ictinnovations.com
A telephony voice agent can produce perfect audio and still be unusable on a phone call. What decides it is timing: whether frames arrive evenly, whether the sentence has holes in it, and how fast the agent shuts up when you talk over it. None of that is visible from inside the agent, so we built an open source benchmark that measures it from the caller's end of the wire.
The tool is called tvbench. It's MIT licensed, it runs against any agent that speaks AudioSocket, and every number it reports comes from arrival timestamps rather than from anything the agent says about itself. This article is about why we ended up there, because the reasoning matters more than the tool.
The bug that started it
We had an Asterisk voice agent that worked on our desks and sounded broken on real calls. Callers reported hearing the tail end of sentences. Sometimes they heard nothing at all and then a fragment. Our logs were clean. Every turn showed the right text, the right byte count, the right duration. Nothing anywhere said a word had gone missing.
The fault turned out to be a single line of synthesis code that wrote a whole utterance to the socket in one call. That is a perfectly reasonable thing to write if you're used to files or HTTP. On a call it destroys the audio, because app_audiosocket hands every frame it receives straight to the channel and does not buffer on your behalf. Two and a half seconds of speech delivered in eight milliseconds means the jitter buffer at the far end keeps a handful of frames and drops the rest.
We wrote up that fix and several others in our engineering log for the voice agent. What stayed with us afterwards wasn't the bug. It was how long it took to find something that obvious, and the reason it took so long: we had no instrument pointed at the right place.
Byte counts are not evidence
Here's the uncomfortable thing about that failure. Every measurement we had said the turn was fine, and every one of them was correct. The agent really did produce 120 frames. It really did write all of them. The duration really was 2.4 seconds of audio. The problem was that those facts have nothing to do with what a caller hears.
A phone call is a real-time system with a fixed clock. Audio is worth something only if it arrives at the rate it will be played. Deliver it faster and the surplus is thrown away. Deliver it slower and the caller hears silence in the middle of a word. Both failures are invisible to anything that counts bytes, and both are trivially visible if you record when each frame showed up.
So we stopped instrumenting the agent and started instrumenting the socket.
Standing where Asterisk stands
tvbench takes Asterisk's place in the conversation. It opens the AudioSocket connection, paces caller audio at one 320-byte frame every 20 ms on a deadline it re-clamps after every frame, and timestamps every frame that comes back. It knows nothing about the agent behind the socket and needs to know nothing, which is why it works against agents written in any language.
The harness only speaks the protocol. Everything it reports is derived from when audio arrived.
That position is the whole design. An agent reporting its own latency is reporting when it decided to do something, not when the result reached anyone. The gap between those two things is where the interesting failures live.
Four numbers, and what each is for
Opening latency is how long the caller waited before hearing anything. Callers are unforgiving about this one and it is the easiest to measure honestly, because an agent's greeting runs without a transcript or a model round trip.
Pacing is two numbers that answer the same question from different sides. The worst burst is the most 20 ms frames that landed inside a single 20 ms window. The realtime ratio is audio delivered divided by wall clock. For both of them, one is the correct answer. Anything much above one means you're overrunning the far end's buffer.
Continuity is the longest hole inside the agent's own speech, measured between its first and last audible frame. This catches an agent that synthesises one sentence at a time and stalls between them.
Barge-in cut is how long the agent kept talking after the caller started. It's timed from the first frame of caller speech the harness puts on the wire to the last frame of agent audio that comes back, so it covers voice detection, the decision to stop, dropping whatever was queued, and anything already handed over that can't be recalled.
The failure that scores well
Put the correct agent and the bursting agent side by side and you get a result that should make anyone nervous about single-number benchmarks.
Identical audio, two delivery patterns. The buffer keeps a few frames from the burst and discards the rest.
The bursting agent scores a better continuity number than the correct one. Zero milliseconds of hole against 21. It has to, because frames that arrive at the same instant leave no gaps between them. If you ranked agents on continuity alone, the broken one would win.
That's not a flaw we papered over. It's the reason the pacing columns exist, and it's the argument against reducing a voice agent to one score. Timing failures pull the metrics in different directions and you need to read them together.
Testing the test
A benchmark that has only ever seen well-behaved agents proves nothing. You have no idea whether it would notice a fault, because it has never been shown one.
So tvbench ships a reference agent with switchable defects, and a validation script that runs the whole matrix. Each defect is designed to move exactly one metric. If a defect stops moving its metric, the build fails, because that means the tool has gone blind to something it's supposed to catch.
Five runs per row on Linux. Read the columns down: each defect moves its own metric and leaves the rest near the correct row.
Two rows are worth dwelling on. The agent that drops queued audio on interruption stops in 97 ms. The one that ignores the caller runs 4,974 ms. That's a factor of fifty on the single number a caller actually notices, and no amount of model quality compensates for it.
The other row worth dwelling on is the burst one, for the reason above. We published it because it undercuts a simple reading of our own tool, and a benchmark you cannot argue with is not measuring anything.
Two scenarios, and why only those two
We were tempted to build a scenario library. We ended up with two, because two is what we could measure honestly.
The greeting run answers the call and stays silent. That sounds like a weak test until you notice that an agent's opening line is the one part of the pipeline that runs without a transcript and without a model round trip. Nothing upstream can vary, so anything wrong in the numbers is a delivery problem and nothing else. It's the cleanest look at pacing available, and it happens to match what a real caller does at that moment, which is wait.
The barge-in run lets the agent get about a second into its turn, then talks over it. This is the scenario that separates demos from products. An agent that can't be interrupted is exhausting to talk to, and the delay is cumulative, because every stage adds to it and the caller experiences the total.
The clock covers four stages. A number measured inside the agent usually covers the first two.
We time the whole window on purpose. Voice detection has to decide the caller is speaking and not coughing. The turn has to be marked interrupted and every producer told to stop. Queued audio has to be dropped instead of played out. Whatever was already handed to the far end is simply gone, which means your queue depth sets a floor on how fast you can possibly stop. A figure covering only the first two stages will look excellent and mislead you, because stages three and four are where most of the delay lives.
Running it against agents that expect a real call
Most production agents won't accept an anonymous connection. Ours doesn't. It expects the call to have been registered first, which is sensible behaviour and awkward for a benchmark.
Rather than ask people to weaken their own security to run a test, the harness takes a registration URL and posts the call id to it before connecting. That keeps the allowlist working and keeps the measurement honest, because it exercises the same path a real call takes instead of a special test path built alongside it. If your agent has no allowlist, you leave the flag off.
Every run writes a JSON file with the raw arrival timestamps, not just the summary. We wanted the underlying data in the artifact so anyone who thinks a metric is computed wrongly can recompute it without rerunning anything.
What it will not tell you
The caller signal is band-limited noise, amplitude modulated at roughly syllable rate and generated from a fixed seed. Voice activity detectors accept it as speech, which is all a timing test needs.
It is not speech. tvbench cannot tell you anything about transcription accuracy and doesn't try, because a word error rate computed from synthetic noise would be a fabrication. Use a real speech corpus for that question and use this for the timing one.
Two more limits we state in the README rather than bury. Run it on Linux, because asyncio.sleep(0.02) takes about 31 ms on Windows and the harness can't pace accurately there. And loopback is not a network, so if you want to know how your agent behaves under jitter and loss, you have to add jitter and loss yourself.
Why we made it open source
We build open source VoIP software for service providers, and this is a measurement problem the whole field has rather than a problem we have. Every team wiring a model into a call has to solve frame pacing and barge-in, and most of them are doing it without an instrument, the same way we were.
There's a self-interested reason too. If our own agent's figures are wrong, the fastest way for anyone to prove it is to run the same tool and post a different result. The results/ directory takes pull requests for exactly that. Releases are archived on Zenodo with a DOI, so a number measured against a given version stays reproducible even if the repository moves.
Disagreement backed by a reproducible number is the point of the exercise.
Frequently asked questions
Does tvbench only work with Asterisk voice agents?
It works with anything that accepts an AudioSocket connection. The harness speaks that protocol and nothing else, so it doesn't know or care whether Asterisk, a custom gateway, or nothing at all sits behind the socket. The agent can be written in any language.
Why measure from outside the agent instead of adding timing logs inside it?
Internal timing tells you when the agent decided to send audio. It can't tell you when that audio arrived, whether a jitter buffer dropped it, or how much was already committed when the caller interrupted. Those are the parts that go wrong, and they're only visible from the far end.
What is a good realtime ratio?
One. A ratio of one means you delivered a second of audio in a second of wall clock, which is what a phone call needs. Well above one means you're bursting and the far end is discarding audio. Below one means you're delivering slower than real time and the caller hears gaps.
How long does a benchmark run take?
A single scenario run takes a few seconds. The default is five runs so you get a median and a range rather than an anecdote, and the full validation matrix across every reference defect finishes in about a minute.
Can I use these numbers to compare commercial voice agent platforms?
Only where the platform exposes an AudioSocket endpoint you can point the harness at. Where it does, the comparison is fair, because the harness treats every agent identically and measures the same thing from the same place. Publish the command line alongside any number you report.
Where does this fit with your other open source telephony work?
It came out of building our open source Asterisk AI voice agent and the AudioSocket library under it. The benchmark is the instrument we wish we'd had first, and it now runs in that project's own test cycle.
Top comments (0)