DEV Community: Marcus Chen

Our voice agent passed every test and still woke me up at 3am

Marcus Chen — Thu, 11 Jun 2026 10:35:24 +0000

Replaying real call transcripts as your test set is a trap. The failures come from the inputs a user produces exactly once.

TL;DR: Our voice-agent regression suite was 312 recorded production calls, all passing. The page at 3am came from a caller who switched between English and Hindi mid-sentence, a pattern that appeared zero times in those 312 calls. Replaying real transcripts tests the confidence you already have. It does not test the inputs that actually break you. We moved to simulating adversarial callers, and below is what I learned trying five tools to generate and grade those simulated conversations (as of June 2026).

The test set was real, and that was the problem
For about four months our regression set was 312 recorded production calls. It felt rigorous. Real audio, real ASR output, real user intents, replayed on every deploy. Green for weeks.

Then the 3am page. A caller switched between English and Hindi inside single sentences. Our ASR mis-segmented the mixed-language audio, the intent classifier saw garbage, and the agent fell into a clarification loop it could not exit. The caller hung up. The dashboards were fine the whole time.

I went looking for that pattern in the 312 calls. It was not there. Not once. The people who code-switch like that had mostly churned months earlier, so the behavior was absent from the recordings exactly because it was a problem we never handled. A test set built from past traffic contains what already happened, weighted toward the common case. The failures that page you are rare by definition, and rare things are missing from a sample of the past.

Why replay gives false confidence
Replaying recorded calls is a regression test for behavior you have already seen. That is useful. It catches the case where a deploy breaks something that used to work. What it cannot do is produce an input you have never received. For that you have to manufacture the input on purpose: the fast talker who never pauses, the caller who interrupts the agent two words in, the code-switcher, the person who changes their mind halfway through a sentence, the line with a TV on in the background. That is simulation, and it is a different activity from replay.

What I tried to generate and grade simulated calls
Five tools, roughly a week each, same eight adversarial caller profiles. None of these is voice-specific; I drove them off transcripts plus a separate ASR/TTS layer. Honest notes, your mileage will differ:

Promptfoo: fast to wire into CI and good for red-teaming a prompt with generated variants. The fiddly part was that conversation state across turns was a manual build.
LangSmith: dataset versioning and the trace view were the best of the set. The simulation half I had to assemble myself.
Future AGI Simulate: persona-based, you define caller personas and it runs them through the agent, which matched how I already thought about adversarial callers (as of June 2026). Voice was not first-class, so I ran it on transcripts with ASR and TTS bolted on.
Braintrust: the nicest UI for eyeballing where a run diverged. Persona definitions lived outside it, in my code.
DeepEval: the most knobs for synthetic-conversation generation. Tuning the synthesizer to stop producing unrealistic turns took a while.
Confident AI: a reasonable hosted layer on top of DeepEval, though it is another account and key to manage. I am deliberately not crowning one. Braintrust had the UI I liked, DeepEval had the most generation control, and the persona abstraction in Future AGI's Simulate (part of their open work at github.com/future-agi) lined up with how I list out adversarial callers. Any of them can run a persona once you have written the persona.

The thing that actually moved the needle was not the tool
It was the persona list. Once we had written eight adversarial callers (the angry caller, the two-words-then-silence caller, the code-switcher, the background-noise line, and so on), every tool above could run them and grade the results. The leverage was in naming the failure modes, not in the framework that executed them. We spent two days arguing about the personas and twenty minutes wiring the runner.

The open question I still have
The space of adversarial callers is infinite, and we maintain eight. We chose those eight from incident postmortems, which means we are still only simulating failures we have already been burned by at least once. The genuinely novel failure, the next 3am page, is still unguarded. I do not have a principled way to pick simulation personas before the incident teaches me the persona. If you have one, that is the comment I want to read.

FAQ
Why not just add each failed call to the regression set after the incident?
We do. It is still reactive. The replay suite trails production by one outage, permanently.

Doesn't simulated traffic drift away from what real users do?
Yes, and that is a real cost. We re-sample the real call distribution monthly and adjust how often each persona fires. Simulation supplements replay; it does not replace it.

Is any of this voice-specific?
Most of it applies to text agents too. Voice just adds two more failure surfaces: ASR segmentation and barge-in timing. The code-switching incident was really an ASR segmentation failure that a text agent would never have hit.

The 4-layer voice-agent latency stack, traced with OTel spans

Marcus Chen — Tue, 09 Jun 2026 09:28:48 +0000

How I instrument ASR, LLM, TTS, and the client with OpenTelemetry, and which number in each layer I actually look at

TL;DR. A voice agent is four moving parts stuck together: speech to text, the model that writes the reply, text to speech, and the client that plays the audio back. End to end latency hides which of those four is slow on any given turn, so I stopped tracking it as one number and started tracing each stage as its own OTel span with a shared session id. The number I watch hardest is barge-in: when the user starts talking over the agent, how many milliseconds until the agent actually stops sending audio. In our setup we want that under 200ms, and when p95 barge-in creeps past that, the agent feels like it is talking at you instead of with you. Everything below is how I wire the spans, what attributes go on each one, and the p95 I page on per layer.

The thing I keep saying, and the thing that keeps being true: voice agents fail in production not because of raw latency but because nobody simulated the audio and LLM pipeline together. You can have a fast ASR, a fast model, a fast TTS, and a voice agent that still feels broken, because the failure lives in the seams between them and in the parts (barge-in, jitter) that no single-stage benchmark touches. Tracing is how I get the seams to show up.

A note before the layers. This is just the setup we run, the spans we emit, and the mistakes that made us add each attribute. Some of it is probably specific to our stack and will not transfer. I will flag that where I can.

The shape of a turn, and why one span is not enough

One turn is: user says a thing, agent says a thing back. Underneath that is roughly: audio frames come in, ASR turns them into text (streaming partials as it goes); the text plus history goes to the LLM, which streams tokens back; as text comes out, TTS turns it into audio, also streaming; the client receives audio frames and plays them, with some buffering to smooth out jitter.

If you wrap the whole turn in a single span and call it voice.turn, you get a duration and almost no ability to act on it. A 1,400ms turn could be a slow first token, or TTS waiting on the full sentence before it starts, or the client buffering too aggressively. Same total, three different fixes.

So the parent span is voice.turn, and each stage is a child span. Every span carries the same audio.session_id and an audio.turn_id, so I can pull one turn out of Tempo and see all four stages laid out in time. The attribute I care about most on the streaming stages is not total duration. It is first byte: how long until the stage produced its first useful output. First byte is what the user feels, because all three stages are streaming and the user starts perceiving progress at the first byte, not the last.

import time
from contextlib import contextmanager
from opentelemetry import trace

tracer = trace.get_tracer("voice.pipeline")

@contextmanager
def stage_span(stage, session_id, turn_id):
    span = tracer.start_span(f"audio.{stage}")
    span.set_attribute("audio.stage", stage)
    span.set_attribute("audio.session_id", session_id)
    span.set_attribute("audio.turn_id", turn_id)
    started = time.monotonic()
    state = {"fb": False}
    def mark_first_byte():
        if state["fb"]:
            return
        span.set_attribute("audio.first_byte_ms", round((time.monotonic() - started) * 1000.0, 1))
        state["fb"] = True
    try:
        with trace.use_span(span, end_on_exit=False):
            yield mark_first_byte
    except Exception as exc:
        span.record_exception(exc); span.set_attribute("audio.error", True); raise
    finally:
        span.set_attribute("audio.total_ms", round((time.monotonic() - started) * 1000.0, 1))
        if not state["fb"]:
            span.set_attribute("audio.first_byte_ms", -1)  # produced nothing
        span.end()

Calling it around the LLM stage: you call first_byte() inside the streaming loop the first time a token shows up, and the wrapper does the timing math.

async def run_llm_stage(session_id, turn_id, messages, llm_client):
    chunks = []
    with stage_span("llm", session_id, turn_id) as first_byte:
        async for token in llm_client.stream(messages):
            first_byte()           # no-op after the first call
            chunks.append(token)
    return "".join(chunks)

I use time.monotonic() and not time.time() on purpose. Wall clock can jump (NTP corrections), and on a sub-second budget a backwards clock gives you negative latencies that poison the percentiles. One more thing I learned the annoying way: audio.session_id is high cardinality, so I keep it as a span attribute for trace lookup, but I do not turn it into a metric label. Stage goes on the metric label. Session id stays on the trace.

ASR: measure first partial, not final transcript

The mistake I made first was timing ASR as audio-in to final-transcript-out. That number is real but it is not the one that matches what the user feels, because a streaming ASR gives you a partial transcript fast and then refines it. So the span gets two numbers: audio.first_byte_ms is time to first partial, and I stash time to final separately.

The other ASR attribute that earned its place is whether the final transcript disagreed badly with the last partial. We had an incident where ASR turned a customer saying they wanted to confirm an order into the word cancel, and the agent acted on it. After that I started recording a rough measure of how much the final revised the partial, so big late revisions show up in traces instead of only in an angry support ticket. What I look at for ASR: p95 of time to first partial. In our setup that sits under 150ms most days, and when it drifts up it is almost always the audio frames not arriving on time from the client, not the ASR model. A nice example of why you trace the whole thing.

The LLM: first token is the whole ballgame, and barge-in lives here too

For the model stage, total generation time barely matters for the felt experience, because TTS consumes tokens as they arrive. What matters is time to first token. If the model takes 600ms before the first token, the user hears 600ms of silence after they stopped talking, and that feels like the agent froze. So the LLM span's headline attribute is time to first token.

Barge-in is the part people forget to instrument, and the part I would instrument first if I were starting over. It is what happens when the user starts talking while the agent is still speaking. The metric: from the moment voice-activity detection fires, to the moment the agent's outbound audio actually goes quiet. The first time we measured it, it was around 500ms and felt terrible, and the breakdown showed most of the time was not detection. It was buffered TTS audio we had already shipped toward the client and could not un-send. We had buffered aggressively to fight jitter, and that same buffer made barge-in slow. Tracing let me see the two goals were fighting. We are at roughly 180ms p95 now.

def run_barge_in(session_id, turn_id, vad, agent_audio):
    with stage_span("barge_in", session_id, turn_id) as first_byte:
        span = trace.get_current_span()
        t0 = time.monotonic()
        vad.wait_for_user_speech()
        span.set_attribute("audio.vad_detect_ms", (time.monotonic() - t0) * 1000.0)
        agent_audio.cancel_generation(); first_byte()
        span.set_attribute("audio.cancel_ms", (time.monotonic() - t0) * 1000.0)
        agent_audio.flush_downstream_buffers()
        span.set_attribute("audio.silence_ms", (time.monotonic() - t0) * 1000.0)

The number I keep on the wall for the model layer is two numbers honestly: p95 first token, and p95 barge-in silence. Both have to be good.

TTS: first audio chunk, and the gap between sentences

TTS is streaming too, so the attribute that matters is first byte, the first chunk of playable audio. We page when p95 first-byte on TTS goes above 350ms, because past that the pause between the user finishing and the agent starting gets long enough that testers describe it as the agent thinking too hard. There is a second TTS thing a single first-byte number misses: the gaps between chunks once audio is flowing. If TTS stalls mid-sentence the user hears a stutter, and average latency looks fine. So I record the largest inter-chunk gap on the TTS span.

I keep ASR, the model, and TTS all using the exact same audio.first_byte_ms attribute name on purpose, even though "first byte" means a slightly different physical thing for each. Same name means one query pulls first-byte across all three stages and I compare them on one screen.

The client: jitter is the number, and you cannot see it from the server

Everything above is server side. The client receives audio over a network you do not control and plays it. The enemy is jitter: frames arriving unevenly. From the server everything can look healthy while the user hears choppy audio. So the client emits its own span per turn, with the jitter it measured and the buffer depth it settled on, shipped to the same collector with the same audio.session_id. Now a glitchy call shows the jitter right next to the three server spans. The honest caveat: client clocks are not synced to your server, so treat client timestamps as approximate. I trust the client span for the jitter and buffer values it reports about itself, not for lining its clock up to the millisecond.

This is the TraceQL I keep saved. It pulls p95 of first-byte latency, grouped by stage.

{ span.audio.stage != "" && span.audio.first_byte_ms >= 0 }
  | select(span.audio.stage, span.audio.first_byte_ms)
  | quantile_over_time(span.audio.first_byte_ms, 0.95) by (span.audio.stage)

The >= 0 filter is there because a stage that produced nothing gets first_byte_ms = -1, and I do not want those poisoning the percentile. To go from aggregate to a single bad call I filter by session: { span.audio.session_id = "sess_8f21c0" }. That gives every span for that session in time order, which is the entire reason I put session_id on every span. A word on percentiles, because it changes what you do: p50 first token might be 280ms and look fine, p99 might be 1,900ms, and in voice that p99 is a real human who had a two-second silence and probably said "hello? are you there?" into the void. Averages I mostly ignore.

What I am still chewing on

How do you set the client playout buffer when you cannot see the user's network until the call is already happening? Is barge-in even the right model, when VAD fires on a cough, an "mm-hm", the user's dog? And the question under all of it: I can trace every layer now, but I still do not have a number for "this call felt natural" that does not eventually come down to a human listening to it. The tracing tells me where time went. It does not tell me whether the conversation was any good.

If you are instrumenting a voice agent and you only have time to add one span this week, add barge-in. It is the one nobody measures and the one users feel the fastest.

3 OTel span attributes I tag on every voice-pipeline span

Marcus Chen — Tue, 02 Jun 2026 17:50:16 +0000

Voice pipelines have 4 stages that need separate latency stories: ASR (speech to text), LLM (the response prompt), TTS (text to speech), and client (jitter on the receiving end). When we wired OTel across all 4, the spans without consistent attributes were useless for queries. 3 attributes ended up on every span and earn their keep.

audio.stage.Enum: asr, llm, tts, client. The single most-queried attribute. The Grafana query for p95 latency by stage is one filter. Without this, you are scrolling raw traces.

audio.session_id: The full conversation. Lets you query "what did the user actually experience" end-to-end. We use a uuid generated at session start, propagated to every downstream call. Tempo's traces by tag lookup is fast on this.

audio.first_byte_ms: The time from request start to first audio byte returned. For ASR and TTS streaming stages. This is what catches barge-in latency regressions before the dashboard's aggregate alert does. We page when p95 first_byte_ms goes above 350ms on TTS.

Honorable mention attributes that didn't survive the first cleanup: audio.codec (covered by the service info), audio.session_turn_index (covered by parent-span linkage), audio.user_id (privacy concerns at scale; left out).

If you are starting voice-pipeline observability: tag stage + session_id on every span from day one. The first_byte_ms is the one you will add after the first production incident; you might as well add it on day two.

Voice agent latency is a lie. The number you care about is barge-in interrupt rate.

Marcus Chen — Tue, 26 May 2026 15:58:32 +0000

Last quarter we shipped our voice agent into production. The p99 end-to-end latency was 280 milliseconds. Our largest competitor's was 450 milliseconds. On every dashboard, we were faster.

Our user research panel said our agent felt slower.

The "felt slower" gap was 8 percentage points on a 5-point Likert. Statistically significant. We had been measuring the wrong thing.

It took us two weeks to figure out what the panel was actually measuring, and four weeks after that to fix the right number. The wrong number was end-to-end latency. The right number was barge-in interrupt rate.

Where the dashboard lied

Voice agent benchmarks measure response time. ASR converts speech to text, the LLM produces a response, TTS turns it into audio, you ship it. The end-to-end clock is what gets reported.

That clock is not what users experience as "speed."

What users experience is the loop between starting to interrupt the agent and the agent shutting up. If they say "wait" mid-sentence and the agent finishes the sentence first, that is a one-to-two-second pause from the user's perspective.

That gap, the barge-in delay, was 380 milliseconds for us. Our competitor's was 60 milliseconds. Users felt that gap on every interruption.

How we measured barge-in interrupt rate

The metric: of attempts where the user starts speaking during agent speech, what percentage result in the agent yielding within X milliseconds?

Two methods.

Synthetic. A corpus of 500 recorded interruption attempts pulled from prior support calls. We fed each audio segment into a copy of the agent and measured time-from-first-syllable to agent-stops-speaking.

python
# barge_in_eval.py (simplified)
def measure_barge_in(agent, recording):
    start = time.monotonic_ns()
    agent.play(recording.agent_response_audio)
    interrupt_t = start + recording.interrupt_offset_ns
    play_user_audio(recording.user_audio, at=interrupt_t)
    stop_t = wait_for_agent_silence()
    return (stop_t - interrupt_t) / 1_000_000  # ms

Real. Instrumented the production audio pipeline to emit one span when VAD (Voice Activity Detection) fires and another when TTS interrupts. Both go to OTel. Subtracting the timestamps gives the per-call barge-in latency.

Our barge-in interrupt rate at the 100ms threshold was 41%. At 250ms it was 89%, but 250ms is too slow to feel responsive.

The three things we changed

1. Pin the audio buffer pages

Our agent ran in a long-lived Tokio runtime. The audio buffers were allocated on the heap and occasionally got paged to swap when the model weights were active.

use libc::{mlock, c_void};

unsafe fn pin_buffer(buf: &[u8]) -> std::io::Result<()> {
    let ret = mlock(buf.as_ptr() as *const c_void, buf.len());
    if ret != 0 {
        return Err(std::io::Error::last_os_error());
    }
    Ok(())
}

After this, VAD detected user speech within 25ms of first syllable.

2. VAD threshold tuning

A/B tested 0.4 to 0.65 on the synthetic corpus. 0.5 was best. 4% earlier detection than 0.6 with only 1.2% false positive increase.

3. TTS interrupt path

The killer. Our TTS streamed audio in 200ms chunks. When VAD fired, the audio queue still held 400ms of buffered audio that played to completion. Users heard the agent finish a fragment of a sentence before silence.

async fn handle_barge_in(state: &mut AgentState) {
    state.llm_handle.cancel();
    state.tts_queue.clear();
    state.audio_out.stop().await;
}

We dropped chunk size to 30ms and flushed the queue immediately on VAD fire.

Results

Four weeks of work. Barge-in interrupt rate at 100ms threshold moved from 41% to 89%. The "felt slower" gap closed within one user research cycle.

Our actual p99 latency went up slightly (280ms to 305ms) because of the smaller TTS chunks. The dashboard number got worse. The user-felt number got dramatically better.

The number that mattered

Voice agent latency is the dashboard number. Barge-in interrupt rate is the user number.

Most voice agent teams I have talked to do not measure barge-in interrupt rate. They measure end-to-end latency, they get a number that feels low, they ship. Then their users say "your agent sucks" and the team cannot reconcile what the dashboard says with what the user says.

The reconciliation is the metric you are not tracking.

What I am still tuning

Eight months in, I have stopped trusting the dashboard more than the user research panel. The dashboard wants to be right. The panel just is.

The barge-in threshold itself is the part I am least sure about. 100ms is our target. 60ms is our competitor's. Whether 60ms gives a meaningful UX delta over 100ms for the users we serve, I genuinely cannot tell yet.

Distinguishing intentional from filler interrupts is the next obvious area. Yielding on "wait" is correct. Yielding on "mhm" is wrong. We currently treat both the same.

And the felt-slower measurement is the one I am most aware of being weak on. Our 5-point Likert is the best we have, and it is not great. If anyone is running rigorous voice agent UX studies, the methodology would be more useful to me than the dashboard ever will.