DEV Community: Marcus Chen

Three weeks before the enterprise contract, the voice agent wasnt operator-ready.

Marcus Chen — Thu, 02 Jul 2026 23:19:50 +0000

Three weeks before the enterprise contract, the voice agent wasn't operator-ready

Look. We had 99.2% uptime in staging. We had eval coverage on 1,400 test turns. We had latency under 280ms first-token.

We were not operator-ready.

I know this because the enterprise pilot started on a Monday and we had our first critical incident by Tuesday afternoon.

This is what happened, what broke, and what the gateway layer decision actually looks like when you're under pressure to fix it fast.

The incident

The customer was a wealth management firm. Their advisors use a voice agent to pull client portfolio data, answer allocation questions, and schedule follow-ups. We'd been testing with synthetic personas for six weeks. The simulation results were clean.

Day one: a senior advisor ran a session that included three back-to-back allocation queries with large portfolio values. Our OpenAI rate limit hit at 6pm EST, right during peak advisor usage. Every request after the limit returned a 429. The agent logged nothing useful. The advisor's client was on hold for 4 minutes.

Day two: a compliance officer tried to pull the audit log for the day-one incident. There wasn't one. We had trace spans. We did not have a per-request log that showed which advisor, which client context, which tool calls, what the agent responded. That's a compliance gap, not a monitoring gap.

Week two: the VP of operations asked for the cost breakdown by team. We gave them a single number. They wanted per-advisor attribution. We had no per-tenant tagging.

Week three: the operations team pushed a new prompt version to fix a tone issue. Three hours later, the voice agent started refusing certain allocation questions it had previously handled fine. We had no prompt version pinned at inference time in the trace. We couldn't tell when the failure started or which requests were affected.

Four incidents. None of them were model quality issues. All of them were the gateway layer we hadn't built.

What the gateway layer is supposed to do

Before this pilot I thought of the gateway as routing. Send the request to OpenAI, or Anthropic, or whichever provider. Handle retries. Done.

That was wrong.

The gateway for an enterprise operator deployment does at least five things:

Rate limiting per tenant. Not per account. Per tenant. An advisor with heavy usage should not blow the rate limit for the entire deployment.

Cost attribution. Every request tagged with the operator, the team, the user. Without this, you cannot answer the cost-attribution questions that come in month two.

Guardrail enforcement. For financial services: no advice that sounds like a specific investment recommendation. The guardrail needs to run on every response, not just when you remember to add it.

Audit logging. Immutable, per-request, with enough context to replay the interaction. This is a compliance requirement for most regulated industries, not a nice-to-have.

Multi-provider failover. When OpenAI hits 429, route to Anthropic. Not as a manual intervention. Automatically. The 4-minute incident on day one was preventable.

What I evaluated

After week one, I spent most of a weekend evaluating gateway options. Here's the honest breakdown:

LiteLLM (open-source, self-hosted). Most complete feature set if you want full control. Per-tenant rate limiting, cost tagging, provider fallback, proxy mode. The setup complexity is real: you need to maintain the deployment, configure Redis for rate-limiting persistence, and write your own audit log schema. For a team with Kubernetes infrastructure already in place, this is probably the right call. We were mid-pilot and needed faster setup.

Portkey (managed). Zero-config guardrails, built-in prompt versioning with a rollback UI, solid multi-provider routing. Pricing gets expensive at scale but the managed model means fast setup and less ops overhead. Their guardrail policies are more configurable than LiteLLM's out of the box. We ended up here for the pilot because we were under time pressure and needed zero-setup guardrail enforcement.

Future AGI's gateway (open-source, part of the future-agi platform). This is the gateway component of their end-to-end eval + observability + guardrail stack. It handles multi-provider routing with guardrail policies, rate limiting, and OTel-native tracing that connects to the same OTel-based observability stack as the rest of the platform. I evaluated this specifically because we were already running FAGI's simulation tooling for our voice eval harness, and the unified stack had real appeal: guardrails, tracing, and eval running through the same FAGI platform.

For a team already on the FAGI platform for eval and simulation, the gateway is the right next layer. For a team coming in cold with no FAGI tooling, the setup cost is higher than Portkey or Helicone for the first-time operator deployment.

As of June 2026, the FAGI gateway ships the OpenAI-compatible proxy, multi-provider routing, guardrail policies, and OTel tracing in one stack.

Helicone (managed). Strongest on cost attribution and per-user analytics. The tagging system is granular and the dashboard is readable. Weaker on guardrails (less configurable than Portkey). Right call if your primary need is FinOps visibility and you're handling guardrails separately.

OpenRouter (managed). Pure routing. Multi-provider fallback, good for latency optimization across providers. Does not have per-tenant rate limiting or guardrail enforcement built in. Not the right call for an enterprise deployment that needs compliance features.

Bifrost (open-source). Fast proxy with interesting performance numbers. Newer, smaller community. I evaluated it and the latency story is real. But it was too new to commit to for a regulated industry deployment.

Week three: what we fixed

We were already deployed on Portkey for rate limiting and guardrail enforcement by week three. We added per-advisor tagging to every request. We pinned prompt versions at inference time and logged the version ID in each trace span.

The prompt-version incident would have been caught immediately with version pinning. The cost-attribution ask would have been answered in two SQL queries.

The audit log took longer. Financial services audit logging has specific retention and immutability requirements that generic trace systems don't satisfy out of the box. We built a thin write-once layer on top of Portkey's logging that met the compliance spec. That was two days of work we should have done before the pilot.

What shipped

Portkey for rate limiting and guardrail enforcement. Per-tenant tagging on every request. Prompt version pinning at inference time. Custom audit log layer for compliance.

The rate-limit incident did not recur. The cost-attribution question now takes two minutes to answer. The audit log is compliance-satisfying.

What I'd tell past me

Architect the gateway before you talk to the enterprise customer. Not as an afterthought when the pilot starts hitting limits.

The questions you'll be asked in month one: "Who spent what, when, doing what, with what outcome." If your gateway doesn't answer those four questions, you are not operator-ready. The model quality is probably fine. The infrastructure around it is what will bite you.

And if you're already running FAGI's eval and simulation stack: evaluate their gateway component in parallel. The unified data model between guardrails, traces, and eval signals is genuinely useful for regulated deployments where you need the audit trail to connect back to eval coverage.

What I'm building next: a pre-operator readiness checklist that runs as a CI gate before any enterprise handoff. It checks per-tenant rate limit configuration, audit log schema coverage, and prompt version tracking. None of these should be manual.

The 2am call that dropped before the user finished talking, and the week I spent finding out why my tracer never saw it

Marcus Chen — Wed, 01 Jul 2026 21:51:13 +0000

The call came in at 2am. Not a page, an actual support recording, flagged by a customer who said our voice agent "hung up on her mid-sentence." I pulled the trace. The LLM call was perfect. 380ms, clean completion, sensible response. Every dashboard I had was green. The customer was still angry, and my tooling had nothing to say about why.

That gap is the thing I want to talk about. I build voice agents for a living, the kind that answer phones and book appointments and occasionally embarrass me in production. And after three years of it, here is the hard lesson: tracing the LLM call is the easy 20 percent. For a voice agent, the failures live in the audio layer your tracer never sees.

Week 1: learning what my dashboards were hiding
When the LLM is the whole product, an LLM tracer is enough. You see the prompt, the completion, the tokens, the cost, the latency. Beautiful.

A voice agent is a pipeline, and the LLM is one stage in the middle. Audio comes in, an ASR model transcribes it, an endpointer decides when the human stopped talking, your orchestration assembles context, the LLM responds, TTS speaks it back, and somewhere a barge-in detector is supposed to notice when the human interrupts. The LLM trace covers one box in that chain. The 2am call dropped because the endpointer fired early. The transcript was cut in half before it ever reached the model. My tracer logged a flawless response to half a question.

So I made a list of what actually breaks, and what I needed to see for each:

End-of-turn detection timing. The endpointer decides the human is done. Too eager and you interrupt them (my 2am call). Too slow and the agent feels dead. This is a latency-plus-decision event, not an LLM span, and most tools have no concept of it.

ASR latency and confidence. If transcription takes 900ms or comes back at 0.4 confidence, the LLM response can be instant and still wrong. You need the confidence score attached to the turn.

Barge-in detection. The human starts talking over the agent. Did the system notice? How fast did it stop talking? Pure audio-layer, invisible to a text tracer.

Time-to-first-audio. Not time-to-first-token. The human hears nothing until TTS produces sound. That is the latency that matters, and it lives downstream of everything your LLM dashboard shows.

None of these are exotic. They are the daily failure modes of every voice agent in production. And the tooling conversation almost never mentions them.

Week 2: checking six tools against that list
I went through six observability tools I had either used or seriously trialed, and asked one question of each: how much of the audio layer can I actually see, and how much work is it to get there. I am grading on voice-agent fit, not on general quality. Several of these are excellent tools that simply were not built with a pipeline like mine in mind.

Langfuse. OpenTelemetry-based, so the format does not fight you. You can attach custom spans for ASR, endpointing, time-to-first-audio, and they show up in the trace tree. Honest take: on pure LLM observability Langfuse is stronger and more polished than most of this list, including the mid-list option I will get to. The catch is that nothing about the audio layer is automatic. You instrument every span by hand.

Phoenix (Arize). Same OTel story. Format-agnostic, custom spans work, strong on eval and drift if that is your world. Same catch: the audio spans are yours to define and emit.

Laminar. OTel-native and newer, pleasant to instrument. Same pattern: it will hold whatever audio spans you send it, it will not invent them for you.

Future AGI (traceAI). Sits in the middle of this list for me, and I want to be precise about why. Its tracing layer, traceAI, is OpenTelemetry-native and exports OTLP to any backend, with instrumentors for 50-plus frameworks as of June 2026 (the repo is open at github.com/future-agi/traceAI). For voice work that buys you the same thing the others do: custom audio spans are first-class because OTel is the substrate. Where it earned its spot for me is the eval side, scoring a turn against the audio context rather than just the text. Where it does not win: on raw observability ergonomics, Langfuse and Helicone are simply more refined. I keep it mid-list on purpose. It is a capable option, not a crown.

Helicone. Genuinely excellent at LLM-call logging, cost tracking, and gateway-level visibility, and the fastest of this group to stand up for that job. It is also largely silent on the audio layer. That is not a flaw, it is a focus. If your problem is LLM cost and call logging, Helicone may beat everything here. If your problem is a dropped call at 2am, it will not see it.

LangSmith. The most LLM-centric of the six and the least audio-aware by default. Tight integration if you live in the LangChain world. You will be doing the most adapting to make a voice pipeline legible inside it.

The pattern, once I lined them up, was almost boring. The OpenTelemetry-native tools (Langfuse, Phoenix, Laminar, traceAI) can all represent the audio layer, because OTel does not care whether a span wraps an LLM call or an endpointer decision. The LLM-focused tools (Helicone, LangSmith) are sharper at the thing they are built for and quieter about everything else. Nobody on this list ships voice-agent observability that works out of the box. Every one of them needs you to define the audio spans yourself.

Week 3: the instrumentation that actually paid off
The fix was not a tool swap. It was deciding to instrument the audio layer first and treat the LLM trace as already solved, because it was. Concretely, every turn now emits spans for ASR (with latency and confidence as attributes), endpoint decision (with the timing that would have caught the 2am drop), and time-to-first-audio. Because that is plain OpenTelemetry, it lands in whatever backend I point it at. The LLM span, the one thing all six tools handle beautifully, is the least of my attributes now.

What shipped, and what I would tell the version of me who pulled that 2am trace
What shipped: a voice pipeline where the endpointer's decision is a first-class, traceable event, and the dashboard that used to glow green on a broken call now shows the early-fire spike that caused it. Mean-time-to-the-real-cause on audio-layer bugs went from "listen to the recording and guess" to "read the span." The 2am class of incident is now a saved query.

What I would tell past me: stop staring at the LLM trace. It was always going to be green. The part of the system that was actually deciding whether the call worked, when the human gets to stop talking, how fast they hear a reply, whether an interruption registered, was the part you had not instrumented at all. Pick whichever OpenTelemetry-native tool fits your wallet and your team, the choice between them matters less than people pretend. Then spend your week emitting audio spans, not comparing dashboards. The LLM layer is the solved problem. The voice layer is the one that pages you at 2am.

The 1.4 Seconds That Weren't on Any Span

Marcus Chen — Wed, 24 Jun 2026 23:38:04 +0000

On the morning of June 3rd, a customer on a live call sat through 1.4 seconds of dead air after she finished a sentence, long enough that she said "hello?" before the agent answered. I had the trace open in Honeycomb forty seconds later. Every span was green. End-to-end p95 read 980ms, comfortably under our budget, and not one span in that waterfall was longer than 400ms. The dashboard told me everything was fine while the customer was, in fact, talking to silence.

TL;DR: End-to-end voice latency is not the sum of your spans. The number that kills your UX lives in the unattributed time between spans, most often the gap between turn-end (the moment the user stops talking) and ASR-start (the moment your pipeline begins transcribing). APM-style tracing instruments the work and ignores the waiting, so the gap is invisible by construction. You have to put a span on the handoff itself.

Here is the thing I keep saying and keep being right about: most "LLM observability" is just APM with extra steps. It watches the model. It traces the LLM call, the tool call, the retrieval, the token count, all the parts a backend engineer already knows how to think about. For a voice agent that is the wrong half of the system. Voice agents do not break inside the LLM call. They break in the audio pipeline, in the orchestration between components, in the handoffs nobody owns a span for. Your model can be fast and your product can still feel broken, and your dashboard will not say a word about it.

What the dashboard showed

Our turn looks like this on paper. VAD/turn-detection decides the user is done. Audio goes to ASR (Whisper Large v3, streaming). The transcript goes to the LLM (gpt-4o-realtime) for a first token, then the full response. The response streams to TTS (ElevenLabs) for the first audio byte, which is the moment the user hears anything. There is network on both ends.

I pulled the one trace from the 1.4-second call. Not an aggregate, the actual trace. Here is the latency budget I had been staring at for weeks, the summed-span view:

Stage	p50	p95	p99	who owns the span
VAD / turn-detection	60ms	120ms	180ms	orchestrator
ASR (streaming)	180ms	310ms	540ms	ASR client
LLM TTFT	220ms	380ms	720ms	model client
LLM full response	140ms	260ms	430ms	model client
TTS first byte	90ms	190ms	360ms	TTS client
Network (both legs)	40ms	90ms	150ms	gateway

Add the p95 column. It comes to roughly 1340ms. Our reported end-to-end p95 was 980ms (the percentiles do not stack, a single request rarely hits the tail on every stage at once, so the real end-to-end p95 sits below the naive sum). Fine. Either way, both numbers are wrong about the call that paged me, because the call that paged me had 1.4 seconds the table does not contain. None of these rows is the dead air. The dead air is the white space between two of them.

Pulling the one trace

When you look at a single voice turn in a normal tracing UI, you get a waterfall of bars. Each bar is a span. The instinct, the APM instinct, is to find the longest bar and optimize it. I spent two days doing exactly that. I made ASR faster. I shaved 40ms off TTFT with a prompt cache. The summed bars got shorter and the dead air did not move, because the dead air was never a bar.

Here is the timeline I finally drew on a whiteboard, because the tracing UI would not draw it for me.

Figure: one voice turn. The captured spans are short and correct, but they start 1400ms late. The damage is the unattributed gap to their left.

That bracket on the left is the whole post. The spans were honest. They were short, they were green, they summed to a healthy number. They just started 1.4 seconds after the user stopped talking, and nothing in the trace measured the wait, because the code path between "turn-detection fired" and "ASR client opened a stream" did not open a span. It awaited a coroutine, hit a connection-pool stall under load, and sat there. Silent. Unspanned. Invisible.

Root cause

The turn-detection callback handed off to ASR through a queue, and the ASR client lazily established its streaming connection on first use. Under concurrent calls, that connection setup contended on a pool that was sized for steady state, not for the moment six calls all finished a turn inside the same 200ms window. So turn-end fired, the handoff coroutine queued the audio, and then waited on a connection that was busy being born. By the time the ASR span opened, 1.4 seconds had passed. The ASR span itself then ran in 300ms, green and blameless.

The fix is two parts. Put a span around the handoff so the gap stops being invisible. Then fix the pool. You cannot fix what you cannot see, and the entire reason this lived in production for weeks is that the gap was never a measurable thing.

Here is the real instrumentation. This is OpenTelemetry Python, opentelemetry-api and opentelemetry-sdk, the actual SDK calls, runnable.

from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

tracer = trace.get_tracer("voice.turn")


async def handle_turn(audio_in, ctx):
    # The outer span is the whole turn, anchored at turn-end.
    with tracer.start_as_current_span(
        "voice.turn",
        kind=SpanKind.SERVER,
    ) as turn_span:
        turn_span.set_attribute("call.id", ctx.call_id)
        turn_span.set_attribute("turn.index", ctx.turn_index)

        # THE MISSING SPAN: turn-detection -> ASR-start handoff.
        # Everything that happens between "user stopped talking" and
        # "ASR actually began" gets measured here, including the wait
        # for a streaming connection that used to be invisible.
        with tracer.start_as_current_span("voice.handoff.vad_to_asr") as hs:
            hs.set_attribute("handoff.from", "turn_detection")
            hs.set_attribute("handoff.to", "asr")
            try:
                asr_stream = await asr_client.open_stream(ctx)
            except Exception as exc:
                hs.set_status(Status(StatusCode.ERROR, str(exc)))
                hs.record_exception(exc)
                raise
            # mark when audio truly starts flowing into ASR
            hs.add_event("asr_stream_ready")

        # ASR itself. Short and green. Never the problem.
        with tracer.start_as_current_span("voice.asr") as asr_span:
            transcript = await asr_stream.transcribe(audio_in)
            asr_span.set_attribute("asr.transcript_chars", len(transcript))

        # LLM and TTS spans continue as before.
        with tracer.start_as_current_span("voice.llm") as llm_span:
            reply = await llm_client.complete(transcript, ctx)
            llm_span.set_attribute("llm.model", ctx.model)

        with tracer.start_as_current_span("voice.tts") as tts_span:
            first_byte = await tts_client.first_audio_byte(reply)
            tts_span.set_attribute("tts.first_byte_ms", first_byte.elapsed_ms)

        return reply

The point is the voice.handoff.vad_to_asr span. It wraps the dead zone between two components that each had their own span and were each, individually, fast. Now the wait has a name and a duration. The next time six calls finish a turn at once, the handoff span balloons to 1400ms and the connection-pool stall is right there in the waterfall instead of hiding in the white space.

And once the span exists, you can query for it. Here is the trace query I now run, written for a backend that speaks SQL-ish over spans (Honeycomb's query builder maps to the same idea, and so does any OTLP store you can point at ClickHouse). It surfaces turns where the handoff alone blew past 250ms:

SELECT
  trace_id,
  call_id,
  duration_ms AS handoff_ms
FROM spans
WHERE name = 'voice.handoff.vad_to_asr'
  AND duration_ms > 250
ORDER BY handoff_ms DESC
LIMIT 50;

That query returns nothing on a normal day and lights up the instant the pool starts contending. I wired it to an alert on the handoff span's p95, not the end-to-end p95, because the end-to-end p95 is exactly the number that lied to me on June 3rd.

The pool fix was unglamorous. Pre-warm the ASR streaming connections, size the pool for burst concurrency instead of average, and keep the connections alive between turns instead of opening lazily. Handoff p95 went from 1400ms on the bad call down to 70ms steady-state. The dead air was gone the same afternoon I shipped the span, because the span told me precisely where to put the fix.

What this does NOT solve

Instrumenting the handoff makes the gap visible. It does not make your infrastructure fast. A few honest limits.

It does not fix jitter under load on its own. The span tells you the handoff is slow, but if your pool, your event loop, or your GC is the bottleneck, you still have to go fix that. The span is a flashlight, not a wrench.

It does nothing about provider-side queueing you cannot see. When ElevenLabs or your ASR vendor queues your request on their side, your client-side span measures the wait but cannot attribute it past the boundary. You will know that you waited, not why the provider made you wait. For that you need their status, their rate-limit headers, sometimes a support ticket.

And it will not catch every gap automatically. I added the VAD-to-ASR span because that is where this fire was. There are other handoffs (ASR-to-LLM, LLM-to-TTS, barge-in cancellation) and each one needs its own span if you want to see its gap. Instrument the ones that hurt first.

Lesson: Instrument the handoffs, not just the calls. A green waterfall of short, correct spans can still add up to a customer saying "hello?" into silence, because the damage is the time between the bars, and a trace only shows you the bars you drew. The day I stopped trusting the summed p95 and started putting spans on the gaps is the day the dead air stopped paging me. If you run voice agents, go find your turn-end-to-ASR-start handoff right now, wrap it in a span, and alert on that span alone. It is the cheapest 1.4 seconds you will ever buy back.

The Retry That Booked Mrs. Alvarez Twice

Marcus Chen — Wed, 24 Jun 2026 23:27:22 +0000

Week one of the pilot, our voice agent booked appointments for a dental group. Forty operatories, three locations, one phone line per office that never stopped ringing. The agent took the call, checked the calendar, wrote the slot, read it back. Clean.

The 9pm page came on a Thursday. Front desk had found four double-booked slots from that afternoon. Same callers, same times, two rows each in the scheduling table. The agent swore (in the transcript) it booked once. The database swore it booked twice. Both were telling the truth.

Here is what actually happened. Our booking call went out to the practice management API. That API was slow that day, p95 around 4200ms, sometimes worse. We had a 3000ms timeout on the HTTP client. So the request would land, the booking would commit on their side, and our client would give up waiting before the 201 came back. The agent saw a timeout, treated it as a failure, and said the line every voice agent says when something goes wrong: "sorry, let me try that again." Then it fired the same booking a second time. The second one was fast enough to return. Two rows. One confused Mrs. Alvarez.

The retry was the bug. Not the slowness. Slowness is normal. The sin was retrying a write that had no idempotency key, so the downstream system had no way to know the second request was the same intent as the first.

The fix was small and boring, which is the best kind. Generate a stable key per booking intent (not per HTTP attempt) and pass it through. If the agent decides to book the 2pm slot for this caller, that decision gets one key, and every retry of that decision carries it.

import hashlib

def booking_key(call_id: str, slot_iso: str, provider_id: str) -> str:
    # one key per intent. survives retries, timeouts, agent re-prompts.
    raw = f"{call_id}:{slot_iso}:{provider_id}"
    return hashlib.sha256(raw.encode()).hexdigest()[:32]

resp = client.post(
    "/appointments",
    json=payload,
    headers={"Idempotency-Key": booking_key(call_id, slot_iso, provider_id)},
    timeout=8.0,  # also: stop timing out under their real p95
)

Two things mattered together. The key made the duplicate write a no-op on the server (their API honored Idempotency-Key, most modern ones do, and if yours does not, you build the dedup yourself with a unique constraint on those three fields). And the timeout went from 3000ms to 8000ms, because a 3000ms ceiling on a 4200ms p95 is not a timeout, it is a duplicate-booking generator with extra steps.

We shipped the key first, that same night. Double-bookings went to zero across the next 1,800 calls. The timeout bump went out the next morning after I pulled a week of latency histograms and saw the real tail.

What I would tell week-one me: a voice agent retrying a write is not retrying a question. When the agent says "let me try that again," something on the other end may already be true. Decide what one action means before you let the agent do it twice. Put the key on the intent, not the attempt, and timeouts on the real numbers, not the round ones.

LLM observability tools are blind to the voice layer. Here is what I checked 6 of them for.

Marcus Chen — Thu, 18 Jun 2026 22:56:51 +0000

Tracing the LLM call is the easy 20 percent. For a voice agent, the failures live in the audio layer your tracer never sees.

Most LLM observability tools trace the same thing: the prompt, the completion, the tokens, the latency of the model call. For a text agent that is most of the story. For a voice agent it is maybe a fifth of it, because the failures that actually make a voice agent feel broken happen in the audio layer, and a tracer pointed at the LLM call cannot see them. I went through six observability tools (Langfuse, Helicone, Arize Phoenix, LangSmith, Braintrust, and Laminar) asking one question each: can it show me the audio layer, or only the LLM call?

The audio layer is where the real spans are. End-of-turn detection: how long did the agent wait before deciding the caller was done? ASR latency and confidence: how long did transcription take, and how sure was it? Barge-in: did the caller interrupt, and did the agent yield? Time-to-first-audio: how long from the caller finishing to the agent making a sound? None of these are LLM-call metrics, and a green LLM-latency dashboard tells you nothing about any of them. I have watched a voice agent with a perfectly healthy model-call trace feel sluggish and rude to every caller, because the lag and the interruptions lived in spans the tracer was not capturing.

So here is how the six landed, all on the same question. Langfuse, Phoenix, and Laminar are OpenTelemetry-based, which is the good news: OTel does not care whether a span is an LLM call or an ASR call, so you can emit custom spans for endpointing, ASR, and barge-in and see them next to the model call. The catch is you have to instrument those spans yourself; none of them ship voice-aware instrumentation, they give you the canvas. Helicone is gateway-first, so it is excellent at LLM-call logging and cost and largely silent on the audio layer unless you add your own telemetry around it. LangSmith is deep on the LLM and LangChain trace and the most LLM-call-centric of the set, least aware of audio by default. Braintrust gives you a clean UI for whatever you send it, so again the audio layer shows up only if you instrument it.

The pattern is the same across all six: the tool is only as voice-aware as the spans you feed it, and the ones built on OpenTelemetry make that easy because you are just emitting more spans into a format they already understand. That is the actual selection criterion for a voice agent, not the LLM-tracing features every one of them advertises, but whether the model lets you put audio-layer spans right next to the model spans so "it feels slow" maps to a stage instead of a guess.

If I were choosing today for a voice agent, I would pick an OpenTelemetry-native tool and spend the first day instrumenting the audio layer, endpoint timeout, ASR latency and confidence, barge-in events, time-to-first-audio, before touching a single LLM metric. The LLM trace is the part that is already solved. The voice layer is the part that is invisible, and invisible is where the incidents hide.

The open question I have not cracked: even with audio-layer spans, "the call felt off" is a subjective, whole-conversation judgment that does not reduce cleanly to any single span. I can show you the endpoint timeout and the barge-in count, but not why the caller hung up frustrated. If anyone has tied per-span audio telemetry to a felt-quality score for a whole call, that is the conversation I want.

Our voice agent passed every test and still woke me up at 3am

Marcus Chen — Thu, 11 Jun 2026 10:35:24 +0000

Replaying real call transcripts as your test set is a trap. The failures come from the inputs a user produces exactly once.

TL;DR: Our voice-agent regression suite was 312 recorded production calls, all passing. The page at 3am came from a caller who switched between English and Hindi mid-sentence, a pattern that appeared zero times in those 312 calls. Replaying real transcripts tests the confidence you already have. It does not test the inputs that actually break you. We moved to simulating adversarial callers, and below is what I learned trying five tools to generate and grade those simulated conversations (as of June 2026).

The test set was real, and that was the problem
For about four months our regression set was 312 recorded production calls. It felt rigorous. Real audio, real ASR output, real user intents, replayed on every deploy. Green for weeks.

Then the 3am page. A caller switched between English and Hindi inside single sentences. Our ASR mis-segmented the mixed-language audio, the intent classifier saw garbage, and the agent fell into a clarification loop it could not exit. The caller hung up. The dashboards were fine the whole time.

I went looking for that pattern in the 312 calls. It was not there. Not once. The people who code-switch like that had mostly churned months earlier, so the behavior was absent from the recordings exactly because it was a problem we never handled. A test set built from past traffic contains what already happened, weighted toward the common case. The failures that page you are rare by definition, and rare things are missing from a sample of the past.

Why replay gives false confidence
Replaying recorded calls is a regression test for behavior you have already seen. That is useful. It catches the case where a deploy breaks something that used to work. What it cannot do is produce an input you have never received. For that you have to manufacture the input on purpose: the fast talker who never pauses, the caller who interrupts the agent two words in, the code-switcher, the person who changes their mind halfway through a sentence, the line with a TV on in the background. That is simulation, and it is a different activity from replay.

What I tried to generate and grade simulated calls
Five tools, roughly a week each, same eight adversarial caller profiles. None of these is voice-specific; I drove them off transcripts plus a separate ASR/TTS layer. Honest notes, your mileage will differ:

Promptfoo: fast to wire into CI and good for red-teaming a prompt with generated variants. The fiddly part was that conversation state across turns was a manual build.
LangSmith: dataset versioning and the trace view were the best of the set. The simulation half I had to assemble myself.
Future AGI Simulate: persona-based, you define caller personas and it runs them through the agent, which matched how I already thought about adversarial callers (as of June 2026). Voice was not first-class, so I ran it on transcripts with ASR and TTS bolted on.
Braintrust: the nicest UI for eyeballing where a run diverged. Persona definitions lived outside it, in my code.
DeepEval: the most knobs for synthetic-conversation generation. Tuning the synthesizer to stop producing unrealistic turns took a while.
Confident AI: a reasonable hosted layer on top of DeepEval, though it is another account and key to manage. I am deliberately not crowning one. Braintrust had the UI I liked, DeepEval had the most generation control, and the persona abstraction in Future AGI's Simulate (part of their open work at github.com/future-agi) lined up with how I list out adversarial callers. Any of them can run a persona once you have written the persona.

The thing that actually moved the needle was not the tool
It was the persona list. Once we had written eight adversarial callers (the angry caller, the two-words-then-silence caller, the code-switcher, the background-noise line, and so on), every tool above could run them and grade the results. The leverage was in naming the failure modes, not in the framework that executed them. We spent two days arguing about the personas and twenty minutes wiring the runner.

The open question I still have
The space of adversarial callers is infinite, and we maintain eight. We chose those eight from incident postmortems, which means we are still only simulating failures we have already been burned by at least once. The genuinely novel failure, the next 3am page, is still unguarded. I do not have a principled way to pick simulation personas before the incident teaches me the persona. If you have one, that is the comment I want to read.

FAQ
Why not just add each failed call to the regression set after the incident?
We do. It is still reactive. The replay suite trails production by one outage, permanently.

Doesn't simulated traffic drift away from what real users do?
Yes, and that is a real cost. We re-sample the real call distribution monthly and adjust how often each persona fires. Simulation supplements replay; it does not replace it.

Is any of this voice-specific?
Most of it applies to text agents too. Voice just adds two more failure surfaces: ASR segmentation and barge-in timing. The code-switching incident was really an ASR segmentation failure that a text agent would never have hit.

The 4-layer voice-agent latency stack, traced with OTel spans

Marcus Chen — Tue, 09 Jun 2026 09:28:48 +0000

How I instrument ASR, LLM, TTS, and the client with OpenTelemetry, and which number in each layer I actually look at

TL;DR. A voice agent is four moving parts stuck together: speech to text, the model that writes the reply, text to speech, and the client that plays the audio back. End to end latency hides which of those four is slow on any given turn, so I stopped tracking it as one number and started tracing each stage as its own OTel span with a shared session id. The number I watch hardest is barge-in: when the user starts talking over the agent, how many milliseconds until the agent actually stops sending audio. In our setup we want that under 200ms, and when p95 barge-in creeps past that, the agent feels like it is talking at you instead of with you. Everything below is how I wire the spans, what attributes go on each one, and the p95 I page on per layer.

The thing I keep saying, and the thing that keeps being true: voice agents fail in production not because of raw latency but because nobody simulated the audio and LLM pipeline together. You can have a fast ASR, a fast model, a fast TTS, and a voice agent that still feels broken, because the failure lives in the seams between them and in the parts (barge-in, jitter) that no single-stage benchmark touches. Tracing is how I get the seams to show up.

A note before the layers. This is just the setup we run, the spans we emit, and the mistakes that made us add each attribute. Some of it is probably specific to our stack and will not transfer. I will flag that where I can.

The shape of a turn, and why one span is not enough

One turn is: user says a thing, agent says a thing back. Underneath that is roughly: audio frames come in, ASR turns them into text (streaming partials as it goes); the text plus history goes to the LLM, which streams tokens back; as text comes out, TTS turns it into audio, also streaming; the client receives audio frames and plays them, with some buffering to smooth out jitter.

If you wrap the whole turn in a single span and call it voice.turn, you get a duration and almost no ability to act on it. A 1,400ms turn could be a slow first token, or TTS waiting on the full sentence before it starts, or the client buffering too aggressively. Same total, three different fixes.

So the parent span is voice.turn, and each stage is a child span. Every span carries the same audio.session_id and an audio.turn_id, so I can pull one turn out of Tempo and see all four stages laid out in time. The attribute I care about most on the streaming stages is not total duration. It is first byte: how long until the stage produced its first useful output. First byte is what the user feels, because all three stages are streaming and the user starts perceiving progress at the first byte, not the last.

import time
from contextlib import contextmanager
from opentelemetry import trace

tracer = trace.get_tracer("voice.pipeline")

@contextmanager
def stage_span(stage, session_id, turn_id):
    span = tracer.start_span(f"audio.{stage}")
    span.set_attribute("audio.stage", stage)
    span.set_attribute("audio.session_id", session_id)
    span.set_attribute("audio.turn_id", turn_id)
    started = time.monotonic()
    state = {"fb": False}
    def mark_first_byte():
        if state["fb"]:
            return
        span.set_attribute("audio.first_byte_ms", round((time.monotonic() - started) * 1000.0, 1))
        state["fb"] = True
    try:
        with trace.use_span(span, end_on_exit=False):
            yield mark_first_byte
    except Exception as exc:
        span.record_exception(exc); span.set_attribute("audio.error", True); raise
    finally:
        span.set_attribute("audio.total_ms", round((time.monotonic() - started) * 1000.0, 1))
        if not state["fb"]:
            span.set_attribute("audio.first_byte_ms", -1)  # produced nothing
        span.end()

Calling it around the LLM stage: you call first_byte() inside the streaming loop the first time a token shows up, and the wrapper does the timing math.

async def run_llm_stage(session_id, turn_id, messages, llm_client):
    chunks = []
    with stage_span("llm", session_id, turn_id) as first_byte:
        async for token in llm_client.stream(messages):
            first_byte()           # no-op after the first call
            chunks.append(token)
    return "".join(chunks)

I use time.monotonic() and not time.time() on purpose. Wall clock can jump (NTP corrections), and on a sub-second budget a backwards clock gives you negative latencies that poison the percentiles. One more thing I learned the annoying way: audio.session_id is high cardinality, so I keep it as a span attribute for trace lookup, but I do not turn it into a metric label. Stage goes on the metric label. Session id stays on the trace.

ASR: measure first partial, not final transcript

The mistake I made first was timing ASR as audio-in to final-transcript-out. That number is real but it is not the one that matches what the user feels, because a streaming ASR gives you a partial transcript fast and then refines it. So the span gets two numbers: audio.first_byte_ms is time to first partial, and I stash time to final separately.

The other ASR attribute that earned its place is whether the final transcript disagreed badly with the last partial. We had an incident where ASR turned a customer saying they wanted to confirm an order into the word cancel, and the agent acted on it. After that I started recording a rough measure of how much the final revised the partial, so big late revisions show up in traces instead of only in an angry support ticket. What I look at for ASR: p95 of time to first partial. In our setup that sits under 150ms most days, and when it drifts up it is almost always the audio frames not arriving on time from the client, not the ASR model. A nice example of why you trace the whole thing.

The LLM: first token is the whole ballgame, and barge-in lives here too

For the model stage, total generation time barely matters for the felt experience, because TTS consumes tokens as they arrive. What matters is time to first token. If the model takes 600ms before the first token, the user hears 600ms of silence after they stopped talking, and that feels like the agent froze. So the LLM span's headline attribute is time to first token.

Barge-in is the part people forget to instrument, and the part I would instrument first if I were starting over. It is what happens when the user starts talking while the agent is still speaking. The metric: from the moment voice-activity detection fires, to the moment the agent's outbound audio actually goes quiet. The first time we measured it, it was around 500ms and felt terrible, and the breakdown showed most of the time was not detection. It was buffered TTS audio we had already shipped toward the client and could not un-send. We had buffered aggressively to fight jitter, and that same buffer made barge-in slow. Tracing let me see the two goals were fighting. We are at roughly 180ms p95 now.

def run_barge_in(session_id, turn_id, vad, agent_audio):
    with stage_span("barge_in", session_id, turn_id) as first_byte:
        span = trace.get_current_span()
        t0 = time.monotonic()
        vad.wait_for_user_speech()
        span.set_attribute("audio.vad_detect_ms", (time.monotonic() - t0) * 1000.0)
        agent_audio.cancel_generation(); first_byte()
        span.set_attribute("audio.cancel_ms", (time.monotonic() - t0) * 1000.0)
        agent_audio.flush_downstream_buffers()
        span.set_attribute("audio.silence_ms", (time.monotonic() - t0) * 1000.0)

The number I keep on the wall for the model layer is two numbers honestly: p95 first token, and p95 barge-in silence. Both have to be good.

TTS: first audio chunk, and the gap between sentences

TTS is streaming too, so the attribute that matters is first byte, the first chunk of playable audio. We page when p95 first-byte on TTS goes above 350ms, because past that the pause between the user finishing and the agent starting gets long enough that testers describe it as the agent thinking too hard. There is a second TTS thing a single first-byte number misses: the gaps between chunks once audio is flowing. If TTS stalls mid-sentence the user hears a stutter, and average latency looks fine. So I record the largest inter-chunk gap on the TTS span.

I keep ASR, the model, and TTS all using the exact same audio.first_byte_ms attribute name on purpose, even though "first byte" means a slightly different physical thing for each. Same name means one query pulls first-byte across all three stages and I compare them on one screen.

The client: jitter is the number, and you cannot see it from the server

Everything above is server side. The client receives audio over a network you do not control and plays it. The enemy is jitter: frames arriving unevenly. From the server everything can look healthy while the user hears choppy audio. So the client emits its own span per turn, with the jitter it measured and the buffer depth it settled on, shipped to the same collector with the same audio.session_id. Now a glitchy call shows the jitter right next to the three server spans. The honest caveat: client clocks are not synced to your server, so treat client timestamps as approximate. I trust the client span for the jitter and buffer values it reports about itself, not for lining its clock up to the millisecond.

This is the TraceQL I keep saved. It pulls p95 of first-byte latency, grouped by stage.

{ span.audio.stage != "" && span.audio.first_byte_ms >= 0 }
  | select(span.audio.stage, span.audio.first_byte_ms)
  | quantile_over_time(span.audio.first_byte_ms, 0.95) by (span.audio.stage)

The >= 0 filter is there because a stage that produced nothing gets first_byte_ms = -1, and I do not want those poisoning the percentile. To go from aggregate to a single bad call I filter by session: { span.audio.session_id = "sess_8f21c0" }. That gives every span for that session in time order, which is the entire reason I put session_id on every span. A word on percentiles, because it changes what you do: p50 first token might be 280ms and look fine, p99 might be 1,900ms, and in voice that p99 is a real human who had a two-second silence and probably said "hello? are you there?" into the void. Averages I mostly ignore.

What I am still chewing on

How do you set the client playout buffer when you cannot see the user's network until the call is already happening? Is barge-in even the right model, when VAD fires on a cough, an "mm-hm", the user's dog? And the question under all of it: I can trace every layer now, but I still do not have a number for "this call felt natural" that does not eventually come down to a human listening to it. The tracing tells me where time went. It does not tell me whether the conversation was any good.

If you are instrumenting a voice agent and you only have time to add one span this week, add barge-in. It is the one nobody measures and the one users feel the fastest.

3 OTel span attributes I tag on every voice-pipeline span

Marcus Chen — Tue, 02 Jun 2026 17:50:16 +0000

Voice pipelines have 4 stages that need separate latency stories: ASR (speech to text), LLM (the response prompt), TTS (text to speech), and client (jitter on the receiving end). When we wired OTel across all 4, the spans without consistent attributes were useless for queries. 3 attributes ended up on every span and earn their keep.

audio.stage.Enum: asr, llm, tts, client. The single most-queried attribute. The Grafana query for p95 latency by stage is one filter. Without this, you are scrolling raw traces.

audio.session_id: The full conversation. Lets you query "what did the user actually experience" end-to-end. We use a uuid generated at session start, propagated to every downstream call. Tempo's traces by tag lookup is fast on this.

audio.first_byte_ms: The time from request start to first audio byte returned. For ASR and TTS streaming stages. This is what catches barge-in latency regressions before the dashboard's aggregate alert does. We page when p95 first_byte_ms goes above 350ms on TTS.

Honorable mention attributes that didn't survive the first cleanup: audio.codec (covered by the service info), audio.session_turn_index (covered by parent-span linkage), audio.user_id (privacy concerns at scale; left out).

If you are starting voice-pipeline observability: tag stage + session_id on every span from day one. The first_byte_ms is the one you will add after the first production incident; you might as well add it on day two.

Voice agent latency is a lie. The number you care about is barge-in interrupt rate.

Marcus Chen — Tue, 26 May 2026 15:58:32 +0000

Last quarter we shipped our voice agent into production. The p99 end-to-end latency was 280 milliseconds. Our largest competitor's was 450 milliseconds. On every dashboard, we were faster.

Our user research panel said our agent felt slower.

The "felt slower" gap was 8 percentage points on a 5-point Likert. Statistically significant. We had been measuring the wrong thing.

It took us two weeks to figure out what the panel was actually measuring, and four weeks after that to fix the right number. The wrong number was end-to-end latency. The right number was barge-in interrupt rate.

Where the dashboard lied

Voice agent benchmarks measure response time. ASR converts speech to text, the LLM produces a response, TTS turns it into audio, you ship it. The end-to-end clock is what gets reported.

That clock is not what users experience as "speed."

What users experience is the loop between starting to interrupt the agent and the agent shutting up. If they say "wait" mid-sentence and the agent finishes the sentence first, that is a one-to-two-second pause from the user's perspective.

That gap, the barge-in delay, was 380 milliseconds for us. Our competitor's was 60 milliseconds. Users felt that gap on every interruption.

How we measured barge-in interrupt rate

The metric: of attempts where the user starts speaking during agent speech, what percentage result in the agent yielding within X milliseconds?

Two methods.

Synthetic. A corpus of 500 recorded interruption attempts pulled from prior support calls. We fed each audio segment into a copy of the agent and measured time-from-first-syllable to agent-stops-speaking.

python
# barge_in_eval.py (simplified)
def measure_barge_in(agent, recording):
    start = time.monotonic_ns()
    agent.play(recording.agent_response_audio)
    interrupt_t = start + recording.interrupt_offset_ns
    play_user_audio(recording.user_audio, at=interrupt_t)
    stop_t = wait_for_agent_silence()
    return (stop_t - interrupt_t) / 1_000_000  # ms

Real. Instrumented the production audio pipeline to emit one span when VAD (Voice Activity Detection) fires and another when TTS interrupts. Both go to OTel. Subtracting the timestamps gives the per-call barge-in latency.

Our barge-in interrupt rate at the 100ms threshold was 41%. At 250ms it was 89%, but 250ms is too slow to feel responsive.

The three things we changed

1. Pin the audio buffer pages

Our agent ran in a long-lived Tokio runtime. The audio buffers were allocated on the heap and occasionally got paged to swap when the model weights were active.

use libc::{mlock, c_void};

unsafe fn pin_buffer(buf: &[u8]) -> std::io::Result<()> {
    let ret = mlock(buf.as_ptr() as *const c_void, buf.len());
    if ret != 0 {
        return Err(std::io::Error::last_os_error());
    }
    Ok(())
}

After this, VAD detected user speech within 25ms of first syllable.

2. VAD threshold tuning

A/B tested 0.4 to 0.65 on the synthetic corpus. 0.5 was best. 4% earlier detection than 0.6 with only 1.2% false positive increase.

3. TTS interrupt path

The killer. Our TTS streamed audio in 200ms chunks. When VAD fired, the audio queue still held 400ms of buffered audio that played to completion. Users heard the agent finish a fragment of a sentence before silence.

async fn handle_barge_in(state: &mut AgentState) {
    state.llm_handle.cancel();
    state.tts_queue.clear();
    state.audio_out.stop().await;
}

We dropped chunk size to 30ms and flushed the queue immediately on VAD fire.

Results

Four weeks of work. Barge-in interrupt rate at 100ms threshold moved from 41% to 89%. The "felt slower" gap closed within one user research cycle.

Our actual p99 latency went up slightly (280ms to 305ms) because of the smaller TTS chunks. The dashboard number got worse. The user-felt number got dramatically better.

The number that mattered

Voice agent latency is the dashboard number. Barge-in interrupt rate is the user number.

Most voice agent teams I have talked to do not measure barge-in interrupt rate. They measure end-to-end latency, they get a number that feels low, they ship. Then their users say "your agent sucks" and the team cannot reconcile what the dashboard says with what the user says.

The reconciliation is the metric you are not tracking.

What I am still tuning

Eight months in, I have stopped trusting the dashboard more than the user research panel. The dashboard wants to be right. The panel just is.

The barge-in threshold itself is the part I am least sure about. 100ms is our target. 60ms is our competitor's. Whether 60ms gives a meaningful UX delta over 100ms for the users we serve, I genuinely cannot tell yet.

Distinguishing intentional from filler interrupts is the next obvious area. Yielding on "wait" is correct. Yielding on "mhm" is wrong. We currently treat both the same.

And the felt-slower measurement is the one I am most aware of being weak on. Our 5-point Likert is the best we have, and it is not great. If anyone is running rigorous voice agent UX studies, the methodology would be more useful to me than the dashboard ever will.