I made my voice agent slower on purpose

Karthik R — Sat, 25 Jul 2026 04:53:26 +0000

My pipeline was fast enough to interrupt people. Here is what I changed, what it cost, and the second bug that made calls go deaf without anyone noticing.

I run a voice AI sales training platform. Users practice cold calls against an AI buyer, in real time, over WebRTC. The stack is Pipecat for orchestration, LiveKit for transport, Deepgram Flux for STT, Cartesia Sonic for TTS, and a small fast model over OpenRouter for the conversation.

In July I had a pipeline I was proud of: p95 of about 880 ms from confirmed end of turn to the first audio byte coming back. Then I listened to real calls, and the AI buyer kept talking over people.

Not at natural turn boundaries. Mid-sentence. A rep would say "so what we do is...", pause half a second to think, and the buyer would answer the unfinished thought.

The pipeline was working exactly as designed. That was the problem.

Speed made it worse, not better

My endpointing was pinned at the aggressive end of the config range:

Setting	Value
`VOICE_EOT_THRESHOLD`	0.5
`VOICE_EOT_TIMEOUT_MS`	700
Eager end-of-turn	on, at 0.35
Turn-commit buffer	0 ms

A one-second thinking pause was enough to commit the turn. Then an 880 ms pipeline delivered a full reply before the caller got going again.

Here is the part I had not internalized. If my pipeline had been slower, the same premature endpointing would have been less noticeable. The reply would have landed after the caller resumed, barge-in would have caught it, and the turn would have recovered. Being fast is what converted a detection error into an interruption.

Latency and endpointing are not independent. Every millisecond you shave off the pipeline raises the cost of a premature end-of-turn decision. Speed is not free. It amplifies your endpointing errors.

Almost every voice latency post I have read treats the pipeline as the thing to optimize and endpointing as a config detail. In my experience it is the reverse. The pipeline was the easy part.

What I changed

I ran a live A/B on staging and production on July 15:

Setting	Before	After
Eager end-of-turn	on (0.35)	off
`VOICE_EOT_THRESHOLD`	0.5	0.7
`VOICE_EOT_TIMEOUT_MS`	700	1200

The cut-ins stopped. The reply now lands roughly half a second later than before, and it reads as a natural beat rather than a delay. Nobody has complained that the agent is slow. Plenty of people had complained that it interrupted.

Then I rewrote the constraint in my own docs, which is the actual takeaway:

Processing p95 must stay under 1 second from the confirmed end of turn to first audio. Listening patience is deliberate and sits outside that budget.

Two different numbers, with two different failure modes. Conflating them is how you end up optimizing the one that was already fine.

The numbers, with percentiles attached

Stage	Contribution
Deepgram Flux STT	~350 ms
VAD fallback	~280 ms
Cartesia Sonic TTS	~250 ms
Total, p95	~880 ms

Some honesty about where that actually sits. Typical production voice deployments in 2026 land around 680 ms p50 and 1,180 ms p95. Well-optimized stacks reach sub-250 ms p50. So 880 ms p95 is solid, not remarkable, and I am quoting the tail rather than the median.

I mention it because a lot of latency posts quote p50 and call it "sub-second." If you are comparing stacks, insist on the percentile. The p95 is what your users actually feel, because the tail is exactly where a conversation breaks down. A median that looks great and a p95 that does not is an agent that feels fine in a demo and frustrating in production.

One small thing that genuinely helped: chunking the first TTS request. I send the first speakable fragment to Cartesia as soon as I have 4 to 8 tokens, capped at a 250 ms wait, rather than buffering a full sentence. It costs nothing and buys a few hundred milliseconds of perceived responsiveness on every single turn.

The second bug: the agent that went deaf and kept talking

The Deepgram Flux websocket is long-lived, and it will occasionally drop abruptly mid-call. The signature in my logs was no close frame received or sent. Rare, but it showed up in Sentry as a class of failure, not a one-off.

My code had a _startup_error latch. Its original job was reasonable: if the connect probe fails at session start, fall back to nova-3. But run_stt treated that latch as permanent. So a mid-call drop that also failed a single reconnect attempt set the latch, and the agent was deaf for the rest of the call while continuing to hold a conversation from context.

Two latent hazards made it considerably worse:

The watchdog got pinned off by the failure itself. If the drop happened mid-user-turn, _flux_user_speaking stayed stuck true. My inactivity watchdog skips its checks while either party is speaking. So a deaf call would never time out. It would run forever, burning credits and vendor spend, with a user talking to something that had stopped listening minutes earlier.
A dead connection that reported itself healthy. A non-connection error in the receive loop killed transcription while the socket object still said open. _is_connected() returned true. Nothing anywhere detected it.

The fix

The latch now engages only before the first successful connect. The session-start probe and the nova-3 fallback behave exactly as before.
After that, any receive-loop death or send failure marks the connection dead, and _is_connected() returns false even when the socket object claims otherwise.
The per-chunk run_stt path retries for the whole call, with capped backoff (0.25s, 0.5s, 1s, 2s, then 5s ceiling), each attempt bounded at 5 seconds. Audio chunks arriving inside the backoff window are dropped, so the pipeline never blocks on a connection already known to be down.
Marking the connection dead force-emits UserStoppedSpeakingFrame, which unpins the watchdog.
start() now swallows a transient connect failure with a warning so the StartFrame still propagates. Raising there left every downstream processor unstarted.

Retry forever is a deliberate decision. An outage of any length recovers the moment Deepgram is reachable again. An outage that never recovers gets ended by the inactivity watchdog, not by a latch. Latches are how you end up with permanently deaf calls.

Logging policy matters as much as the retry logic: a warning on every attempt, exactly one Sentry error per outage (fired on the third consecutive failure), and an info line with total outage duration on recovery. That gives you real signal on sustained outages without drowning in transient blips.

Zero cost on the healthy path, because all of it only runs when _is_connected() is already false. Eleven new tests, suite at 690 passing.

Three things I would tell you

Measure endpointing separately from pipeline latency. They are different numbers with different failure modes, and making the pipeline faster makes endpointing errors more visible, not less.
Assume every long-lived websocket in your stack will drop mid-call. Make "dead" a state you can leave rather than one you latch into, and check that your watchdogs are not disabled by the very failure they exist to catch.
State the percentile. Every time.

The counterintuitive lesson underneath all three: my users never asked for a faster agent. They asked for one that would let them finish a sentence.

I build real-time voice systems: Pipecat, LiveKit, Deepgram, Cartesia. The product above is ConvoSparr. I am currently taking on voice AI contract work, so if you are fighting latency, turn-taking, or reliability in a production voice agent, get in touch: karthik@convosparr.com