The first version of my voice agent took 4.2 seconds to answer a question. Not "felt slow." Measured: 4,247ms median from the last syllable a human spoke to the first syllable that came back. Voice AI latency is the one metric where users don't need your dashboard to notice the regression. They just say "hello?" into the silence and then start repeating themselves, which produces a second transcript, which the bot also answers.
I spent about three weeks taking that number apart. Here's the honest waterfall, what each fix actually bought in milliseconds, and the three things that broke when it got fast.
TL;DR
- Median round-trip went from 4,247ms to 780ms. The LLM was never the main problem.
- The single biggest win was not waiting for the full completion: stream the first sentence into text-to-speech. Worth ~1,430ms on its own.
- The second biggest was turn detection, not inference. A fixed 1,200ms silence window was burning more time than the model did.
- Dropping the silence window to a flat 250ms made it fast and awful: it cut people off on 31% of turns. Adaptive endpointing got that to 6%.
- p95 is still 1.6s, almost entirely prompt-cache misses when someone thinks quietly for several minutes. The slowest answer goes to the person who paused the longest, which is exactly backwards.
What actually causes voice AI latency?
Voice AI latency is a stack of six serial waits, and most of them are not the model. Here's my measured baseline, median over 40 recorded sessions:
| Stage | Baseline |
|---|---|
| Endpointing (fixed 1,200ms silence window) | 1,200ms |
| Deepgram final transcript after endpoint | 310ms |
| LLM time-to-first-token (6.8k-token system prompt, uncached) | 1,050ms |
| Rest of the completion (structured JSON, ~180 tokens) | 1,240ms |
| ElevenLabs full generation before playback | 380ms |
| Network + player buffer | 67ms |
| Total | 4,247ms |
Look at what that table says. Inference is 2,290ms of 4,247. The other 1,957ms is me waiting on purpose: waiting for silence to "confirm" the turn ended, waiting for a complete JSON object, waiting for a finished audio file. Every one of those waits was a default I never chose.
How do you measure voice agent latency without lying to yourself?
Measure at the speaker, not at the server. My first instrumentation put a timestamp where the request hit my backend, and it reported numbers about 200ms better than reality. The browser's audio capture buffer and the upload leg are invisible from there, and they are latency the user feels.
What I ended up with:
- Record the raw input track per session.
- Run offline voice-activity detection over that recording to find
t_last_speech, the true end of the human's last syllable. - Timestamp first audible output frame on the client, not "response sent" on the server.
- Every stage event goes into one monotonic clock. Mixing
Date.now()across two machines gave me a stage that appeared to take negative time, which is how I found the bug.
The system all of this runs in is Preterview, a platform I built and operate that runs realistic voice interviews with three interviewer styles and returns a scored written report (full disclosure: I built it, preterview.com/en). It's a useful testbed for latency work because the failure is so legible: a human is talking to it under pressure, and a pause that would be fine in a chat UI reads as the machine being broken. Most of the numbers in this post came out of that production traffic, not a synthetic loop.
What actually cut the latency, ranked by milliseconds saved
1. Stream TTS from the first sentence boundary (-1,430ms). The original code did await llm.complete() then await tts.generate() then play. Nothing about that is necessary. Send the first complete sentence to ElevenLabs the moment the token stream produces one, keep feeding it, and the user hears audio while the model is still writing. This deleted the 1,240ms "rest of completion" wait and replaced a 380ms full generation with a ~90ms first chunk.
2. Prompt caching on the system prompt and session context (-740ms). My system prompt plus the running interview state is around 6.8k tokens and it is identical turn to turn. Uncached time-to-first-token was 1,050ms. Cached, 310ms. This is the cheapest win in the entire post: it's a cache-control marker on a prefix that never changes.
3. Adaptive endpointing (-920ms). A fixed 1,200ms silence window assumes every pause means the same thing. It doesn't. If the transcript ends on a complete clause ("...so that's how I handled the migration"), I wait 220ms. If it ends on a filler or a hanging conjunction ("...and then, um"), I wait 900ms. Median across real turns: 280ms.
4. Plain text in the speaking path (-180ms). The turn response was JSON, because the scoring pipeline wanted structured fields. But the scoring doesn't have to happen in the same call the human is waiting on. I split it: the spoken turn is plain text, and a second non-blocking call does the structured extraction after the audio is already playing. Constrained decoding also made time-to-first-token less predictable, and the tail matters more than the median when the user is staring at a microphone.
Final measured budget, same methodology:
| Stage | After |
|---|---|
| Adaptive endpoint decision | 280ms |
| Deepgram final transcript | 120ms |
| LLM TTFT (cached prefix, plain text) | 180ms |
| Tokens to first sentence boundary | 70ms |
| ElevenLabs first audio chunk | 90ms |
| Network + player buffer | 40ms |
| Total | 780ms |
What broke when I made it fast?
Barge-in went from a bug to a feature request. At a flat 250ms endpoint I clocked 418 turns by hand across 12 sessions. The bot started talking over the human on 31% of them. People pause mid-sentence to think, and a thinking pause and a finished pause look identical to a silence timer. Adaptive endpointing dropped it to 6%. Not zero. 6% of turns still get stepped on, and I have not found a clean fix that doesn't cost me back 400ms.
Sentence splitting is not split('.'). My first chunker cut "3.5 years" into "3." and "5 years at a startup," and did the same to "Node.js." The audio came out with a hard stop in the middle of a number. The fix was boring: require period + space + capital letter, and enforce a 60-character minimum chunk before flushing to TTS.
Streaming ahead wastes audio you never play. When a user interrupts, the TTS stream is already two sentences past what got heard. At one point 18% of generated speech characters were discarded audio nobody listened to. Capping the lookahead to two sentences pulled that down to about 7%. The fast version of the pipeline cost measurably more per session than the slow one, which is not the direction I expected.
The p95 belongs to quiet people. p95 is 1.6s, and the cause is almost entirely prompt-cache expiry. Cache entries have a time-to-live. A candidate who goes quiet for several minutes composing a thought comes back to a cold prefix and pays the full uncached TTFT. The person who most needed a fast, encouraging reply gets the slowest one. I now fire a cheap keepalive during long silences, which helps, and I'd rather report the ugly tail than a median that flatters me.
So what's the real answer on voice AI latency?
If your voice agent feels slow, the model is probably not your bottleneck. Instrument the pipeline end to end from the last human syllable to the first audible output frame, and you'll usually find that most of the wall clock is deliberate waiting: a fixed silence window before you'll admit the turn ended, a full completion before you'll start speaking, a full audio file before you'll start playing. Streaming text-to-speech from the first sentence boundary, prompt-caching a static prefix, and making the endpoint threshold adaptive to how the sentence ended took my own system from 4,247ms to 780ms median without changing the model. Then budget for the consequences: faster turn detection means more interruptions, and streaming ahead means paying for speech nobody hears.
Written by the developer behind Preterview, an interview prep platform.
Top comments (0)