My first voice agent took 1,200ms to answer a spoken sentence. Then I rewrote three seams in the pipeline and it dropped to 340ms. No new hardware, no new models, no smaller LLM. The words the user says, the words the agent says back, the same. What changed was the shape of the wait.
If you have ever built a voice agent that felt polite but slow, this is the part of the pipeline where the seconds hide.
The 1,200ms baseline was polite and wrong
Here is what my first version did, in the order it did it:
- Record until the user stops talking (~200ms of tail silence).
- Send the whole clip to Whisper. Wait for the transcript.
- Send the transcript to the LLM. Wait for the full response.
- Send the full response to Piper. Wait for the WAV.
- Play the WAV.
Each stage was fine on its own. The pipeline was a one-lane road. Whisper could not start until recording finished. The LLM could not start until Whisper finished. Piper could not start until the LLM was done. The user waited for the sum.
The car metaphor gets old fast, so I will use a real one. This is what the timeline looked like on my machine:
[record]--[200ms silence]--[whisper 380ms]--[LLM 480ms]--[piper 340ms]--[playback]
^
1,200ms
Every one of those bars was blocking the next. I had built a relay race where each runner waited for the previous runner to sit down.
Trick 1: Frame-based STT so Whisper starts before the user stops
The first fix is to stop treating the user's speech as a single file. Feed the audio to Whisper in 20-30ms frames as it is captured. By the time the user hits the tail silence, most of the transcription is already done. You only wait for the last few frames plus a short flush.
Pipecat is the reference implementation. Its whole model is frame-based: every stage processes 20-30ms chunks and hands them forward as soon as they are ready. There is no batch, no full-clip handoff, no "wait for this stage to complete." Its own docs quote sub-500ms voice-to-voice when all models are hosted on the same GPU cluster.
If you do not want to adopt a full framework, the primitive is a VAD that emits chunks plus a streaming ASR endpoint. Deepgram, AssemblyAI, and Whisper-based streaming wrappers like WhisperX all support this shape.
For latency shopping in 2026: Deepgram's Nova-3 streaming endpoint targets sub-300ms server-side latency, and independent benchmarks put total client-side latency in the 200-500ms range once network transit is included. This is the number I actually planned around, not the marketing headline.
After this trick alone, my timeline changed shape:
[record + whisper (overlapping)]--[flush 80ms]--[LLM 480ms]--[piper 340ms]
^
900ms
I got 300ms back by removing a wait, not by getting anything faster.
Trick 2: Sentence-level pipelining so TTS starts on the first period
The next block was the LLM-then-TTS handoff. The LLM streams tokens. TTS wants full text. What I was doing was buffering all the tokens, then handing the whole response to Piper.
The trick is to buffer up to the first sentence boundary, hand that to TTS, then keep buffering. TTS starts synthesizing the first sentence while the LLM is still generating the second one. The user hears speech as soon as the first sentence is spoken.
import asyncio
async def stream_to_speech(llm_stream, tts, audio_out):
buffer = ""
async for token in llm_stream:
buffer += token
# Cheap sentence detection - period/question mark + space
if buffer and buffer[-1] in ".!?":
asyncio.create_task(tts.synthesize_and_play(buffer, audio_out))
buffer = ""
if buffer:
asyncio.create_task(tts.synthesize_and_play(buffer, audio_out))
The asyncio.create_task is doing the work here. It hands off synthesis without blocking the LLM stream. The LLM keeps producing tokens while Piper is already turning the first sentence into audio.
Time-to-first-audio now depends on how long the LLM takes to produce one sentence plus Piper's time-to-first-byte, not on the full response length. In my measurements the first-sentence LLM latency was ~180ms, Piper TTFB was ~120ms. First audio in 300ms after transcription completes.
New timeline:
[record + whisper]--[flush]--[LLM sent1 180ms][piper sent1 120ms]--[playback starts]
[LLM sent2 in parallel]
^
580ms to first audio
Trick 3: Token-level TTS with a 1-2 word lookahead
Sentence-level pipelining gets you into the sub-second range. To push under 400ms, you have to hand TTS smaller chunks than a sentence.
The naive version is to stream one token at a time. Do not do this. Piper (and most neural TTS) predicts prosody from context. Feed it one word at a time and it will sound like a chopped-up robot because it cannot see far enough ahead to pick the right intonation.
The empirical sweet spot is k=1-2 word lookahead. Wait for the token after the current word before you send the current word to TTS. Two words of context is enough for Piper to pick the right intonation for the first one.
async def token_stream_to_speech(llm_stream, tts, audio_out, lookahead=2):
words = []
async for token in llm_stream:
words.append(token)
if len(words) > lookahead:
chunk = words.pop(0)
await tts.synthesize_and_play(chunk, audio_out, prev_words=words[:lookahead])
for w in words:
await tts.synthesize_and_play(w, audio_out)
At this point my end-to-end timeline was:
[record + whisper]--[flush]--[LLM first 2 tokens][piper starts]--[playback]
^
340ms end to end
That is where 1,200ms went. Not into any single stage getting faster. Into the gaps between stages getting closed.
The network layer is doing work you cannot ignore
I skipped over this above, but the transport matters more than most tutorials admit. Sending audio over HTTP request-response adds 100-200ms per stage just from TCP setup and TLS handshake. WebSocket removes the repeated setup. WebRTC removes it and switches to UDP, so you are not paying for retransmit.
Here is the tradeoff table I keep on my desk:
| Protocol | Latency | Setup | Use case |
|---|---|---|---|
| WebRTC | 50-100ms | Complex | Browser/app real-time conversation |
| WebSocket | 100-200ms | Simple | Server integration, most chat |
| SIP | 200-400ms | Hardest | Legacy telephony, call centers |
I spent three days getting WebRTC ICE negotiation working. STUN, TURN, the whole tour. It was worth it for the 50-100ms floor, but if you are building a server-to-server pipeline where the user is on the other end of your app, WebSocket is a fine default and you can ship it in an afternoon.
LiveKit is the pattern I ended up with: LiveKit's SFU handles the WebRTC transport, Pipecat handles the STT-LLM-TTS orchestration. The SFU does selective packet forwarding without re-encoding, which is where the low overhead comes from. If you want the two-liner mental model: Pipecat is the chef, LiveKit is the runner.
What the 340ms floor is made of
There is no magic left after this. The 340ms breaks down roughly as:
- ~80ms: audio flush after user stops talking
- ~180ms: LLM time-to-first-token (varies wildly by model)
- ~80ms: Piper TTFB on first chunk
- Rest is network transit, framing, and jitter
The LLM number is the one I cannot compress further without changing the model. That is why I stopped at 340ms. Getting to sub-300 would require either a faster model, speculative decoding, or moving to something like Gemini 2.0's native audio-in-audio-out where the STT→LLM→TTS boundary does not exist. All of these are their own project.
If you are chasing sub-300, that is where the road forks: parallelize what you have (this article), then either shrink the LLM or unify the pipeline (a different article).
The one thing I would tell past me
Stop optimizing single stages. Time each stage in isolation, sure, but the wins are in the gaps between them. Every stage boundary in a naive pipeline is a place where nothing is happening. Frame-level STT, sentence-level TTS handoff, token-level TTS lookahead - each one is the same trick applied at a different granularity. Close the gap.
The version of this write-up in the book has the full timeline diagrams, the code for a working Pipecat + LiveKit + Whisper + LLM + Piper pipeline, and the perceptual tricks that let you feel faster than 340ms even when the physics stops giving: The Voice AI 300ms UX Guide. Chapter 7 is the parallelization playbook the article above compresses. Chapter 8 is what to do when you cannot compress further.
Top comments (0)