A voice assistant feels human at about 500 milliseconds from the end of your sentence to the start of its reply, and feels broken past about 1.2 seconds. That budget is spent by six things, most of which are fixed. This page derives the sum term by term so you can see which term is yours to fix, and gives the interruption logic in full, because barge-in is where every voice project actually stumbles.
The latency budget, derived
Mouth-to-ear latency is the wall-clock gap between the user’s last syllable and the first audible syllable of the reply. It decomposes into terms that add:
T_total = T_endpoint + T_upload + T_asr_final + T_prefill
+ T_first_token + T_tts_first_chunk + T_playout
T_endpoint silence you wait for before deciding the turn ended
T_upload last audio frame reaching the server
T_asr_final speech recogniser finalising the transcript
T_prefill model reading the prompt (grows with prompt length)
T_first_token model producing its first token after prefill
T_tts_first_chunk synthesiser producing its first audio chunk
T_playout that chunk reaching the speaker (buffering, jitter)
A worked budget with a 700 ms target:
T_endpoint 500 ms <- the largest term, and it is a CHOICE
T_upload 30 ms <- one network hop
T_asr_final 80 ms <- streaming recogniser, finalising only
T_prefill 90 ms <- ~1,500-token prompt
T_first_token 60 ms
T_tts_first_chunk 120 ms <- streaming synthesis, first chunk only
T_playout 80 ms <- 2 x 40 ms jitter buffer
--------
960 ms over budget by 260 ms.
Every number above is a placeholder chosen to show the shape of the sum, not a measurement. Replace each with one you time yourself; the value of the model is that it tells you where to look. And it points somewhere counter-intuitive: the biggest term is the silence you chose to wait for, and no model upgrade touches it.
The second observation is that only T_prefill scales with your design decisions. A 6,000-token system prompt costs roughly four times the prefill of a 1,500-token one, so a voice agent is the one place where trimming the system prompt buys user-visible latency rather than just money. Time to first token and tokens per second are different numbers and only the first one is in this budget — once audio is playing, the model only has to keep ahead of speech, which is roughly three tokens per second of audio.
Endpointing: deciding they stopped talking
Endpointing is the decision that a turn has ended. Wait too long and the assistant feels slow; wait too little and it interrupts somebody who was thinking. It is a single threshold with a large effect and it should not be one constant.
- Short silence, 200–350 ms — after a complete-sounding utterance ending in a falling pitch or a clear question. Feels snappy, occasionally cuts people off.
- Medium, 500–700 ms — the safe default, and the value in the budget above.
- Long, 900–1,200 ms — after a filler word (“um”, “so”, “and”), after a trailing conjunction, or when the recogniser’s partial transcript is grammatically incomplete. The user is mid-thought.
The cheap version of adaptive endpointing needs no model: take the recogniser’s running partial transcript and lengthen the timeout when the last token is a filler or a conjunction. That single rule removes most of the interruptions that make an assistant feel rude.
TRAILING = {"um", "uh", "er", "so", "and", "but", "because",
"like", "well", "the", "a", "to", "of", "for"}
def endpoint_ms(partial_transcript: str) -> int:
words = partial_transcript.strip().lower().split()
if not words:
return 700
if words[-1].strip(",.") in TRAILING:
return 1100
if partial_transcript.rstrip().endswith("?"):
return 300
return 600
The barge-in state machine
Barge-in means the user can talk over the assistant and be heard. It is pure state logic, it is where the bugs live, and it is short enough to write out completely.
The four states and what each does on the two events that matter — voice activity detected, and voice activity ended:
# barge_in.py — transport-agnostic. No API calls in here on purpose.
IDLE, LISTENING, THINKING, SPEAKING = "idle", "listening", "thinking", "speaking"
BARGE_IN_MS = 250 # sustained user speech required to interrupt
class Turn:
def __init__(self, io):
self.io = io # .stop_playback() .clear_output_buffer()
# .cancel_generation() .start_capture()
self.state = IDLE
self.speech_ms = 0
self.spoken_prefix = "" # what the user actually HEARD
def on_voice(self, frame_ms):
self.speech_ms += frame_ms
if self.state == SPEAKING and self.speech_ms >= BARGE_IN_MS:
# 1. stop the sound first: the user must hear the interruption work
self.io.stop_playback()
self.io.clear_output_buffer()
# 2. stop paying for tokens nobody will hear
self.io.cancel_generation()
# 3. remember only what was actually played, not what was generated
self.spoken_prefix = self.io.played_text()
self.state = LISTENING
elif self.state in (IDLE, THINKING) and self.speech_ms >= 60:
if self.state == THINKING:
self.io.cancel_generation()
self.state = LISTENING
def on_silence(self, silence_ms, partial):
if self.state == LISTENING and silence_ms >= endpoint_ms(partial):
self.speech_ms = 0
self.state = THINKING
return "commit" # caller sends the turn to the model
return None
Three details in there are the whole reason barge-in is hard.
- Order matters. Stop playback before cancelling generation. The perceptual event the user is waiting for is the sound stopping; cancelling a request first adds a network round trip to the thing that must feel instantaneous.
- Clear the buffer, not just the source. If you have two seconds of synthesised audio queued, stopping the synthesiser leaves the assistant talking for two more seconds. Everything already buffered has to be dropped.
- The conversation history must record what was heard. The model generated a full sentence; the user heard six words of it and then interrupted. If you append the full sentence to the history, the assistant later refers to something it never said, and the user concludes it is lying.
played_text()— the prefix that was actually rendered — is what goes in the transcript, marked as interrupted.
Then there is echo. If the user is on a speakerphone, the microphone hears the assistant, and naive voice-activity detection interprets the assistant’s own speech as a barge-in — so it interrupts itself, repeatedly, in a loop that is genuinely funny the first time. Acoustic echo cancellation is a solved problem in every mainstream WebRTC stack and is one of the strongest reasons to use one rather than raw WebSockets.
What the audio path has to do
This is where the page deliberately stops naming functions. Realtime speech APIs — the socket protocols, event names and audio formats — are the fastest-moving surface in this entire cluster, and a method name printed here would be wrong within months. So here is the requirement list to check any candidate against, which does not go stale:
| Requirement | Description |
|---|---|
| Duplex streaming | Audio up and audio down on one connection, both while the other is active. |
| Partial transcripts | Words as they are recognised, not only on finalisation — the endpointing rule above needs them. |
| Cancellation | A way to abandon an in-flight generation and stop being billed for it. Ask explicitly; not every API has one. |
| Chunked synthesis | First audio out before the full sentence is synthesised, or T_tts_first_chunk becomes the length of the reply. |
| Server-side VAD, optional | Convenient, but check whether you can override its endpoint timing. If not, you have lost your largest budget term. |
Check each of these against the current documentation of whatever you are considering, including whether cancellation stops billing or merely stops delivery. That distinction is rarely prominent and it decides what a heavily interrupted conversation costs.
Buying back latency you cannot remove
- Speculative start. Begin the model call on a partial transcript at around 300 ms of silence, and cancel if the user resumes. Turns the endpointing wait into prefill time you were going to spend anyway. Costs cancelled generations; measure the ratio before committing.
- Filler audio. A 300 ms “mm-hm” or “let me check” played immediately on commit hides the entire model latency behind something a human would also do. This is the highest-value trick on the list and the most disliked when overused — one filler per turn, never twice in a row.
- Sentence-level synthesis. Synthesise the first sentence as soon as it is complete rather than waiting for the whole reply. Requires streaming the model output and splitting on sentence boundaries, which is fifteen lines.
- Shorten the system prompt. Directly reduces prefill, every turn, for free. Voice agents accumulate prompt the way everything else does, and nobody notices because the cost shows up as sluggishness rather than a bill.
- Put the model near the user. Two network hops at 40 ms each are 80 ms of an 700 ms budget. Region choice is worth more here than in any text application.
Telephony changes the numbers
A voice agent in a browser and the same agent on a phone call are different engineering problems, and the budget above is the browser case. Four differences matter.
- Narrowband audio. Traditional telephony carries 8 kHz audio, which discards everything above about 3.4 kHz — the band that distinguishes “s” from “f”. Recognition accuracy is measurably worse than on the 16 kHz or better audio a browser gives you, and it is worst on exactly the things people spell out: names, postcodes, reference numbers.
- Extra hops. The carrier, your telephony provider and your server are three legs before the model sees anything, each adding to
T_uploadandT_playout. Budget a hundred milliseconds or so that simply is not there in a browser session, and measure it rather than assuming. - No echo cancellation you control. A speakerphone in a car is the barge-in nightmare case, and the mitigation is on the device rather than in your code. Raise
BARGE_IN_MSon phone sessions, and consider requiring speech that does not match what you are currently saying. - Key presses exist. Phone keypad tones are a perfectly reliable input channel and they cost nothing. For anything with a fixed answer set — confirm, cancel, an account number — offer the keypad as an alternative and take it over recognition every time.
The design consequence is to keep the agent conservative on the phone: shorter replies, explicit confirmations for anything consequential, and a documented path to a human that does not require the caller to convince the agent. A voice agent that cannot be escaped is the complaint that follows every deployment of one, and it is a product decision rather than a technical limit.
The failures that make it feel wrong
- It interrupts people who paused to think. Endpointing too aggressive. The adaptive rule above is the first fix; the second is lengthening the timeout after any turn the user immediately talked over.
- It keeps talking after being interrupted. Buffered audio was not cleared. Distinguish it from the previous fault by timing: a fixed one-to-two second tail is a buffer, a variable one is a cancellation that is not arriving.
- It answers a question the user did not finish asking. A speculative start that was not cancelled. Always discard a speculative result if any speech arrived after it was launched — do not try to reconcile it.
- It refers to things it never said. Interrupted text written to history in full. Covered above, and it is the failure users find most unsettling, because it looks like dishonesty rather than a bug.
The latency structure of voice agents and the transport choices underneath streaming both go further into individual terms of the budget above.
Top comments (0)