Why waiting longer makes voice AI worse

Daksh Verma — Sun, 02 Aug 2026 09:53:38 +0000

Most voice agents decide you are done talking by measuring silence. Voice activity detection flags the audio as speech or not speech, a timer runs during the not-speech, and when the timer crosses a threshold the turn ends and generation begins.

It is a reasonable first architecture. It is also wrong in a way that gets worse the more natural the conversation gets.

The failure looks like this. Someone is thinking while they speak. They pause mid-sentence, because that is what people do when they are constructing a thought rather than reading one out. The pause crosses the threshold. The system ends the turn and starts responding, and the user gets cut off by a machine, mid-thought.

The fix that presents itself immediately is to raise the threshold. Wait longer before deciding the turn is over.

This is the part worth writing down: raising the threshold trades one failure for a worse one. Latency in conversation is not a smooth cost curve. Below a certain point people do not perceive a delay at all. Above it, they perceive a system that is slow, and slowness in a voice interface reads as broken in a way that visual latency does not, because there is no spinner, no loading state, nothing to indicate the machine is alive. Silence from a voice agent is indistinguishable from failure. So you push the threshold up to stop the interruptions, and you lose the thing that made the product feel real.

Both ends of the tuning range are bad. That is the actual shape of the problem, and it is why threshold tuning is not the answer at any value.

The reframe that helped: silence is the weakest available signal for a decision this important, and it is the only one a pure VAD approach is using. Humans do not detect turn ends by timing gaps. We use syntax, whether the clause has resolved. Prosody, whether the pitch fell into a terminal contour or stayed suspended. Semantics, whether the thing said is complete enough to answer. We are running a continuous prediction of whether the other person is finished, and we start preparing our response before they actually stop.

Which points at a different architecture. Instead of a timer that fires on silence, run a lightweight endpointing model over the partial transcript that outputs a probability the turn is complete, and let silence be one input to that rather than the whole decision. A trailing-off pause after a resolved clause and a pause mid-clause are the same event to a VAD and completely different events to anything that reads the text.

The second piece is decoupling detection from commitment. A turn-end prediction does not have to be a single irreversible decision. You can begin generating on a probable endpoint and cancel cheaply if the user resumes. That converts a hard classification problem into a soft one, and soft problems tolerate being wrong.

None of this is solved. Barge-in handling, cancellation cost, and what happens when the user talks over the response are all still open on my end, and I would rather write about them while they are open than after I have tidied them into a conclusion.

If you have shipped real-time voice, I want to hear what your endpointing actually looks like in production, particularly what you do when the prediction fires early and generation has already started.

DEV Community: Daksh Verma

Why waiting longer makes voice AI worse