Every voice assistant makes one decision more often than any other and discusses it less than any other: the person went quiet, have they finished the thought or are they taking a breath? You can get it wrong in both directions and both are obvious. Move too early and you cut the person off mid-sentence. Play it safe and they sit in silence wondering whether they were heard at all. Between those two failures lies a gap of a few hundred milliseconds, and almost the whole industry closes it the same way, with a timer.
We are building WhyWave, a voice companion for children aged three to five, and this timer is where we came to a halt. What follows is what we found while working out why.
On the time a machine actually has
Start with how this works between people, because people are what sets the feeling of a live conversation. The largest measurement available is the 2009 PNAS paper by Stivers and colleagues: ten languages, from English and Japanese to Tzeltal and Yélî-Dnye, 350 consecutive questions in each, 101 conversations. They measured the offset between the end of a question and the start of the answer. The overall mode of the distribution is zero milliseconds, which is to say the most common case is a person starting to answer at the exact moment the other one stops. The cross-linguistic median is plus one hundred milliseconds and the mean across the whole set is plus two hundred and eight; Japanese is fastest with a mean of seven milliseconds, Danish slowest at four hundred and sixty-nine.
The consequence we keep in front of us at WhyWave is this. People answer each other faster than almost any pipeline of recognition, language model and synthesis can run. A person in conversation starts talking before the other has finished, because they predict where the sentence is going. A machine waits for silence, and half its budget is gone right there.
How that budget gets spent varies. The standard approach: a voice activity detector splits the stream into speech and non-speech, and a timer sits on top of it, so many milliseconds of continuous silence and the turn is declared over. In OpenAI's Realtime API, server_vad treats half a second as the end of a turn: silence_duration_ms defaults to 500, another three hundred milliseconds of audio before speech onset is pulled back in through prefix_padding_ms, the activity threshold sits at 0.5, and the documentation says plainly that a shorter value makes the model respond faster while it starts cutting into short pauses. Pipecat sets VAD_STOP_SECS to two tenths of a second. LiveKit Agents puts min_delay at half a second and max_delay at three, and those drop to 0.3 and 2.5 once the audio turn-detection model is switched on. Silero VAD has min_silence_duration_ms at one hundred. A fivefold spread across thresholds has a simple explanation: there is no universally correct value, and each team picks its own point on the same curve, with interruptions at one end and dithering at the other.
On silence as a signal
The trouble is that silence inside a turn and silence after a turn are physically identical. The only thing separating them is what the person was about to say, and that is not in the signal. How often people go quiet in the middle of their own thought was measured by Melissa Redford in her work on pauses in children's and adults' narratives: adults in her sample put seven per cent of their pauses in grammatically incomplete positions, after a preposition, a conjunction, an article, between an auxiliary and its main verb, while for five-year-olds that figure was eighteen per cent. The five-year-olds also stopped nearly twice as often, 1.98 times per hundred words, and that effect holds up in the paper. The sample is tiny, ten pairs of a child and their own parent, so it points a direction without settling anything, and the direction is clear enough: the younger the speaker, the more of their silence falls in the middle of a phrase.
With children the timer stops working for three reasons at once. They answer more slowly: in the work of Casillas, Bobb and Clark on a corpus of English-language dialogue the median response latency for children was 625 milliseconds against 371 for adults in the same material, and while it shrinks with age, from 867 milliseconds at one year eight months to 523 by three and a half, that trend did not reach statistical significance. Set that beside a default of five hundred milliseconds: a child who is only just getting ready to answer has already been written off as finished. Recognition is the second reason. In a published benchmark of children's speech, zero-shot Whisper-large returns 43.9 per cent word error on ages four to seven against 18.6 on teenagers, so the end of the turn is placed wrongly and the content of the turn is read wrongly too. The third is that smooth turn transitions, where a person comes in exactly on the pause, settle in around the age of six, and before that a child is still learning to predict when the other speaker will stop.
On models in place of a timer
The last two years have produced a whole class of solutions where a model decides the turn is over. OpenAI added semantic_vad to the Realtime API: a classifier looks at the words spoken, scores the probability that the person is done and sets the timeout dynamically, with an impatience setting, eagerness, that defaults to auto, which is equivalent to medium, and maximum timeouts of eight, four and two seconds for low, medium and high.
Pipecat went further and put its model in the open. Smart Turn v2 is wav2vec2 with a linear head, 360 megabytes, 12 milliseconds of inference on an L40, fourteen languages. The next version got radically smaller: eight million parameters, eight megabytes as quantised ONNX, a Whisper Tiny encoder with a shallow linear classifier on top, the same 12 milliseconds now on a CPU, and a training set of 270,429 samples across 23 languages. Its input is the raw waveform, which is how it picks up the intonation of an unfinished phrase. LiveKit has its own audio model in two sizes, the smaller one running locally on a CPU, across fourteen languages; their text model, now being retired, reported true positive rates of 99.3 to 99.4 per cent at 50 to 160 milliseconds of latency, while true negative rates ranged from 85.1 per cent on Italian to 96.3 on Hindi. Those two figures are worth reading together: recognising the end of a turn is easy, and not mistaking an ordinary pause for the end is considerably harder, with quality depending on the language.
Which brings me to the reason for writing any of this. Take the dataset card for Smart Turn's training data: 270,429 recordings, 23 languages, durations from 0.36 to 32.6 seconds, four contributing organisations. On the age of the speakers it says nothing. The same holds across every other piece of documentation I read: languages are listed everywhere, age composition nowhere. Whether those corpora contain preschoolers cannot be checked from outside, though how such sets get collected is no secret, through crowdsourcing and synthesis, which means adult voices. And general-purpose models degrade on children's speech in a way that is predictable and large. So a product for preschoolers that takes any of these models off the shelf is relying on a tool whose quality on its own audience has been measured by nobody.
On what to do about it
We have no honest answer yet, only a set of options, and since this piece is read by people working on the same thing, here they are. The cheapest is to raise the thresholds: if a child answers more slowly, give the child more time. The price is a sluggish conversation, and a child responds to a second and a half of silence no better than to being interrupted. Another option is the grammar of the incomplete: eighteen per cent of children's pauses fall after a preposition or a conjunction, exactly the class of position a partial transcript makes visible, which works but demands fast streaming recognition, itself the weak link on children's speech. A turn model can be fine-tuned on children's data, since Smart Turn is open all the way down, training code and datasets included; that runs into the supply of preschool recordings labelled for end of turn, and no such corpus is public. Or you can change the signal: a child talking to a toy behaves differently from an adult talking to an assistant, they look at it, touch it, call it by name, and some of that can be caught without a microphone.
WhyWave has not launched, and this piece describes a place where the ready-made pieces do not fit together. It may be useful for one reason. Plenty has been written about recognising children's speech, plenty about detecting the end of a turn, and the two subjects almost never appear in the same place, even though for a voice product used by a four-year-old they are one problem. If you are building a voice agent for adults, one line of all this matters to you: the default in your framework was chosen by somebody else for somebody else's scenario. Go and look at what it is set to.
Top comments (0)