Ten days before launch, our voice agent kept cutting users off: an end-of-turn detection war story

TL;DR. Our phone voice agent kept interrupting people. We had shipped end-of-turn detection as a single silence timeout: if the caller went quiet for 700 milliseconds, the agent decided they were finished and started talking. It cut people off mid-sentence. When I raised the timeout to stop the interruptions, the agent started hanging in dead silence instead. The reframe that fixed it was to stop deciding on a fixed silence timeout alone and start weighing three signals together. You need three signals at once. How long the silence has lasted, whether the transcript looks grammatically finished, and whether the last sound was a real turn or just a backchannel like "uh-huh." Here is the story, the transcripts that showed me the bug, and the endpointing loop we shipped.

Day 0: the demo that worked

The first demo of our voice agent was clean. I called the number, asked to check an order, and the thing answered me like a person. My co-founder called it from the parking lot and it handled his accent. We recorded a 40-second clip, put it in the investor update, and I went home thinking the hard part was behind us.

The hard part was not behind us. The hard part is that a demo is one careful person speaking in complete sentences in a quiet room. Real callers pause in the middle of a thought. They say "so, the thing is" and then go quiet for a second while they remember their order number. They read a card number out loud in groups with gaps between them. They say "uh-huh" while you are still talking, not because they want to interrupt, but because that is how humans signal they are still listening.

Our agent treated every one of those pauses as the end of a turn.

Day 3: the setup, and the one number that ran everything

Here is what we had. Audio came in over the phone network, through a WebRTC bridge, into a streaming speech-to-text service that emitted partial transcripts every couple of hundred milliseconds. On top of the audio we ran Silero VAD, an open-source voice activity detector (github.com/snakers4/silero-vad), which gives you a speech probability per short audio frame. When the speech probability dropped below a threshold and stayed there long enough, we called it the end of the user's turn, sent the final transcript to the language model, and started speaking the reply.

"Long enough" was one constant in a config file. Seven hundred milliseconds. I had picked it the way everyone picks it the first time, which is to say I made it up. It felt about right in the demo. That single number decided, on every single turn of every single call, whether we waited for the caller or talked over them. I did not appreciate that at the time.

Endpointing is the unglamorous name for this problem: deciding the exact moment a person has finished speaking and it is your turn to respond. Get it wrong short and you interrupt. Get it wrong long and you feel slow, or worse, you never respond at all. There is no value of a fixed timeout that is right, and it took me an embarrassing while to understand why.

Week 1: "it keeps interrupting me"

We put the agent in front of a small beta group, maybe 30 people, mostly friendly. The feedback came back fast and it rhymed. "It talks over me." "It cut me off." "I had to say my order number three times." One tester, who was very patient, said the agent felt like a person who was just waiting for their turn to speak instead of listening.

That last one stuck with me, because it was literally true. The agent was waiting for a gap, any gap, and pouncing on it.

I did the thing you do. I lowered nothing and raised nothing yet. I went and got the data. We had call recordings and aligned transcripts in staging, so I pulled 312 calls and started reading turn boundaries. Not listening to full calls, that would have taken a week. Reading the transcript around every point where the agent decided to speak, and tagging whether the caller had actually finished.

The transcripts that showed me the bug

The pattern was ugly and consistent. Here is a real one, lightly anonymized, with timestamps in seconds from the start of the caller's turn:

0.00  user (partial): "yeah i want to return the"
0.61  <silence 610 ms>
0.70  ENDPOINT FIRED
0.70  agent: "Sure, I can help you start a return. Which order..."
0.95  user (partial): "...the blue one not the black one"

The caller took a breath after "the." Six hundred and ten milliseconds later, our threshold tripped, the agent barged in, and the caller's actual object ("the blue one") landed on top of the agent's reply and got lost. The speech-to-text kept transcribing "the blue one not the black one" into the void while the agent was already talking about something else.

I counted these. Across 312 calls, about 18 percent of user turns showed a truncation like this, where the final transcript was cut and the caller either repeated themselves or the agent answered the wrong half of the sentence. Eighteen percent. Almost one in five turns was damaged by a single config constant.

And it was not random. It clustered. Callers who paused mid-sentence to think got hit constantly. People reading numbers out loud got hit on every gap between digit groups. One tester who spoke English as a second language and paused a beat longer between clauses got interrupted on nearly every turn, which is its own kind of unacceptable, because your latency policy should not punish people for how they talk.

The overcorrection I shipped (and rolled back the next morning)

This is the part I am not proud of.

The fix looked obvious. The timeout was too short, so make it longer. I pushed the silence threshold from 700 milliseconds to 1500 on a Thursday afternoon, watched a few test calls go smoothly, and shipped it to the beta. Truncations dropped immediately. I told the team we had fixed the interrupting bug. I was wrong in two directions at once.

First, the agent now felt dead. Every reply came a beat and a half after you stopped talking, which does not sound like much until you are on the phone with it. Conversation has rhythm, and a flat 1.5-second gap after every single turn reads as "this thing is slow" or "did it hear me." Median time-to-first-response on the agent's side went to roughly 1.8 seconds once you added the model and the speech synthesis on top of the wait. Testers stopped trusting that it had heard them, so they started repeating themselves into the gap, which created overlapping speech, which confused the transcript. I had traded interruptions for a different failure.

Second, and this is the one that actually scared me, the agent started hanging. Silently. On some calls it would just never respond. I could hear the caller finish, wait, say "hello?", wait, and hang up. Dead air on a phone call is worse than an interruption, because an interruption at least tells the user the thing is alive.

I rolled it back Friday morning and went to find out why raising a timeout could make an agent stop responding entirely.

The silent hang, explained

The hang was the more interesting bug, so let me stay on it.

A fixed silence timeout only fires if you actually accumulate that much continuous silence. On a clean headset in a quiet room, you do. On a phone line, you do not always. Phone audio carries background noise, and Silero VAD, like any voice activity detector, will occasionally flicker its speech probability above the threshold on a cough, a door, a bit of line static, a TV in the next room. Each of those flickers reset my silence counter back to zero.

With a 700-millisecond budget, an occasional flicker did not matter much. You would still gather 700 milliseconds of quiet soon enough. With a 1500-millisecond budget, the window was more than twice as long, and on noisy lines the counter kept getting reset before it ever reached 1500. The turn never ended. The agent waited forever for a silence that noise kept interrupting. My "safer" longer timeout had made the endpoint condition genuinely unreachable on exactly the calls that were already the hardest.

The lesson landed hard. A single silence timeout was not something I could tune my way out of. Whatever value I picked, it was one number trying to answer two different questions, when to wait for a thinking caller and when to jump on a finished sentence, and one number cannot answer both. I needed the endpoint decision to depend on more than the clock.

The 3am realization: what the transcripts were telling me

I will spare you the exact hour, but the idea that fixed it came from re-reading my own truncation transcripts and noticing something I had been looking straight past.

Every bad early endpoint had a tell in the text. "I want to return the." "My order number is." "Can you check on." "It's the blue one and." These are not sentences a person stops on. They end on a preposition, an article, a conjunction, a dangling word that grammatically demands more. A human listener knows, without thinking about it, that "I want to return the" is not a complete turn no matter how long the pause is. The silence after "the" means "I am thinking," not "I am done."

And the reverse was true for the hangs and the laggy turns. "My order number is 4021." "I want to return the blue one." Those are complete. A human would jump in fast after them, and so should the agent. Waiting 1500 milliseconds after a clearly finished sentence only makes the agent feel slow.

So the endpoint should combine the silence with what was actually said:

• If the transcript looks grammatically finished, endpoint fast. A short pause is enough, because the sentence is done.
• If the transcript looks open, that is, it ends on a dangling word, wait much longer, because the caller is mid-thought.
• If the last thing you heard was a backchannel like "yeah" or "uh-huh," do not endpoint at all, and do not let it interrupt the agent either. It is not a turn.
• And always keep a hard maximum so a noisy line can never hang forever.

This is the same insight the open-source turn-detection work has been converging on. LiveKit ships a turn-detector plugin that uses a small learned model over the transcript to predict whether the user is actually done, and Pipecat has an open "Smart Turn" model that does the same job from the audio. I read both while I was digging out of this. You do not always need a learned model to get most of the benefit, though. A surprising amount of the win is just refusing to endpoint on a dangling word.

The fix, in code

Here is the shape of what we shipped, stripped down to the endpointing loop. It runs one VAD frame at a time, tracks silence, and, crucially, picks its silence budget based on whether the current partial transcript looks finished. It guards backchannels, and it enforces a hard cap so a noisy line can never hang.

import torch

# Silero VAD: github.com/snakers4/silero-vad
model, _ = torch.hub.load("snakers4/silero-vad", "silero_vad", trust_repo=True)

SAMPLE_RATE = 16_000
FRAME_MS = 32                       # 512-sample windows at 16 kHz
SPEECH_PROB = 0.5

# two silence budgets instead of one: this was the whole fix
SILENCE_DONE = 550                  # transcript looks finished, endpoint fast
SILENCE_OPEN = 1300                 # trailing "to", "and", "um", wait longer
HARD_CAP_MS = 8000                  # never hang past this, even on a noisy line

BACKCHANNELS = {"uh huh", "mm hmm", "yeah", "right", "okay", "sure"}
DANGLING = {"to", "and", "or", "but", "the", "a", "for", "with", "um", "uh", "so"}

def is_open(text: str) -> bool:
    words = text.strip().lower().split()
    return not words or words[-1] in DANGLING

def endpoint(frames, partial_transcript):
    silence_ms = elapsed_ms = 0
    heard_speech = False
    for frame in frames:                        # 512-sample float32 tensors
        elapsed_ms += FRAME_MS
        if model(frame, SAMPLE_RATE).item() >= SPEECH_PROB:
            heard_speech, silence_ms = True, 0
            continue
        if not heard_speech:
            continue                            # ignore leading silence
        silence_ms += FRAME_MS
        text = partial_transcript()             # latest partial from your ASR
        if text.strip().lower() in BACKCHANNELS:
            return None                         # backchannel, keep the agent going
        budget = SILENCE_OPEN if is_open(text) else SILENCE_DONE
        if silence_ms >= budget or elapsed_ms >= HARD_CAP_MS:
            return text                         # end of turn
    return None

A few things are load-bearing in there and are not obvious.

The two budgets are the point. Five hundred and fifty milliseconds when the sentence is finished, thirteen hundred when it is open. The finished case feels snappy because it is snappy. The open case gives the thinking caller room. The gap between the two numbers is doing the work that no single number could.

The DANGLING set is a crude heuristic, and I want to be honest that it is crude. It is a word list. It does not understand grammar. But it catches the overwhelming majority of the truncations I had tagged, because English sentences really do tend to stall on the same couple dozen function words. If you want to do better, this is exactly the seam where you swap in a learned turn model like LiveKit's or Pipecat's. The heuristic is the 80 percent version that you can ship this afternoon.

The HARD_CAP_MS is the scar tissue from the silent hang. It guarantees the turn always ends, noise or no noise. It is not elegant, but it guarantees the turn always ends, and I will not ship an endpointer without it again.

Backchannels and barge-in: the other half of the bug

Cutting people off was only one of the two failures. The mirror image is barge-in, which is when the caller starts talking while the agent is still speaking and you want to stop the agent and listen. You need barge-in, because callers will interrupt, and an agent that plows through your interruption is infuriating.

But here is the trap. If you treat any speech during the agent's turn as a barge-in, then every "uh-huh" and "yeah" and "mm-hmm" stops the agent dead. Those are backchannels, not interruptions. The caller is not trying to take the floor, they are just signaling that they are still there. In my logs, before we handled this, the agent stopped itself on a backchannel roughly one out of every six times it spoke a longer reply. It would start explaining the return policy, the caller would say "mm-hmm" to be polite, and the agent would stop, assume it had been interrupted, and ask "sorry, go ahead." The caller had nothing to go ahead with. It was maddening on both ends.

Two guards fixed most of it. First, require a minimum duration of continuous speech before you treat it as a real barge-in. We used about 240 milliseconds. A quick "yeah" usually does not clear that bar; an actual interruption does, because a person taking the floor keeps talking. Second, check the partial transcript against the same backchannel list before you stop the agent. If the only thing the speech detector caught was "uh huh," keep talking. Between the duration gate and the word check, false barge-ins on backchannels went from that one-in-six rate to something I stopped being able to find in the logs.

When this does not apply

I want to be careful not to sell this as a universal fix, because it is not, and a few of these thresholds are specific to the mess we were in.

If you have a push-to-talk interface or any explicit signal for when the user is done, you do not need most of this. A button that says "I am finished" beats every heuristic. Endpointing is hard precisely because we are inferring the turn boundary from audio instead of being told.

If your users are on clean headsets in quiet rooms, a plain fixed timeout will carry you a long way, and the silent-hang failure mode mostly will not happen, because you will actually accumulate the silence you are waiting for. The noise-resets-the-counter problem is a telephony problem. It got much worse for us specifically because we were on the phone network.

If your latency budget is brutal, sub-300-millisecond end to end, you may not be able to afford a learned turn model in the hot path, and even the transcript check costs you the time it takes your speech-to-text to emit a stable partial. In that case the word-list heuristic is your friend precisely because it is nearly free. It runs on a string you already have.

And the biggest caveat: the DANGLING list is English. It leans on the fact that English stalls on prepositions and articles and conjunctions. That intuition does not transfer cleanly to other languages, some of which put the load-bearing word at the end of the clause. If you are multilingual, you either build a per-language list or you go straight to a multilingual turn-detection model, and you test it on real speakers of each language, not on your own careful demo voice. I learned the demo-voice lesson once already. I do not need to learn it again per language.

What shipped, and what I would tell past me

What shipped, in the end, was not clever. It was a state machine with two silence budgets chosen by a dumb little transcript check, a backchannel guard, a minimum-duration gate on barge-in, and a hard cap so the thing can never hang. That is it. Truncated turns went from about 18 percent to about 3 percent in the same staging set. Median time-to-first-response after a clearly finished sentence came back down to roughly 850 milliseconds, snappy again next to the 1.8 seconds the overcorrection had caused, while the mid-thought pausers finally got the room they needed. The silent hangs disappeared, because the hard cap made them impossible by construction.

Here is what I would tell the version of me who typed SILENCE_MS = 700 into a config file and moved on.

Endpointing deserves the same design attention as anything the user can see on a screen, because it is one of the things they feel most on a call. Build it as a state machine with real logic instead of leaving it as one constant in a config file.

Log every turn boundary with the audio and the partial transcript at the moment you decided. I could not diagnose any of this until I could sit and read the exact text that was on the screen when the endpoint fired. If I had built that logging on day one instead of week three, I would have found the truncation pattern in an afternoon.

Measure truncation rate as a first-class metric, right next to latency. If I had been watching "what fraction of turns got cut off" from the start, the 18 percent would have been a screaming red number on a dashboard instead of a slow trickle of "it interrupts me" complaints.

Treat the fixed-timeout VAD endpoint as a temporary placeholder. It is the thing you ship in week one to get a demo working, and it is the thing you must plan to replace. The open-source turn detectors from LiveKit and Pipecat exist because a lot of teams walked into this same wall. I just walked into it in production, ten days before a launch, with real callers as my test set.

The demo lied to me because the demo was one calm person in a quiet room. Real conversation is pauses and "uh-huh" and someone reading a card number with gaps between the digits. Once I stopped tuning a single number and built the agent to account for those pauses and backchannels, the interruptions and the dead air both went away.