Human conversation does not wait 900ms after every pause.
Stivers et al. (PNAS, 2009) measured turn transitions across ten languages. The mode is 0–200ms. Levinson later pointed out the awkward implication: language production takes ~600ms to plan a word, so people are already encoding the reply while the other person is still talking. A voice agent that waits a fixed 0.9s after Deepgram says speech_final is not being careful. It is being obviously not-human.
That was us, six weeks ago. Fixed debounce after every pause. It felt polite in the lab and dead on a real phone. "What's your name?" sat in silence. "My number is…" got cut off. Same timer, opposite failure modes. Silence timers cannot tell "finished" from "thinking." LiveKit's open eot-bench puts numbers on this: a VAD threshold sits around ~1600ms of dead air to match the interruption rate a semantic detector hits at ~543ms.
We threw the timer out.
The new loop is one decision: did they hand the floor over?
Pipeline is still the boring one everyone has:
PSTN → SignalWire Stream → Deepgram (mulaw) → LLM → Cartesia → PSTN
The interesting part is what happens at the pause.
Deepgram speech_final is only a candidate end of turn. A cheap classifier then estimates P(caller finished), from the utterance plus the last few dialogue turns. Extra wait is a decreasing function of that probability:
- P ≥ 0.75 → commit now (crisp question / complete request)
- P ≤ 0.25 → wait up to 1.1s (trailing "and the…", "my number is")
- in between → linear ramp 1.1s → 0s
- classifier late or missing → 0.55s fallback
Obvious cases never hit the model. A trailing conjunction (and, but, because, the, to…) returns P=0.05 in a few microseconds. A question mark returns 0.95. One- or two-word acknowledgements ("okay", "yeah") lean incomplete, because on a phone they usually mean more is coming. Ambiguous text goes to a 4-token gpt-4o-mini call that is only allowed to emit a float.
The classifier runs in the background on every finalized STT segment, not at the endpoint. By the time Deepgram fires speech_final, a fresh P is usually already cached. Decision budget is 45ms of waiting for an in-flight score; if it misses, the HTTP call is shielded so the result warms the next pause instead of being thrown away. That is the difference between "classifier on the hot path" and "classifier adjacent to the hot path."
Speculative generation hides the remaining wait
While we decide, we already started LLM → streaming TTS into an asyncio queue. Committing a complete turn means flushing audio that already exists. The caller hears the first Cartesia chunk almost as they finish. If they resume mid-window we cancel the task, drop the queue, and reseed the turn detector with the earlier fragment so the next endpoint concatenates old + new words. The LLM never sees a half-thought as a finished turn.
That join matters more than it sounds. Without it you get:
Human: "My account number is"
(pause, we almost answer)
Human: "44821"
LLM hears only "44821"
With reseed it hears "My account number is 44821". Same audio. Different product.
Barge-in is a VAD event, not a transcript event
The playback loop used to notice interruptions when it got around to checking a flag. If TTS was starved, await audio_queue.get() parked and the agent talked over the human for another 200–400ms. That is the failure people describe as "it won't shut up."
SpeechStarted from Deepgram fires on voice energy, before any words. We now:
- set the barge-in flag immediately
- flush the provider audio buffer in the VAD handler, not in the playback loop (~50ms)
- poll the flag at least every 50ms even when the TTS queue is empty
- ignore barge-in for the first 250ms / 3 chunks of playback, because analogue line echo of the agent's own first syllable looks exactly like the caller starting to speak
Only words actually heard get committed. Partial agent replies are stored with [interrupted] so the next LLM turn knows what the human already heard. Context matching reality is a latency feature. A model that thinks it finished a sentence the caller never heard will double-speak.
Transcript writes, voice persistence, webhook fan-out for non-live events — all asyncio.create_task. The hot path is: receive mulaw, maybe cancel, maybe flush, maybe start generation.
"Agent-first" is not a landing-page adjective
The latency work is useless if the thing on the other end of the call is a dashboard a human has to click. The customer for AgentLine is an agent runtime — Hermes, OpenClaw, Claude Code, Codex, Claude.ai via MCP — that needs a real PSTN number.
That forced a different product surface.
Signup is email OTP over plain REST. MCP itself requires a Bearer key to connect, so you cannot put signup behind MCP. POST /v1/auth/otp then POST /v1/auth/verify creates the account and returns a key. Same email always maps to the same account, so the agent that signed up and the human who later logs in with that inbox are not two users. There is no "create account" form an agent can get stuck on.
Keys start with al_live_, not sk_live_. That is not branding. Stripe's secret-key pattern is in every agent runtime's redaction list. We shipped sk_live_ first. Hermes/OpenClaw/Warp ate the key and the agent only ever saw <REDACTED:STRIPE_SECRET>. The account existed. The tool output was empty. We renamed the prefix.
Live turns do not go through a human inbox. A one-file relay (agentline_relay.py install --agent-id agt_xxx) detects the local runtime, installs a user service (systemd / launchd / Task Scheduler), and holds a WebSocket. Each phone call maps to one persistent runtime session — Claude --resume, Codex thread id, Hermes X-Hermes-Session-Key, OpenClaw --session-key. Untrusted SMS and lifecycle events land in a local inbox the agent consumes later. Live utterances invoke the runtime now.
The webhook/WS payload carries a push_token. The agent POSTs the spoken answer back with X-Push-Token. No API key on the hot path, because the most common 401 we saw was "runtime has the key in env but the model refused to put it in the header." The text it pushes is spoken verbatim. If it writes a Slack message instead, the caller hears "let me check…" until the turn expires. That is why the event is loud about ACTION_REQUIRED: LIVE CALLER WAITING.
Hosted voice LLM handles social turns locally ("hey, how are you"). Factual / tool turns emit [CHECKING], play a stall phrase, and wait for the agent's push. If the webhook is known-dead we stop emitting call.utterance for the rest of the call and answer hosted. A dead agent must not hold the PSTN stream.
Inbound answer XML is the other sacred path. publish_event() inserts the mailbox row and schedules webhook delivery as a background task. We used to await the webhook before returning LaML. Slow or missing endpoints meant Stream never started, duration=0, failed / no-answer. A webhook that cannot delay the phone from ringing is an agent-first constraint, not a reliability nice-to-have.
Claude.ai talks to us over MCP + OAuth 2.1 (PKCE, dynamic client registration). Desktop uses mcp-remote + the Bearer key. Same tools either way: provision a number, start a call, push context, hang up.
What this actually sounds like
Caller: "Can you check if John emailed me back?"
Agent (hosted, 0 extra wait — it's a complete question): "Hmm, let me check that for you…"
[CHECKING]
Relay hits the local runtime, runtime reads mail, POSTs:
"Yeah — John replied twenty minutes ago, he can do Thursday at 2."
Cartesia speaks that sentence. Caller never sees a dashboard.
Caller: "My order number is…"
Classifier: P≈0.05, we wait. They continue "…44821."
Joined turn goes to the LLM once.
Caller talks over the agent two words in. Provider buffer clears on SpeechStarted. Transcript stores what was actually heard. Next turn does not repeat it.
Honest limits
The classifier is still a tiny chat completion, not a local audio turn-detector. We left the same decide_wait interface so we can swap in LiveKit's ONNX turn-detector (~50ms CPU), Pipecat SmartTurn, or a VAP model (Ekstedt & Skantze / MaAI) without touching the pipeline. Text-only P(finished) cannot hear a drawn-out "soooo…" that means "still thinking." Acoustic fusion is the next obvious cut.
We also do not claim sub-200ms end-to-end on PSTN. Carrier jitter, Deepgram endpointing (~300ms), Cartesia TTFA, and the PSTN itself are a floor. What we removed is the self-inflicted 900ms, the barge-in that waited on a playback loop, and the product shape that assumed a human would operate the phone.
If you have an agent that already does work, it can have a phone number. If you have been fighting a silence timer, steal the probability-to-wait function — it is forty lines and it is the whole trick.
Top comments (1)
Voice agents make latency feel like personality. If the silence timer changes, the turn-taking contract has to be tested with real conversations, not only average response time.