Voice Robot for Outbound Calls: A Latency Budget from ASR to TTS

A voice robot that confirms appointments can be perfectly accurate and still feel broken. The customer says "yes, I'm coming", then hears nothing for two seconds, says "hello?", and the robot starts answering on top of them. The transcript looks fine. The call does not.

On the phone, latency is the product. People tolerate a short pause after they finish speaking, and they start filling the silence themselves somewhere around one second. So the useful question is not "which speech model is best" but "where do the milliseconds go between the end of the customer's phrase and the first sound of our reply". This post breaks that path into stages, gives each stage a budget, and lists the traps that quietly eat it.

The metric that matters

Measure one number per turn: time from end of user speech to first audio byte played back. Call it turn latency. Not the ASR processing time, not the LLM time, not the TTS real-time factor. Those are components. The customer experiences only the sum, plus whatever the telephony network adds.

A workable target for short outbound scenarios:

Turn latency (p50)	How it feels
under 700 ms	natural, like a quick human operator
700-1200 ms	acceptable, slightly "robotic"
over 1500 ms	people talk over the robot or repeat themselves

Track p90 as well. A robot with a 600 ms median and a 2.5 s p90 will produce a talk-over in roughly one turn out of ten, and in a four-turn confirmation call that means a large share of calls have at least one awkward moment.

The pipeline, stage by stage

A typical outbound call path looks like this:

SIP trunk -> media server (RTP) -> VAD / endpointing -> streaming ASR
  -> dialog logic (rules or LLM) -> streaming TTS -> RTP back to caller

Here is a budget that fits under 800 ms at the median, with realistic ranges for each stage:

Stage	Budget	Typical range
Network + jitter buffer (both directions)	120 ms	80-250 ms
Endpointing (deciding the user stopped)	300 ms	200-800 ms
ASR final result after endpoint	80 ms	50-400 ms
Dialog decision	100 ms	5 ms (rules) to 1500 ms (LLM)
TTS time to first audio chunk	150 ms	80-600 ms
Playback start (buffering, codec)	40 ms	20-100 ms
Total	~790 ms

The two biggest and most variable items are endpointing and the dialog decision. That is where most of the engineering goes.

Endpointing: the silent budget killer

Before you can answer, you need to decide that the person has finished. The naive approach is a fixed silence timeout: if the VAD reports no speech for N milliseconds, the turn is over.

Set N to 1000 ms and every turn starts one full second in the hole before any model even runs. Set it to 200 ms and the robot will interrupt people who pause mid-sentence: "I'd like to... reschedule".

Practical settings that work for short confirmation scripts:

• Adaptive silence threshold by expected answer. After a yes/no question ("Will you come tomorrow at 15:00?") use 250-350 ms. After an open question ("Which day suits you?") use 600-800 ms.
• Early finalization on a confident short answer. If streaming ASR already has a partial "yes", "да", "иә" with high confidence and 200 ms of silence follows, end the turn. Short confirmations are exactly where this pays off.
• Minimum speech duration. Ignore bursts shorter than ~150 ms. Coughs, a door, a car horn and line clicks otherwise trigger a turn.

A simplified endpointing loop with a frame-level VAD:

FRAME_MS = 20

def endpoint(frames, vad, asr_partial, expect_short: bool):
    silence_limit = 300 if expect_short else 700
    speech_ms, silence_ms = 0, 0
    for frame in frames:                      # 20 ms PCM frames, 8 kHz
        if vad.is_speech(frame):
            speech_ms += FRAME_MS
            silence_ms = 0
        else:
            silence_ms += FRAME_MS
        if speech_ms < 150:
            continue                          # not real speech yet
        p = asr_partial()
        if expect_short and p.is_short_answer and p.confidence > 0.85 \
                and silence_ms >= 200:
            return "early"
        if silence_ms >= silence_limit:
            return "silence"

The values are starting points. Tune them on recorded calls, not on your own voice in a quiet office.

ASR: streaming or nothing

Batch recognition ("record the turn, then send the file") adds the full processing time after the endpoint. Streaming ASR processes audio as it arrives, so when the endpoint fires the hypothesis is already almost done and only finalization remains.

Things to check when choosing or self-hosting an engine:

• Telephony audio is 8 kHz narrowband, usually G.711. Many models are trained on 16 kHz. Upsampling does not restore lost frequencies; test accuracy on real call recordings, not on podcast clips.
• Language switching. In Kazakhstan a customer may answer a Russian script in Kazakh, or mix both in one sentence. Either run a model that handles both or detect the language on the first turn and keep the session in that language. Detection on every turn adds latency and flips on short words.
• Keyword boosting. Confirmation calls revolve around a few dozen words: dates, weekdays, "reschedule", clinic or branch names. Phrase hints or a constrained grammar for yes/no questions cut both errors and finalization time.
• GPU contention. A self-hosted model that answers in 60 ms with one call can take 400 ms at 40 concurrent calls if you did not batch or size the hardware. Load test at your real peak concurrency.

The dialog decision: rules first, LLM where needed

For a booking confirmation the set of meaningful answers is small: confirm, decline, reschedule, "who is this?", "call me later", and everything else. A classifier or a rule set over the transcript decides in single-digit milliseconds.

An LLM is useful for the long tail: "I'm on the road, can my wife come instead?" But a full generation call easily costs 500-1500 ms to first token, which blows the whole budget. Patterns that keep it under control:

1. Intent first, generation second. Run a fast intent classifier on every turn. Only route "other" to the LLM.
2. Pre-written replies for known intents. The response to "confirm" is fixed text, so its TTS audio can be cached in advance.
3. Hard timeout with a fallback. If the LLM has not produced the first sentence in 900 ms, play a neutral filler and offer a transfer or callback.

async def decide(transcript, state):
    intent = classify(transcript)            # ~5 ms
    if intent in SCRIPTED:
        return SCRIPTED[intent], intent
    try:
        text = await asyncio.wait_for(llm_reply(transcript, state), 0.9)
        return text, "llm"
    except asyncio.TimeoutError:
        return "Let me pass this to a colleague, they will call you back.", "escalate"

Do not hide a slow LLM behind fillers like "hmm, let me think" on every turn. One filler is human. Three in a minute sound like a stalling machine.

TTS: time to first chunk, not total time

A TTS engine that synthesizes a 3-second phrase in 300 ms looks fast, but if it returns audio only after the whole phrase is ready, you wait the full 300 ms. Streaming TTS that emits the first 100-200 ms of audio early lets playback start while the rest is still being generated.

Cheap wins:

• Cache everything static. Greetings, the confirmation reply, the goodbye, the "I didn't catch that" prompt. Synthesize once per voice and language, store as 8 kHz G.711 so no transcoding happens at call time.
• Template with slots. "Your appointment is on {day} at {time}" can be split into a cached prefix plus a short synthesized slot. The prefix starts playing immediately.
• Split long LLM replies by sentence. Send the first sentence to TTS as soon as it is complete instead of waiting for the full answer.
• Match the output format. Generating 24 kHz audio and resampling to 8 kHz on every chunk costs CPU and a few milliseconds per chunk. Ask the engine for telephony format directly if it supports it.

Barge-in: when the customer talks over the robot

Even with a good budget, people interrupt. "Your appointment at the clinic tomorrow at..." "Yes, yes, I know." The robot must stop speaking within roughly 200-300 ms, or it sounds like a recorded message.

That requires:

• Running VAD and ASR on the inbound stream while TTS is playing.
• Echo handling. On some lines the robot hears its own voice back. Without echo cancellation or a raised VAD threshold during playback, it interrupts itself.
• Flushing the outbound buffer. If you queued 3 seconds of audio into the media server, "stop" must clear that queue, not just stop sending new chunks.
• Deciding whether the interruption is meaningful. A short "mm-hm" during playback is acknowledgement, not a turn.

Network and telephony: the part you don't control

The SIP trunk and mobile network add their own delay, often 100-250 ms round trip, plus the jitter buffer on your media server. Two settings matter:

• Jitter buffer size. A fixed 200 ms buffer adds 200 ms to every turn. An adaptive buffer starting at 40-60 ms is usually better for voice bots.
• Media server location. Host the media server and the speech models close to the trunk provider. Sending RTP across continents to a cloud ASR region adds a round trip every turn.

Measuring it in production

Log timestamps per turn, in one clock domain, on the media server:

{
  "call_id": "c-20261109-0412",
  "turn": 2,
  "user_speech_end": 18.420,
  "endpoint_fired": 18.720,
  "asr_final": 18.790,
  "decision_ready": 18.796,
  "tts_first_chunk": 18.930,
  "playback_start": 18.965,
  "intent": "confirm",
  "barge_in": false
}

user_speech_end comes from the VAD, so it is an estimate, but it is consistent, which is what you need for trends. From this you get the per-stage breakdown, p50 and p90 per stage, and a direct view of which stage regressed after a deploy or a model update.

Also record the business outcome per call: confirmed, declined, rescheduled, no answer, escalated. A latency improvement that does not move the confirmation rate or reduce talk-overs probably was not the bottleneck.

Pitfalls that show up only in real calls

• Answering machines and voicemail. A greeting recording will happily "answer" your question. Detect long uninterrupted speech in the first seconds and end the call or leave a short message.
• Background noise. Markets, cars, children. Raise VAD aggressiveness and rely more on confident ASR partials than on pure energy.
• The first turn. People often say "hello" twice before listening. Start the script after the first short utterance, not after the pickup event.
• Time zones and calling hours. If the customer base spans several countries or time zones, a reminder sent at 9:00 server time can land at 7:00 for some customers. Store the customer's time zone and schedule by their local time, with a calling window per scenario.
• Consent. Only call people who left their number and expect the call. Beyond legal risk, cold lists destroy answer rates and get trunk numbers flagged as spam.

A checklist

• Turn latency measured as end of speech to first audio, p50 and p90
• Adaptive endpointing by expected answer type, early finalization on short confident answers
• Streaming ASR tested on 8 kHz call recordings in every language the script uses
• Intent classifier first, LLM only for the long tail, with a hard timeout
• Cached static prompts in telephony format, streaming TTS for the rest
• Barge-in with buffer flush and echo handling
• Per-turn timing logs tied to the call outcome

Short, predictable scenarios like booking confirmations, visit reminders and one-question surveys are where these budgets are easiest to hit, because most replies can be scripted and cached. The usual path is to start with one scenario, listen to the test call recordings, tune pauses and wording, and only then scale the volume. If you need this for a clinic, salon or delivery service with scripts in Russian and Kazakh, here is how we set up an AI voice robot that handles confirmation and reminder calls.