Sub-50 ms Is Not a Voice-Agent SLO: Build a Turn Budget and Overload Envelope

A voice-AI demo can report a 42 millisecond model response and still make a caller wait three seconds. It can start speaking quickly and still interrupt a hesitant customer. It can pass a laboratory latency test and fail during the first Monday-morning traffic spike. The number is not necessarily false; it is incomplete.

Production voice systems are pipelines. Audio capture, endpoint detection, transport, speech recognition, dialogue state, retrieval, tools, model inference, safety policy, speech synthesis, jitter buffers, and telephony all consume time. Some work overlaps, some runs sequentially, and some happens only on difficult turns. A component benchmark therefore cannot serve as the service promise.

The practical alternative is a voice-turn budget with an overload envelope. The budget defines exactly where a turn begins and ends, allocates latency to observable stages, and reports percentiles by turn class. The overload envelope defines what the system is allowed to degrade, defer, or hand off when demand or dependencies exceed safe capacity.

This guide adapts the infrastructure questions in a Swiss voice-AI architecture analysis. It adds measurement semantics, a trace schema, an executable evaluator, failure modes, and a release gate. The objective is not the smallest marketing number. It is a conversation that remains understandable, interruptible, accurate, and recoverable under real load.

The decision in one sentence

Do not approve a voice agent because one component is fast. Approve it only when representative calls meet end-to-end turn, interruption, task-success, and safety objectives at normal load and at a declared overload point.

That rule changes the buying conversation. Instead of asking whether the model is “real time,” ask:

• What event starts the clock?
• What event stops it?
• Which turns are measured?
• Which percentiles are reported?
• How are interruptions and false endpoints counted?
• What happens when a tool, carrier, or model is slow?
• At what concurrency does the promise stop applying?
• What evidence proves that degradation is controlled?

If those answers are missing, a sub-50 ms claim is a laboratory observation, not an operational commitment.

Define the voice turn before measuring it

Teams frequently compare incompatible numbers. One measures network round-trip time. Another measures time to the first language-model token. A third measures time from a final ASR transcript to the first synthesized audio byte. The caller experiences the entire silence after finishing a turn.

Use explicit events:

Event	Definition	Why it matters
speech_start	First detected user speech frame	Anchors barge-in and listening measurements
speech_end_candidate	First silence that may end the turn	Shows endpoint detector behavior
speech_end_committed	System decides the user finished	Separates deliberate endpointing from compute time
asr_final	Final transcript accepted	Exposes recognition finalization delay
plan_ready	Dialogue policy chooses answer/tool path	Separates orchestration from generation
tool_start / tool_end	External operation begins/ends	Makes dependency delay visible
first_text_token	First generated response token	Useful component diagnostic, not caller SLO
first_audio_ready	First playable synthesized audio exists	Captures synthesis and buffering
first_audio_played	Caller endpoint begins playback	Best end event for perceived response gap
barge_in_detected	User interrupts while agent speaks	Measures listening while speaking
playback_stopped	Agent audio actually stops	Measures interruption responsiveness

The primary response metric should normally be:

turn_gap = first_audio_played - speech_end_committed

Also retain a second, user-centered metric from the last meaningful user audio frame. It includes endpointing hesitation and reveals whether an aggressively tuned detector improves a dashboard by cutting people off.

Latency is a vector, not a scalar

A single average hides the failures users remember. Report at least median, p90, p95, and p99 for each important class. Keep counts beside percentiles because a p99 from 40 turns is nearly meaningless.

Separate turn classes:

• greeting and fixed prompt;
• simple FAQ from cached knowledge;
• retrieval-backed answer;
• one-tool transaction;
• multi-tool transaction;
• clarification;
• error recovery;
• human transfer;
• interrupted turn;
• long or multilingual utterance;
• safety-sensitive refusal or escalation.

A fixed greeting can begin in 100 ms while a booking tool takes two seconds. Combining both into one p50 rewards traffic mix rather than engineering quality. Each class needs its own sample size, budget, and failure policy.

Build the budget from the caller backward

Start with an acceptable conversation outcome, not the GPU specification. Allocate the total gap across stages while leaving a contingency reserve.

Stage	Illustrative p95 budget	Control question
Endpoint commitment	350 ms	Does it preserve natural hesitation?
Final ASR and normalization	180 ms	Are accents and noisy channels represented?
Policy and routing	80 ms	Are safety and tenant rules cached safely?
Retrieval or tool selection	140 ms	Can optional context be skipped under load?
First model output	420 ms	Is this warm and at target concurrency?
First speech audio	260 ms	Does the voice preserve pronunciation?
Transport and playout	170 ms	Are carrier and jitter effects included?
Contingency	200 ms	Is the reserve consumed and reported?
Total	1,800 ms	End-to-end p95 objective

These values are examples, not universal thresholds. Task complexity, language, telephony route, user expectations, and error cost change the acceptable allocation. The discipline is that every millisecond in the total has an owner and a measurement point.

Do not force all turns under one number. A compliance disclosure may deliberately take longer. A transaction confirmation may wait for a durable write. A human transfer can acknowledge quickly, then complete asynchronously. The caller should receive an honest progress cue rather than fabricated certainty.

The trade-off that benchmarks hide

Endpoint detection illustrates why “faster” can become worse. Commit too early and the agent interrupts a pause, splits one request into two turns, or launches a tool with incomplete parameters. Commit too late and every answer feels sluggish.

Research on natural conversational turn prediction explicitly models hesitations, filled pauses, repetitions, and multiple queries. One published system reported more than 97% recall and 85% precision for true turn-taking at 100 ms latency on its constructed test set. Those figures are useful evidence of possibility, not a production guarantee: the test data, languages, acoustics, and error costs may differ.

The contradiction is central. Lower endpoint delay reduces silence but can increase false-endpoint rate. More context improves some answers but increases compute and retrieval time. Speculative tool execution can hide latency but may create unnecessary calls or side effects. Richer safety checks can prevent harm but consume time. The correct design does not maximize one metric; it constrains several outcomes simultaneously.

Add an overload envelope

An SLO without a load boundary is incomplete. State the operating region in which it applies:

• concurrent calls per region;
• turns per second;
• audio codec and sample rate;
• supported languages;
• typical and maximum utterance duration;
• tool mix;
• model and hardware version;
• warm-capacity assumption;
• dependency latency distribution;
• measurement window and minimum sample count.

Then define overload states.

Green: normal operation

All turn classes receive the normal model, retrieval depth, voice, and tool policy. Admission is open. Reserves cover ordinary variance.

Amber: constrained operation

The system preserves safety, interruption, and transactional correctness. It may shorten nonessential responses, use cached context, reduce optional retrieval, route simple intents to a smaller validated model, or delay noncritical writes. It must not silently remove consent checks, authentication, confirmation, or audit logging.

Red: protective operation

New calls receive a transparent busy message, queue estimate, callback option, or human route. In-progress transactions finish safely or roll back. The system rejects work before it creates a conversation it cannot complete.

Black: dependency integrity failure

If identity, payment, clinical, emergency, or compliance dependencies are unreliable, the voice agent stops the affected workflow. A fluent guess is not graceful degradation.

The envelope makes trade-offs reviewable. “We degraded gracefully” becomes a set of declared behaviors with test evidence.

Architecture and trace flow

flowchart LR
    A[Caller audio] --> B[Edge capture and VAD]
    B --> C[Streaming ASR]
    C --> D[Turn manager]
    D --> E{Intent class}
    E -->|Cached| F[Policy response]
    E -->|Knowledge| G[Retrieval]
    E -->|Transaction| H[Tool gateway]
    F --> I[Response model]
    G --> I
    H --> I
    I --> J[Streaming TTS]
    J --> K[Jitter buffer and carrier]
    K --> L[Caller playback]
    B -. barge-in .-> M[Playback cancel]
    M --> D
    B --> T[Turn trace]
    C --> T
    D --> T
    G --> T
    H --> T
    I --> T
    J --> T
    K --> T

Every stage emits timestamps under one trace identifier. Do not put raw audio, full transcripts, access tokens, or unnecessary personal data into latency telemetry. Use pseudonymous call and tenant identifiers, structured outcome codes, and retention appropriate to the risk.

A minimal trace schema

One record per turn should include:

• trace ID and turn sequence;
• environment, region, model, voice, and deployment version;
• call transport and codec;
• language and turn class;
• every event timestamp;
• ASR confidence bucket, not necessarily raw text;
• tool name category and result code;
• cache status;
• interruption detected and playback-cancel delay;
• endpoint revision count;
• answer outcome: completed, clarified, transferred, refused, abandoned, or failed;
• overload state;
• sampled quality label;
• consent and disclosure policy version where applicable.

Schema versioning matters. If the definition of speech_end_committed changes, old and new measurements cannot be mixed without a migration note.

Google’s Site Reliability Engineering guidance treats distributed scheduling as a reliability problem involving failure, duplicate execution, and coordination rather than a simple timer. The same reasoning applies to voice tools: retries and timeouts need idempotency, deadlines, and ownership (Google SRE on distributed periodic scheduling). This reference is not a voice benchmark; it supports the reliability principle behind controlled external work.

Executable evaluator

The following Python program reads newline-delimited JSON turn traces and calculates turn-gap and interruption percentiles. It fails the release if sample size is insufficient or if declared objectives are missed.

import json
import math
from collections import defaultdict
from pathlib import Path

def percentile(values, q):
    if not values:
        return None
    ordered = sorted(values)
    position = (len(ordered) - 1) * q
    low = math.floor(position)
    high = math.ceil(position)
    if low == high:
        return ordered[low]
    return ordered[low] + (ordered[high] - ordered[low]) * (position - low)

groups = defaultdict(lambda: {"gap": [], "barge": [], "success": 0, "total": 0})

for line in Path("turns.ndjson").read_text(encoding="utf-8").splitlines():
    if not line.strip():
        continue
    turn = json.loads(line)
    key = (turn["turn_class"], turn["overload_state"])
    group = groups[key]
    group["total"] += 1
    if turn["outcome"] in {"completed", "clarified", "transferred"}:
        group["success"] += 1
    if "speech_end_committed_ms" in turn and "first_audio_played_ms" in turn:
        group["gap"].append(
            turn["first_audio_played_ms"] - turn["speech_end_committed_ms"]
        )
    if "barge_in_detected_ms" in turn and "playback_stopped_ms" in turn:
        group["barge"].append(
            turn["playback_stopped_ms"] - turn["barge_in_detected_ms"]
        )

failed = False
for key, group in sorted(groups.items()):
    count = group["total"]
    success_rate = group["success"] / count if count else 0
    p95_gap = percentile(group["gap"], 0.95)
    p99_gap = percentile(group["gap"], 0.99)
    p95_barge = percentile(group["barge"], 0.95)
    print(key, {"n": count, "success": success_rate,
                "gap_p95_ms": p95_gap, "gap_p99_ms": p99_gap,
                "barge_p95_ms": p95_barge})
    if count < 200 or success_rate < 0.97:
        failed = True
    if p95_gap is None or p95_gap > 1800:
        failed = True
    if group["barge"] and p95_barge > 250:
        failed = True

raise SystemExit(1 if failed else 0)

The thresholds are illustrative. A medical intake, debt collection line, restaurant booking service, and internal help desk should not inherit the same target. What matters is that thresholds are written before the test, attached to turn classes, and evaluated with sufficient samples.

Test the whole system

A credible release test combines controlled and live-like conditions.

Network variation

Include stable broadband, mobile jitter, packet loss, delayed packets, codec changes, and geographically distant carrier routes. Measure playback at the caller-side test endpoint, not only inside the data center.

Speech variation

Use accents, dialects, code-switching, quiet speech, loud speech, background noise, overlapping speakers, hesitations, corrections, long numbers, names, addresses, and spelling. Synthetic audio helps scale regression tests but does not replace consented human evaluation.

Dialogue variation

Test silence, barge-in, repeated interruption, topic changes, ambiguous intent, denial, frustration, and requests outside scope. Fast answers are irrelevant if the agent cannot repair misunderstanding.

Dependency variation

Inject slow retrieval, expired credentials, tool timeouts, partial responses, duplicate callbacks, stale caches, unavailable human queues, and rate limits. Verify deadlines propagate rather than allowing hidden work to continue after the caller has moved on.

Load variation

Ramp gradually beyond the declared concurrency limit. Hold at plateaus long enough to expose queue growth and thermal or memory effects. A short burst can miss backlog accumulation. Report both offered load and admitted load.

Safety and compliance variation

Test disclosure, consent, authentication, opt-out, restricted topics, recording policy, and transfer paths. In the United States, the FCC confirmed in 2024 that AI-generated voices fall within TCPA restrictions on artificial or prerecorded voice calls; applicable calls require the relevant consent and identification controls (FCC declaratory ruling summary). This is jurisdiction-specific and not a global compliance shortcut.

Failure modes

The clock starts after endpointing

The dashboard excludes the detector’s silence window, making the system appear faster when it may actually wait or interrupt.

Control: publish both committed-end gap and last-audio gap.

Only successful turns are measured

Timeouts, abandoned calls, and failed tools disappear from percentiles.

Control: every admitted turn produces a terminal outcome; missing end events count as failures.

The average passes while the tail fails

Most FAQ turns are fast, but transactional p99 grows without bound.

Control: stratify by class and enforce tail objectives with minimum counts.

Warm demos stand in for capacity tests

One call runs on an idle GPU with cached prompts.

Control: declare concurrency, cache state, model version, and offered load beside results.

Faster endpointing creates more interruptions

Latency improves while completion and satisfaction fall.

Control: gate on false endpoints, repair turns, barge-in success, and task outcomes together.

Fillers hide dangerous tool delay

The agent says “one moment” while a payment or booking request times out and retries.

Control: fillers may explain progress but never replace a tool deadline, idempotency key, or confirmation.

Degradation removes safeguards

Under load, the system disables authentication, disclosure, or validation to save time.

Control: label safeguards non-sheddable; reject or transfer work instead.

Traces become a privacy liability

Raw transcripts and audio are retained indefinitely for performance analysis.

Control: minimize fields, separate quality samples, restrict access, set retention, and verify deletion.

A consulting forecast becomes evidence

Industry commentary is cited as proof that a deployment will improve outcomes.

Control: use analyst material to frame operating questions, then validate with the organization’s controlled measurements. A Gartner research document can inform a management discussion, but it is not proof that a specific voice pipeline meets its capacity, quality, or business objectives.

Capacity is an admission-control decision

Latency usually deteriorates before a service becomes visibly unavailable. Queues lengthen, memory pressure increases, caches churn, connection pools saturate, and external APIs begin to throttle. If the service continues accepting every call, the failure spreads across all callers. An overload envelope therefore needs an admission controller, not only autoscaling.

Define capacity in work units. Concurrent calls are useful, but two calls can impose very different loads. A caller listening to a long response consumes streaming and synthesis capacity; a caller triggering retrieval and three tools consumes orchestration capacity; a silent caller may hold telephony resources without using the model. Track at least active audio streams, model tokens per second, synthesis characters per second, tool operations, and outstanding turns.

Set a concurrency limit from a sustained test, then reserve headroom for failover and variance. If a region can sustain 300 representative calls while meeting objectives, advertising 300 as normal capacity leaves no room for a node loss or traffic imbalance. The production limit should include a documented reserve and a reason for that reserve.

Admission policy should be deterministic enough to audit. For example:

1. admit authenticated safety-critical callbacks first;
2. preserve callers already inside a transactional workflow;
3. admit new routine calls while the predicted queue remains inside the amber objective;
4. offer callback or human alternatives before the red boundary;
5. reject new automated transactions when durable dependencies are unhealthy.

Do not prioritize callers using protected or sensitive attributes unless a lawful, reviewed service obligation requires it. Prefer service class, workflow state, and explicit contractual priority.

Autoscaling still matters, but it is slower than a queue can grow. New accelerators may need model loading, health checks, cache warming, route propagation, and telephony registration. Record time to usable capacity, not merely instance startup time. Test scale-down too: removing workers with active audio can truncate speech or lose dialogue state.

Use deadlines, not independent timeouts

Independent timeouts are a common source of tail latency. The gateway allows five seconds, retrieval allows four seconds, and a tool allows six seconds. When executed sequentially, the caller waits far longer than the intended turn budget.

Create one turn deadline and propagate the remaining budget. At each stage, calculate:

remaining = turn_deadline - current_monotonic_time

The stage may proceed only if it can finish or fail safely within the remaining time. Reserve enough time for the agent to explain failure, offer an alternative, and stop playback if interrupted. A tool should not receive the full original deadline after retrieval has already consumed most of it.

Cancellation must propagate with the deadline. When a caller barges in, hangs up, or changes intent, model generation, retrieval, synthesis, and tools should be cancelled where safe. For side-effecting tools, cancellation is not equivalent to rollback. Use an idempotency key and query final state before telling the caller what happened.

Hedged requests can reduce tail latency for idempotent reads by starting a second request after a delay. They also increase load and can worsen an incident. Permit them only in the green state, cap the duplicate rate, and cancel the loser. Never hedge a payment, booking, account update, or other non-idempotent operation without a protocol designed for it.

Design degradation by semantic value

Performance teams often remove whatever is expensive. That can delete the context required for a correct answer. Classify work by semantic value instead.

Non-sheddable work includes identity checks, consent and disclosure, authorization, critical validation, transaction confirmation, audit events, and safe handoff. If these cannot run, reject or transfer the workflow.

Conditionally sheddable work includes secondary retrieval, stylistic personalization, long explanations, sentiment adaptation, and nonessential analytics. Remove it only when the shortened response remains accurate and clear.

Deferrable work includes summaries, CRM enrichment, quality scoring, and some post-call analytics. Queue it with a durable event and an explicit retention limit. The live call must not wait for it.

Prohibited substitutes include inventing a balance when the account tool is slow, claiming a booking succeeded before confirmation, or presenting a cached policy as current when freshness is required. Fast fiction is worse than a transparent delay.

Write one degradation table per intent. Include green behavior, amber behavior, red behavior, maximum age of cached data, handoff path, and evidence that the reduced response was approved. Exercise every row in a game day.

Calibrate the endpoint detector as a risk model

Endpointing is a classification decision with unequal errors. A false positive declares the user finished when they did not. A false negative keeps waiting after a completed turn. Their costs differ by workflow.

For a yes-or-no confirmation, cutting off “yes, but…” can authorize the wrong action. For a directory lookup, a slightly longer wait may only be annoying. Use intent-specific thresholds where evidence supports them, rather than one universal silence duration.

Build an evaluation set containing actual pause patterns, with consent and minimization controls. Label true turn ends, within-turn hesitations, backchannels, interruptions, and overlapping speech. Report precision, recall, false-positive rate, and delay distribution. Break results down by language, accent group where ethically and lawfully evaluated, channel quality, and utterance length.

Do not tune only on speakers who already know how to talk to bots. Include people who restart sentences, search for words, use fillers, speak slowly, or mix languages. A detector that is quick for scripted testers may systematically interrupt real customers.

The release decision should combine endpoint and business measures. If a new detector saves 180 ms but increases clarification turns, abandoned calls, or incorrect tool parameters, it has not improved the service. Record the trade-off in the release evidence.

Run an overload game day

A load test produces charts; a game day tests decisions. Give the team a scenario it does not fully control. Begin at normal traffic, then introduce a realistic traffic increase, a slow tool, and loss of one compute pool. Observe whether the system enters declared states and whether operators understand the transition.

The game day should answer:

• Did monitoring detect queue growth before callers experienced timeouts?
• Did admission control activate at the documented threshold?
• Did existing transactions receive priority without starving emergency paths?
• Did amber degradation remove only approved optional work?
• Did red mode offer an honest callback or handoff?
• Were side-effecting tools protected against duplicates?
• Did scale-out become usable within the assumed time?
• Did rollback restore the previous model, voice, policy, and routes?
• Did telemetry remain available without exposing conversation content?
• Could the team reconstruct one failed turn from timestamps and outcome codes?

Measure recovery separately from detection. A system can alert in seconds but take twenty minutes to drain a queue. Specify time to protective mode, time to stable service, and time to clear deferred work. Do not declare recovery while callers still receive stale or duplicated operations.

End with a written reconciliation. Compare expected and observed thresholds, list unplanned degradation, assign owners, and block expansion until high-severity gaps close. Game-day success is not “the service stayed up.” It is “the failure stayed inside the declared envelope, and evidence explains every exception.”

Release gate

Before production, require evidence that:

• [ ] Clock events and timestamp source are documented.
• [ ] Caller-side first audio is measured.
• [ ] Endpoint delay and false endpoint rate are both reported.
• [ ] p50, p90, p95, and p99 exist by turn class.
• [ ] Sample counts and confidence limitations are visible.
• [ ] Task success, clarification, transfer, abandonment, and failure are counted.
• [ ] Barge-in detection and playback cancellation meet objectives.
• [ ] Network, speech, tool, and load variations are represented.
• [ ] The overload envelope was exercised beyond its boundary.
• [ ] Safeguards remain active in amber and red states.
• [ ] Tool calls use deadlines and idempotency where side effects exist.
• [ ] Privacy fields, access, retention, and deletion were reviewed.
• [ ] Consent and disclosure rules were tested for each jurisdiction.
• [ ] A rollback or traffic-shift procedure was rehearsed.
• [ ] Results identify model, voice, hardware, region, and deployment versions.

Operational review after launch

Review the system as a changing service, not a finished benchmark. Daily operations should watch admission, queue time, turn-gap percentiles, barge-in delay, failure outcomes, carrier segments, tool latency, and overload-state minutes. Weekly review should examine regressions by language, intent, tenant, and deployment. Monthly review should replay representative traces, validate retention, and reassess capacity assumptions.

Use error budgets carefully. A team must not “spend” safety failures because its latency month was good. Maintain separate budgets for availability, latency, task outcome, and safety-critical errors. If one is exhausted, slow releases or restrict the affected workflow.

Compare versions with paired or randomized traffic where safe. Traffic mix changes can make a new version look faster even when it is not. Record the assignment, exclude invalid traces by a predeclared rule, and keep the same turn definitions.

The durable conclusion

Sub-50 ms can be a valuable component result. It can show that a network hop, decoder, or inference stage is efficient. It cannot prove that a caller receives a timely, natural, correct response.

A production voice agent needs a shared clock, a stage budget, class-specific tail metrics, interruption outcomes, task results, and a tested overload envelope. The team must know which work can be shed and which controls can never be removed. Claims should state load, environment, versions, and measurement boundaries.

Optimize the conversation, not the screenshot. When every turn can be traced from the caller’s last audio to audible response, when slow tools fail safely, and when overload produces an honest controlled outcome, latency becomes an engineering property instead of a marketing adjective.