Your Voice Agent Does Not Have One Latency: Build a Conversation SLO

“Sub-50 ms” can be a true measurement and a useless production claim at the same time.

A team may measure a fast inference kernel, a local voice-activity detector, or the first token from a model and then present that number as the latency of a phone conversation. The caller experiences something else: carrier setup, jitter buffering, speech detection, end-of-turn judgment, transcription, retrieval, tool execution, generation, speech synthesis, audio transport, and sometimes a human handoff. A fast component cannot certify a fast turn.

The right engineering question is not, “How low is our model latency?” It is: what timing, interruption, and recovery behaviour will callers receive for each class of conversational turn, including the tail?

This article adapts the architecture described in the source discussion of hybrid voice AI and low-latency inference, but changes the unit of control. Instead of treating infrastructure speed as the product outcome, it defines a conversation service-level objective, or SLO. The SLO covers when the system listens, when it decides that a turn ended, when the first intelligible audio begins, whether the caller can interrupt, and how the system behaves when a dependency slows or fails.

The goal is not to make a machine “sound human.” That phrase is subjective and often encourages theatrical optimisation. The goal is to make the call understandable, interruptible, predictable, and recoverable for a bounded business task.

Start with the latency the caller can observe

A conversational turn has several clocks. If the team reports only one, it will optimise the wrong thing.

Call setup latency

This begins when the calling or receiving workflow is initiated and ends when two-way media is available. It can include carrier routing, session negotiation, authentication, and media establishment. A system can respond quickly after audio arrives while still making callers wait during setup.

End-of-turn decision latency

The agent cannot answer until it decides whether the caller has finished. Silence is ambiguous. A pause can mean completion, hesitation, a search for a document, a breath, or a temporary network gap. An aggressive endpoint detector creates fast responses by interrupting people. A conservative detector avoids interruption but creates dead air.

Turn-taking research explains why a simple silence threshold is inadequate. A broad review of conversational turn-taking systems describes the roles of verbal cues, prosody, breathing, gaze in embodied settings, end-of-turn detection, and interruption handling. On a telephone call, many visual signals disappear, which makes timing and acoustic evidence more important, not less.

Response planning latency

After the turn is accepted, the system may classify intent, retrieve context, apply policy, call a business system, and generate an answer. This is not one model request. A tool call can dominate the path even when the language model is fast.

Time to first intelligible audio

The caller does not perceive a text token. The relevant event is the beginning of speech that communicates useful content. A filler sound or a generic “one moment” can reduce perceived silence, but it must not conceal an unbounded wait or falsely suggest that an action has completed.

Turn completion latency

Streaming can produce a fast first word while the full answer arrives slowly, stalls, or contradicts itself. Measure the complete turn as well as the first audio. For transactional tasks, also measure time until the business action is durably accepted.

Interruption latency

When the caller speaks over the agent, how quickly does output stop? Does the system preserve the caller's speech? Does it cancel an uncommitted action or continue in the background? A fast answer that cannot be interrupted is not a responsive conversation.

Recovery latency

When transcription confidence falls, a tool times out, or synthesis fails, how long until the system asks a bounded clarification, retries safely, transfers, or ends the attempt? Recovery is part of performance because failures occur during real calls.

The measurement boundary should begin at the telephony edge that the company controls and end at audio delivered back to that edge. If the carrier provides timestamps, retain them. If it does not, distinguish internal latency from carrier latency rather than silently mixing the two.

Replace the headline number with a latency budget

A latency budget assigns time and failure expectations to the stages on the critical path. It is not a promise that every stage always consumes its full allocation. It is a control that reveals where the tail originates and what can degrade independently.

Stage	Start event	End event	Primary measure	Hidden failure to expose
Media ingress	Audio received at edge	Frames available to pipeline	p50, p95 jitter and gaps	Reordered or missing frames
Speech activity	Speech begins	Activity detected	detection delay, false starts	Noise classified as caller
Endpoint decision	Last meaningful speech	Turn accepted	p50, p95, p99	Caller pause cut off
Transcription	Accepted audio span	Stable transcript	partial and final latency	Late transcript revision
Policy and routing	Transcript stable	Route selected	decision latency	Unsafe default route
Retrieval/tool	Request sent	Verified result	p50, p95, timeout rate	Stale or partial result
Generation	Context ready	first usable text	first-token and completion	Unsupported assertion
Synthesis	usable text	first intelligible audio	first-audio latency	audible filler only
Media egress	audio emitted	edge delivery	queue and packet delay	backpressure or clipping
Durable action	user confirmation	committed state	commit and confirmation time	speech says success before commit

The budget must separate eligible turn classes. A greeting, a short factual answer, an appointment lookup, an address correction, and a transaction confirmation do not share the same dependency path. Combining them into one average rewards systems that are fast on easy turns and unreliable on important ones.

Define a small taxonomy before testing:

• LOCAL_ACK: acknowledgement with no external tool;
• KNOWLEDGE: retrieval-backed informational answer;
• LOOKUP: read from a business system;
• WRITE_PREVIEW: prepare a change but do not commit;
• WRITE_COMMIT: commit after explicit confirmation;
• CLARIFY: resolve low confidence or ambiguity;
• HANDOFF: connect to a human or queue;
• RECOVERY: respond to a dependency or policy failure.

Each class gets a different SLO and a different allowed fallback. A local acknowledgement may need a tight first-audio objective. A durable transaction may accept a longer wait if the agent clearly marks progress and never claims success before the commit response.

Use distributions, not a mean

The mean conceals callers who receive the worst experience. Report p50 to show the typical path, p95 to expose recurring slow paths, and p99 when volume is sufficient to make the estimate stable. Also report the sample count and confidence interval or observation window. A p99 calculated from a handful of turns creates false precision.

Google's Site Reliability Engineering guidance emphasises latency distributions and warns that coarse observations can miss spikes that drive tail latency in its chapter on monitoring distributed systems. A voice pipeline is a distributed system with an unusually visible user sitting at the end of it. The same principle applies: aggregate averages are not enough to find the critical path.

Slice the distributions by language, acoustic condition, carrier, geography, turn class, tool, model version, and release. Do not create so many slices that every chart is sparse. Start with factors that can change routing or computational work.

A sub-50 ms component is not a sub-50 ms conversation

The source architecture's focus on edge inference and bare-metal acceleration can be valuable. Moving a function closer to the media edge may reduce transport and queueing. Dedicated capacity may make a component more predictable. Streaming may overlap stages that would otherwise run sequentially. None of those facts makes an end-to-end claim automatic.

Consider a simplified path. Speech detection takes 20 ms after frames arrive. Endpointing waits 350 ms to avoid cutting off natural pauses. A partial transcript stabilises after 180 ms. Retrieval takes 240 ms. Generation produces usable text in 160 ms. Synthesis emits intelligible audio after another 120 ms. Even if one inference stage takes 40 ms, the sequential portion of the turn is already much larger.

Those numbers are illustrative, not benchmarks. The important engineering exercise is to determine which work can overlap and which work cannot.

For example, streaming transcription can begin before the caller finishes, but committing the intent too early can misroute a negation or correction near the end. Retrieval can be speculated from a partial transcript, but the result must be discarded if the final intent changes. Speech synthesis can begin from the first sentence, but early audio constrains later generation and makes corrections audible.

Optimisation therefore creates consistency risks. Every speculative stage needs a cancellation rule, version identifier, and state boundary.

Measure conversation correctness beside speed

A latency SLO without quality guardrails will teach the system to answer quickly and badly. Pair timing with:

• endpoint false-cut rate;
• missed-speech rate;
• interruption success rate;
• transcript semantic error for task-critical fields;
• tool selection correctness;
• unsupported-answer rate;
• durable-action mismatch rate;
• clarification success rate;
• handoff completeness;
• caller abandonment by turn class.

Do not compress these into one “conversation score.” A severe action mismatch should remain visible even if all other measures are strong. Use red-line events that block promotion regardless of average latency.

Research on human dialogue also cautions against treating the shortest gap as universally best. A study of vocal reaction to speech offsets reports that transitions below roughly 200 ms are unlikely to be purely reactive to the final boundary; speakers use predictive cues. The operational implication is not a universal 200 ms target. It is that good turn-taking depends on prediction, prosody, context, and interruption repair—not simply waiting for a fixed silence and racing to speak.

Define the conversation SLO

An SLO should specify the population, measurement point, threshold, percentile, window, exclusions, and quality guardrail. Here is a fictional example for a bounded appointment-status line. The values demonstrate structure only; they are not market benchmarks.

Objective	Eligible population	Target	Window	Guardrail
First intelligible audio after accepted turn	LOCAL_ACK turns with valid media	p95 within the declared internal threshold	rolling 7 days	false-cut rate below approved limit
Retrieval-backed response start	KNOWLEDGE turns with healthy index	p95 within class threshold	rolling 7 days	citation/grounding acceptance above limit
Tool lookup response	LOOKUP turns where dependency accepts request	p95 within class threshold	rolling 14 days	no stale result presented as current
Barge-in stop	caller speech during agent output	p95 output stop within threshold	rolling 7 days	caller audio retained for recognition
Safe recovery	timed-out tools	response or handoff within threshold	rolling 30 days	no success statement before result
Durable action	confirmed WRITE_COMMIT turns	p99 commit or explicit failure within threshold	rolling 30 days	zero unconfirmed commits in sample

The thresholds should come from observed baselines, task risk, user testing, and the cost of capacity—not from a competitor's marketing page. Record who approved them and when they expire.

The SLO needs an error budget. If 99% of eligible turns must satisfy a timing objective, the remaining 1% is not permission to ignore callers. It is a budget for expected variation that governs releases. When the budget is consumed too quickly, feature rollout slows and reliability work takes priority.

Create separate budgets for latency and critical correctness. Never allow fast turns to compensate for unsafe actions. A latency miss can use a graceful progress response or handoff. A critical correctness failure may require immediate rollback.

Account for silence as a product decision

Endpointing silence is part of the interaction design. Test multiple caller behaviours:

• fluent speech with a clear final cadence;
• hesitation inside an address or account reference;
• self-correction after a pause;
• background conversation;
• a caller reading from a document;
• a caller using a second language;
• packet loss that resembles silence;
• intentional interruption of the agent.

Report latency from both “last audio frame” and “turn accepted.” The first captures what the caller feels; the second isolates downstream processing. Without both, a team can improve the model path while endpointing remains the dominant delay.

Instrument a turn as a trace

Logs answer what happened at one component. A trace connects one conversational turn across components and releases. Give every call a pseudonymous call identifier and every turn a turn identifier. Avoid putting raw audio, transcripts, phone numbers, or customer records into general observability fields.

A useful trace includes timestamps for:

1. first and last received audio frames;
2. voice-activity start and stop events;
3. endpoint acceptance and reason;
4. partial and stable transcription;
5. route and policy decision;
6. each retrieval or tool request and response;
7. generation start, first usable text, and completion;
8. synthesis request, first audio, and completion;
9. egress queue and delivery;
10. caller barge-in and output cancellation;
11. proposed, confirmed, and committed business actions;
12. fallback, retry, handoff, and final disposition.

Attach model, prompt, policy, tool, knowledge-index, and voice versions. If latency changes after a release, versioned traces make the regression attributable. Without versions, a dashboard can reveal deterioration but not explain it.

Make the calculation executable

The following Node.js example turns trace events into stage durations and evaluates a fictional class-level SLO. It intentionally rejects incomplete traces rather than treating missing events as zero latency.

// Save as conversation-slo.js and run with: node conversation-slo.js
const turns = [
  {
    id: "t-001",
    class: "LOOKUP",
    lastAudioMs: 1000,
    endpointAcceptedMs: 1340,
    transcriptStableMs: 1485,
    toolStartMs: 1510,
    toolEndMs: 1730,
    firstUsableTextMs: 1840,
    firstAudioMs: 1950,
    bargeInMs: null,
    outputStoppedMs: null,
    criticalFailure: false,
  },
  {
    id: "t-002",
    class: "LOOKUP",
    lastAudioMs: 4000,
    endpointAcceptedMs: 4390,
    transcriptStableMs: 4540,
    toolStartMs: 4560,
    toolEndMs: 5100,
    firstUsableTextMs: 5210,
    firstAudioMs: 5350,
    bargeInMs: 5600,
    outputStoppedMs: 5710,
    criticalFailure: false,
  },
];

function duration(end, start, label, id) {
  if (!Number.isFinite(end) || !Number.isFinite(start) || end < start) {
    throw new Error(`invalid ${label} timestamps for ${id}`);
  }
  return end - start;
}

function percentile(values, p) {
  if (!values.length) return null;
  const sorted = [...values].sort((a, b) => a - b);
  const index = Math.ceil(p * sorted.length) - 1;
  return sorted[Math.max(0, index)];
}

function analyse(turns, config) {
  const eligible = turns.filter(t => t.class === config.turnClass);
  const measured = eligible.map(t => ({
    id: t.id,
    endpointMs: duration(t.endpointAcceptedMs, t.lastAudioMs, "endpoint", t.id),
    downstreamMs: duration(t.firstAudioMs, t.endpointAcceptedMs, "downstream", t.id),
    callerWaitMs: duration(t.firstAudioMs, t.lastAudioMs, "caller wait", t.id),
    bargeInStopMs: t.bargeInMs == null ? null
      : duration(t.outputStoppedMs, t.bargeInMs, "barge-in stop", t.id),
    criticalFailure: t.criticalFailure,
  }));

  const waits = measured.map(t => t.callerWaitMs);
  const p95 = percentile(waits, 0.95);
  const criticalFailures = measured.filter(t => t.criticalFailure).length;
  const passing = measured.filter(t => t.callerWaitMs <= config.p95LimitMs).length;

  return {
    turnClass: config.turnClass,
    sampleSize: measured.length,
    p95CallerWaitMs: p95,
    objectiveAttainment: measured.length ? passing / measured.length : null,
    criticalFailures,
    decision: measured.length < config.minimumSample
      ? "INSUFFICIENT_EVIDENCE"
      : criticalFailures > 0
        ? "ROLLBACK_OR_CONTAIN"
        : p95 <= config.p95LimitMs
          ? "SLO_MET"
          : "ERROR_BUDGET_AT_RISK",
    turns: measured,
  };
}

console.log(analyse(turns, {
  turnClass: "LOOKUP",
  p95LimitMs: 1600,       // fictional threshold, not a benchmark
  minimumSample: 2,       // deliberately tiny for demonstration only
}));

For a real service, compute percentiles with a validated metrics system and enough observations. The code's value is the explicit boundary: caller wait begins at the last received speech and ends at first intelligible agent audio. Endpoint and downstream time remain separately diagnosable.

Design degradation before overload arrives

Low-latency systems fail badly when every request insists on the premium path. Define degradation levels that preserve truth and caller control.

Level 0: full service

The normal model, tools, knowledge source, voice, and observability path are healthy. All eligible intents are available.

Level 1: bounded simplification

When generation or synthesis capacity is constrained, shorten responses, disable nonessential enrichment, and prefer deterministic confirmations. Do not silently lower safety checks.

Level 2: read-only mode

Disable writes when the transaction dependency is unstable or confirmation evidence cannot be retained. The agent may provide verified information and offer a later callback or human queue.

Level 3: structured capture

Collect only the minimum details required for a safe handoff. Repeat them for confirmation, explain that the action is not complete, and create a traceable work item.

Level 4: transparent unavailability

End the automated path cleanly. Do not loop apologies or imply that an action is pending when no durable record exists.

Every transition needs an observable trigger and recovery condition. Queue depth, dependency timeout rate, error-budget burn, model saturation, and media loss may trigger degradation. A manual operator must also be able to force a level during an incident.

The SRE principle of graceful degradation is more valuable than a heroic benchmark. Production engineering must assume partial failure, not merely fast success.

Treat telephony obligations as architecture inputs

Latency engineering cannot be separated from call purpose. An inbound support line, an appointment reminder, and an outbound sales call have different consent, disclosure, recording, and opt-out requirements across jurisdictions.

For US traffic, the FCC consumer guidance on unwanted robocalls and texts explains that prerecorded or artificial-voice calls can be subject to consent and revocation requirements. That page is not a complete compliance design, and rules vary by call type and destination. Its engineering consequence is clear: permission, suppression, and revocation cannot be an afterthought in a dialler. They must be checked before call initiation, versioned, and propagated quickly enough to stop future calls.

A low-latency outbound system that calls the wrong person faster is not better. Keep consent state out of the model prompt as an informal note. Enforce it in a deterministic policy layer. Log the policy decision without exposing private data to general telemetry.

Governance also needs ownership. KPMG's discussion of AI in risk management frames trusted AI around governance and controls rather than model capability alone. The relevant operational translation is to assign owners for models, data, business actions, incidents, and user recourse. A latency SLO belongs beside those controls, not above them.

Validate with a staged production-readiness review

Do not promote from a lab benchmark directly to unrestricted calls. Use stages that widen one risk dimension at a time.

flowchart TD
  A[Representative recorded test set] --> B{Timing and quality gates pass?}
  B -- No --> X[Profile, redesign, or narrow scope]
  B -- Yes --> C[Shadow traffic with no business actions]
  C --> D{Trace complete and tails understood?}
  D -- No --> X
  D -- Yes --> E[Internal callers and synthetic failures]
  E --> F{Barge-in, timeout, and rollback pass?}
  F -- No --> X
  F -- Yes --> G[Small live cohort]
  G --> H{Error budget healthy?}
  H -- No --> I[Degrade or rollback]
  H -- Yes --> J[Expand one turn class or traffic slice]
  J --> K[Post-release review]
  K --> H

The recorded test set should include accents, speech rates, pauses, corrections, noise, low volume, clipped audio, and task-specific entities. Synthetic network impairment can add jitter, loss, and delay. Tool dependencies should be made slow, unavailable, stale, and contradictory on purpose.

Shadow traffic tests classification and planning without speaking to callers or committing actions. Internal calling tests the full media loop. A small live cohort validates behaviour that recordings cannot reproduce, but only after rollback and human escalation are proven.

At each stage, compare the latency distribution with the quality guardrails. A release that improves p50 while worsening p99, interruption, or critical correctness should not be promoted merely because the demo sounds faster.

Production-readiness evidence table

Review question	Required evidence	Block condition
Is the timing boundary honest?	edge-to-edge trace and clock validation	component-only metric presented as end-to-end
Are tails understood?	p50/p95/p99 by turn class and dependency	sparse or aggregated data hides slow path
Can callers interrupt?	barge-in tests with retained caller audio	output continues or caller speech is lost
Are actions truthful?	proposal, confirmation, commit trace	spoken success precedes durable commit
Can the service degrade?	tested level transitions and recovery	premium path is the only path
Can it roll back?	previous version restored under load	rollback depends on manual reconstruction
Are calls authorised?	deterministic policy decision and revocation propagation	permission lives only in prompt or CRM note
Can a human recover context?	concise handoff packet and disposition	transfer loses intent, evidence, or caller state

Failure modes that fast demos conceal

Optimising first token instead of first meaning

The agent begins with a filler phrase quickly, then waits. Report first intelligible task content separately from first audio. If progress language is used, it must accurately describe the state.

Endpointing cuts off hesitant callers

The p50 improves because the system accepts turns too early. Track false cuts and corrections. Test languages and caller groups separately rather than assuming one silence rule.

Speculative work commits on stale intent

Retrieval or a tool call starts from a partial transcript, then the caller corrects the request. Speculation must be cancellable, and writes must wait for stable intent and explicit confirmation.

The mean improves while the tail collapses

Dedicated fast paths handle common turns, but a saturated tool or cold model makes a minority wait several seconds. Use percentiles, dependency spans, and error-budget burn alerts.

Barge-in stops audio but not the action

The agent appears interruptible, yet the backend continues a write. Separate audio cancellation from action cancellation. Use idempotency keys and explicit commit states.

Retry produces a duplicate action

A timeout is ambiguous: the tool may have committed before the response was lost. Never retry a write blindly. Query status by idempotency key or transfer for resolution.

Clock skew corrupts the trace

Distributed timestamps make negative or impossible durations. Validate clock synchronisation, prefer monotonic clocks inside a process, and record where timestamps were created.

Compliance checks sit outside the critical path

The agent starts a call before suppression or permission is confirmed because policy lookup is considered “administrative.” Make the deterministic policy result a precondition, cache only with a justified freshness rule, and propagate revocation.

Degradation removes the safety layer

During overload, the system skips retrieval, confirmation, or policy to preserve speed. Degradation may reduce features, never remove non-negotiable controls.

A handoff resets the conversation

The human receives a call without the caller's verified intent, completed checks, or failed tool state. Measure handoff completeness and time to human comprehension, not only transfer connection time.

Final validation checklist

Before a release receives production traffic, verify:

• The SLO names a caller-observable boundary, not a component benchmark.
• Turn classes are defined and mutually understandable to engineering and operations.
• Call setup, endpointing, downstream processing, first intelligible audio, completion, interruption, and recovery are measured separately.
• p50, p95, and the supported tail percentile include sample counts and windows.
• Latency is sliced by the few dimensions that can change the critical path.
• Quality guardrails prevent speed from rewarding false cuts or wrong actions.
• Red-line correctness failures cannot be averaged away.
• Speculative transcription, retrieval, or generation has cancellation and version rules.
• A spoken success statement follows, never precedes, a durable commit.
• Write retries use an idempotency or status-reconciliation mechanism.
• Caller barge-in stops output and preserves the caller's speech.
• Every external dependency has a timeout, fallback, and trace span.
• Degradation levels preserve consent, policy, confirmation, and truthfulness.
• Permission and suppression are deterministic preconditions for applicable outbound calls.
• Observability avoids raw phone numbers, transcripts, and customer records in general metrics.
• Model, prompt, policy, voice, tool, and knowledge versions are attached to traces.
• The test set covers pauses, corrections, noise, languages, loss, jitter, and slow tools.
• Shadow, internal, and limited-live stages pass before wider promotion.
• Rollback is tested under realistic load.
• Human handoff carries a concise, verified context packet.
• Error-budget burn changes release decisions rather than merely colouring a dashboard.

If the team cannot answer one of the first six checks, it does not yet have a conversation performance measure. If it cannot guarantee action truthfulness, permission enforcement, or rollback, it does not yet have a production service.

Optimise the turn, not the benchmark

Edge inference, dedicated accelerators, streaming models, and regional media processing can all reduce latency. They are implementation choices inside a larger contract.

The contract begins when the caller finishes a meaningful turn and ends when useful audio arrives, an action is safely completed, or a transparent recovery path takes over. It includes endpointing, tools, tails, interruptions, and failure. It distinguishes a greeting from a transaction and an internal timestamp from the caller's experience.

A component that runs in 50 ms is worth measuring. It is not permission to call the conversation “sub-50 ms.” Build the trace, define class-specific SLOs, pair them with correctness guardrails, and let the error budget govern release speed. That produces a voice system that is not merely fast in a demo, but predictable when callers hesitate, interrupt, correct themselves, or reach a dependency on its worst day.