Measuring and Reducing Turn Latency in Voice AI Agents

In a voice agent, response latency shapes how natural the conversation feels. A text chatbot can pause for a second or two without much harm. On a phone call, the same pause is noticeable and often reads as a dropped line. This post breaks one conversational turn into stages, shows how to instrument each stage, and lists common techniques for reducing latency.

Where a turn spends its time

A turn runs from the moment the caller stops speaking to the first audio of the reply. The main stages are:

1. Endpointing: deciding that the caller has finished speaking.
2. ASR finalization: producing the final transcript after the last audio frame.
3. LLM time to first token (TTFT): the delay before the model starts generating.
4. Tool calls: external lookups such as CRM, calendar or database queries.
5. TTS time to first byte (TTFB): the delay before speech synthesis emits its first audio chunk.

Without streaming, total turn latency is approximately the sum of these stages. With streaming, some stages overlap (covered below). Either way, each stage needs its own measurement before you decide where to optimize.

Instrumenting a turn

A small tracer is enough to start. It stores the duration of each stage in milliseconds and reports percentiles, because averages hide the slow turns that callers actually notice.

import time
import statistics
from collections import defaultdict
from contextlib import contextmanager

MIN_SAMPLES = 20  # a p95 needs enough samples to mean anything


class TurnTracer:
    """Collects per-stage durations (in milliseconds) across many turns."""

    def __init__(self):
        self.samples = defaultdict(list)

    def add(self, stage, elapsed_ms):
        self.samples[stage].append(elapsed_ms)

    @contextmanager
    def record(self, stage):
        start = time.monotonic()
        try:
            yield
        finally:
            self.add(stage, (time.monotonic() - start) * 1000)

    def report(self):
        out = {}
        for stage, values in self.samples.items():
            if len(values) < MIN_SAMPLES:
                continue
            cuts = statistics.quantiles(values, n=20)  # 19 cut points
            out[stage] = {
                "count": len(values),
                "p50_ms": round(statistics.median(values), 1),
                "p95_ms": round(cuts[18], 1),  # 95th percentile
            }
        return out

For stages that complete in one call, such as a tool lookup, wrap them with record:

tracer = TurnTracer()

with tracer.record("tool_call"):
    result = lookup_customer(phone_number)  # your own function

For streaming stages such as LLM TTFT, stop the clock at the first token rather than at the end of the response, then keep consuming the stream:

start = time.monotonic()
first_token_seen = False

for token in llm_stream(prompt):  # your own streaming client
    if not first_token_seen:
        tracer.add("llm_ttft", (time.monotonic() - start) * 1000)
        first_token_seen = True
    # pass each token on to the next stage (for example, TTS) here

Use time.monotonic() instead of wall-clock time so system clock adjustments cannot distort durations.

Reading the numbers

Compare p95 with p50. When p50 looks fine but p95 is high, the cause is often intermittent: cold connections, long prompts, retries or network jitter. Then compare stages against each other, find the one that takes the largest share of the turn, and work on that one first.

Streaming between stages

Waiting for each stage to finish before starting the next one adds up quickly. Overlapping stages is one of the most effective ways to reduce perceived latency:

• Stream ASR partial results so the final transcript is available soon after speech ends.
• Stream LLM tokens and start TTS as soon as the first complete sentence or clause is ready.
• Stream TTS audio to the caller in chunks instead of waiting for a finished file.

With streaming, the caller can hear the start of the reply while later parts are still being generated.

Barge-in

Callers interrupt. When voice activity is detected during playback, stop audio output, cancel any in-flight generation and return to listening. Treat the interruption path as part of your latency budget, because a slow reaction to an interruption feels as unnatural as a slow answer.

A regression test for latency

Latency often creeps up as prompts, tools and models change. One approach is to keep a fixed set of recorded test utterances, replay them in CI and fail the build when a stage exceeds a budget.

def test_turn_latency_budget():
    report = run_replay_suite()  # your own helper: replays utterances, returns tracer.report()
    assert report["llm_ttft"]["p95_ms"] < 800
    assert report["tool_call"]["p95_ms"] < 300

The thresholds above are placeholders. Set budgets from your own measurements and product requirements.

Summary

• Break each turn into stages and measure every stage.
• Track p50 and p95 rather than averages.
• Stream between stages so they overlap.
• Handle barge-in as a first-class path.
• Guard your latency budgets with a replay test in CI.

Written by the team at VRM AI Technology, an AI development company building Generative AI systems, conversational voice agents and custom machine learning software.