How Fast Should Your AI Voice Agent Respond?

#voiceai #latency #aiagents #customerservice

A production AI voice agent should start speaking within 800ms-1.2 seconds of the caller finishing a sentence. Past 2 seconds, callers assume the line dropped, start repeating themselves, or hang up. If you're evaluating vendors or building this in-house, latency isn't a nice-to-have metric — it's the difference between an agent people trust and one they route around.

Why latency decides whether people trust the agent

Human conversation has a rhythm. Research on turn-taking across languages found the average gap between one person finishing and the next starting is around 200 milliseconds — often even before the first speaker finishes (Stivers et al., PNAS). Nobody expects an AI to hit 200ms today, but the baseline it's competing against is that fast, not the 3-5 seconds a customer might tolerate from a slow web page.

On the interface side, Nielsen Norman Group's response-time thresholds are the classic reference: under 0.1s feels instant, up to 1s keeps the user's flow of thought uninterrupted, and past 10s they've mentally checked out. Voice compresses that scale — because there's no spinner, no progress bar, just silence — so a 2-second gap on a call feels much longer than the same gap on a webpage.

This matters most for anything transactional: booking changes, order status, support triage, outbound sales. If you're weighing this against a text-based channel, the latency bar for an AI WhatsApp sales agent is far more forgiving — a few seconds reads as normal typing time. Voice has no such cover.

Where the milliseconds actually go

A typical voice agent pipeline has three stages, and most implementations run them sequentially:

Speech-to-text (STT) — transcribing the caller's audio. 150-500ms depending on whether it's streaming or batch.
LLM generation — deciding what to say and generating the full response. 500ms-2s+ depending on model size, prompt length, and whether it's calling tools (looking up an order, checking a calendar) before it can answer.
Text-to-speech (TTS) — converting the full text reply into audio. 200ms-1s.

Add those up sequentially and you're easily at 1.5-3.5 seconds before the caller hears anything — even before network round trips. That's the gap that makes an agent feel broken.

The fix isn't a faster model, it's not waiting for the full response before you start talking. Streaming the LLM's output token-by-token into TTS, and speaking as soon as the first sentence is ready instead of the whole answer, is what actually collapses the perceived delay — the same idea captured in the "streaming sentences, not audio" approach some teams have landed on. It's a pipeline decision, not a model upgrade, and it's usually the highest-leverage fix available.

Tool calls make this harder. If the agent needs to check a database before it can answer ("let me check your order"), that lookup has to happen inside the gap, and it needs its own filler strategy — a natural acknowledgment ("one sec, pulling that up") rather than dead air. This is one of the reasons agentic voice systems need more careful orchestration than a simple scripted IVR replacement — the agent is doing real work mid-conversation, not just reciting text.

How to evaluate a vendor on latency

Don't take a demo video at face value. Ask for:

A live, unscripted call on your own network, not a pre-recorded demo run on idle infrastructure.
Time-to-first-audio-byte, measured — not "response time," which vendors sometimes quote as time-to-first-token from the LLM, ignoring TTS.
Latency under concurrent load. A single test call tells you nothing about what happens at 50 simultaneous calls, when GPU queuing and rate limits kick in.
What happens on tool calls. Ask them to trigger a real lookup (CRM, calendar, order system) mid-call and time the gap.
Regional routing. If your callers are in the Gulf or elsewhere outside the vendor's default region, ask where inference actually runs — a model hosted three continents away adds real, physics-limited round-trip time no amount of engineering fixes.

This last point compounds if you're also dealing with accented speech or a non-English language — Arabic ASR in particular has real accuracy-versus-latency tradeoffs, since higher-accuracy models for Gulf dialects are often slower and pricier than generic English STT.

If you're putting any of this in a contract, treat latency the same way you'd treat an accuracy number — define it, and hold the vendor to it. The same discipline used for evals in AI vendor contracts applies directly: without a measurable SLA, "fast" is whatever the vendor felt like on demo day.

Red flags

Vendor can't tell you their p50/p95 time-to-first-audio-byte off the top of their head.
Demo only works well on scripted questions — ask something off-script and watch the gap widen.
"Fast enough" is the only answer you get when you ask for a number.
No mention of streaming anywhere in their architecture — if they wait for a complete LLM response before TTS starts, you're locked into the slow path no matter which model they swap in.

The bottom line

Latency is an architecture decision, not a model-selection decision. A well-engineered pipeline on a mid-tier model will consistently beat a poorly-streamed pipeline on the best model available. Before you sign with a vendor or greenlight a build, get a real number, not a vibe — and if you're integrating AI into an existing product, budget engineering time for the streaming plumbing up front, because retrofitting it later is a rewrite, not a patch.

If you're scoping a voice agent and want a second opinion on whether a vendor's latency claims hold up, let's talk.

Originally published on the Pykero blog.