A Faster Model Will Not Fix Your Slow Voice Agent

The first thing I check when a client tells me their AI voice agent "feels slow" is not the model. It is almost never the model. I have swapped a faster model into a sluggish agent and had callers notice nothing at all, because the seconds they were feeling were never being spent where everyone assumed.

If you have ever profiled a slow page and found the real cost was a render blocking script rather than the function you spent all week optimizing, you already understand this post. Voice just makes the mistake more expensive, because a phone call has no spinner. No loading state, no skeleton UI, no way to signal "still working" other than talking. Silence is the only progress indicator you get, and to a human on a phone, silence means the line dropped, the other person is confused, or nobody is listening. All three make callers talk over the agent, repeat themselves, or hang up.

I build production voice agents on Retell wired into n8n, GoHighLevel and Twilio. Before that I spent years in Unity and AR, where a late frame was a safety problem rather than a conversion problem. That is where I learned to treat latency as a property of the whole chain rather than a number attached to one component.

The metric is turn latency, not time to first token

The number everybody quotes is time to first token, or whatever response time the vendor publishes. The number callers experience is turn latency: the gap between the moment they stop speaking and the moment they hear the agent start.

That gap is a sum, and the model is one term in it:

1. Endpointing. The system waits until it is confident the caller is done. A deliberate wait, and frequently the largest single item in the budget.
2. Transcription. Speech becomes text. Usually streaming, so most of it overlaps with the caller still speaking.
3. Reasoning. The model reads state and decides what to say, and whether to call a tool.
4. Tool calls. Anything the agent must ask an external system before it can answer.
5. Speech generation. The reply becomes audio, and audio takes real time to play no matter how fast it was generated.
6. Network and telephony. Delay you do not control.

Two things fall out of writing the chain down. Shaving 200ms off step 3 while step 1 sits at two seconds is the classic optimizing-the-wrong-thing move. And steps 4 and 5 are where badly behaved agents actually lose people.

Endpointing is the config you set once and never looked at again

Endpointing is the agent deciding you finished your sentence. Too short and it interrupts people mid thought, the rudest failure mode a phone system has. Too long and every turn carries a dead pause, so the agent feels slow even when everything downstream is fast.

The mistake is treating it as one global constant. Different moments deserve different patience:

• Short answers get short waits. "Yes." A confirmation. A one word choice. Lingering here is what makes an agent feel sluggish across an entire call.
• Open questions get long waits. Ask someone why they are calling and they think out loud, pause, resume. Cut them off and you lose the actual reason for the call.
• Anything recited digit by digit gets the most patience. People say phone numbers in chunks with real gaps. An agent that jumps in after the first chunk gets the number wrong, and a wrong callback number is a lost lead no matter how snappy the call felt.

Same shape as the rest of agent design: loose where the caller is exploring, tight where the caller is confirming.

Rule one: nothing on the critical path the next sentence does not need

This one rule fixes most "slow agent" complaints I get handed.

A voice agent wired into a real business touches a CRM, a calendar, a messaging platform, an automation layer. The instinct is to do all of it inline, mid conversation, while the caller listens to nothing. That is how you get four seconds of silence after someone says "yes, book me in".

Split the work by one question: does the caller's next sentence depend on it?

• Blocking work is anything the agent must know before it can speak truthfully. Real availability before it offers a slot. Whether an account exists before it claims one does. Keep this list short and defend its shortness.
• Non-blocking work is everything else. Creating the contact, tagging the lead, firing the confirmation SMS, writing the summary, kicking off follow up. None of it belongs in the caller's waiting time.

In practice the agent fires a webhook at the automation layer and keeps talking. This is the job queue pattern every backend engineer already knows, applied where latency is measured in human patience. Nobody is standing at the CRM watching for the row to appear, so the row can appear five seconds late.

The one wait worth taking, and how to cover it

Calendar availability is the honest exception. If the agent offers a slot it has not verified, you traded a two second pause for a double booking, which is far more expensive to explain to a client.

So take the wait, but never take it in silence. Speak first, then look up. "Let me check what we have this week" costs about a second and a half of speech, roughly what the lookup needs, and the caller experiences zero dead air because a voice was talking the whole time.

That is not a trick. It is what a receptionist does while their booking screen loads, and it is the same instinct as an optimistic UI update. The rule generalizes: if you cannot remove a wait, cover it with speech that was going to happen anyway. What you must not do is fill the gap with filler promising progress the system is not making. Callers notice an agent that says "one moment" three times in a row.

Rule two: the agent's own sentences are part of the budget

Generated speech plays at human speed. A four sentence answer takes four sentences worth of seconds no matter how fast the model produced it. If your agent opens with a paragraph, you spent more of the caller's patience on your greeting than on any technical delay in the stack.

• Cap reply length in the prompt and mean it. One or two sentences, then a question. Long thorough answers read beautifully in testing and feel interminable on a phone.
• Front load the useful part. Answer, then explain if asked. A caller who has their answer will interrupt the rest, and that is a good outcome.
• Never read back more than necessary. Confirm what would be expensive to get wrong. Skip the rest.
• Keep the greeting to one line. It is your time to first byte, and every caller pays it.

Rule three: keep the per-turn work small

Prompt size and tool count both cost time, and both grow quietly. Every rule added after one bad call, every edge case pasted into the system prompt, every tool bolted on for a feature used twice a month adds work to every turn of every call forever. It is dependency creep, and it bills you per request.

The structural fix is the one that also makes agents more reliable: break the single long prompt into a flow where each node carries only the context and tools it needs. The node collecting a phone number does not need the refund policy in its prompt.

How I actually test this

Average latency is a comforting and largely useless number. Callers do not remember the average turn. They remember the one four second gap in the middle of giving their address. Voice is a p99 problem wearing a p50 dashboard.

• Listen to whole recordings, not dashboards. Latency problems are obvious in twenty seconds of audio and invisible in a summary metric.
• Test the paths that touch other systems. The booking turn, the lookup turn, the handoff turn. That is where the seconds hide.
• Test messy callers. Someone who volunteers their number before being asked, someone who pauses mid sentence, someone with a TV on in the background. Those turns reveal what your endpointing settings really are.
• Test when the business is busy, because third party systems slow down under load and your agent inherits all of it.

Where I learned to think this way

In my AR years, latency was not a conversion metric. If an overlay lagged the operator's head, the illusion broke and the person stopped trusting the headset entirely. The fix there is the fix here: decouple what the user feels from what the system does. A firefighter looking at a stale marker and a caller listening to three seconds of silence are having the same experience, which is a system failing to prove it is still there.

The short version

If an agent feels slow, work down this list in order:

1. Tune endpointing per moment instead of globally. Tight on short answers, patient on open questions and digits.
2. Move everything the next sentence does not need off the critical path.
3. Cover the one unavoidable wait, availability, with speech that was going to happen anyway.
4. Cut the agent's own sentences down, starting with the greeting.
5. Shrink prompts and tool lists per node.
6. Test the worst turn on messy calls, not the average turn on clean ones.

None of it requires a faster model. Speed on a phone call is a design decision about where the work happens, made over and over, and it is usually the difference between an agent a business trusts with its phone and one that quietly costs it leads.

I write more about voice agent builds at nabeelbaghoor.com/blog.