A founder pinged us with a UX problem disguised as an engineering question. His team had launched an AI chat product. Users were abandoning the conversation before the agent finished responding. The team had measured p95 response latency at 31 seconds. Their assumption was that they needed to switch to a faster model.
The actual model was responsible for about 35% of the total latency. The other 65% was sequential tool calls, unnecessary intermediate LLM steps, and a missing streaming layer. Switching to a smaller model would have cut maybe 5 seconds off the worst case while degrading response quality.
We made four changes that did not touch the model. P95 latency dropped from 31 seconds to 8 seconds. User abandonment rate dropped 70%. The model stayed the same.
This is the latency stack that most teams either do not measure or do not know how to optimize. Here is the pattern.
Where the time actually goes
Before any optimization, instrument the agent and trace where time is spent. The breakdown for a typical multi-step agent looks something like:
- LLM calls (cumulative): 30-50% of total latency
- Tool calls (cumulative): 30-40% of total latency
- Network and serialization overhead: 5-10%
- Application logic between steps: 5-10%
The LLM call portion is hard to optimize without changing the model. The other 50-70% is where most of the win is, and most teams do not look there because the LLM feels like the obvious bottleneck.
The founder's agent had:
- 8 LLM calls per response (planner + 4 tool dispatchers + critic + 2 retries)
- 6 tool calls (4 distinct tools, with 2 retries)
- All sequential
- No streaming back to the user
Total p95 was 31 seconds. The model was responsible for 11 seconds of that. The other 20 seconds were structural.
Change 1: Parallelize independent tool calls
The biggest win in most agent systems is making tool calls concurrent.
The founder's agent had a step where it needed to look up the user's account information, the customer's order history, and the related support tickets. The original code did this sequentially:
- Look up account (1.5s)
- Look up orders (2.0s)
- Look up tickets (1.8s)
Total: 5.3 seconds, all wait time.
The fix was async-await with parallel dispatch:
- Dispatch all three lookups concurrently
- Wait for all to complete
- Total: 2.0 seconds (the slowest of the three)
That single change saved 3.3 seconds per request. The actual code change was 8 lines.
The pattern: any time the agent calls multiple tools and the results are independent, those tool calls can run concurrently. Use asyncio.gather in Python, Promise.all in JavaScript. Modern agent frameworks (LangGraph, CrewAI Flows) support this natively.
Audit your agent for serial tool calls that could run in parallel. There is almost always one or two.
Change 2: Cut the unnecessary intermediate steps
The original agent had a planner LLM that generated a plan, then a separate critic LLM that reviewed the plan, then the executor.
The critic was rejecting roughly 8% of plans, which meant 92% of the time it was an extra LLM call (3 seconds) for nothing. And of the 8% it rejected, the planner produced a similar plan on the next iteration anyway, suggesting the critic was not adding much real value.
We removed the critic, kept the planner, and added validation rules instead (does the plan reference real tools, does it have any cycles, does it stay under the step limit). The validation runs in milliseconds, not seconds, and catches the same class of bad plans the critic was catching.
3 seconds saved per request, on average. Quality measured no different on the eval set.
The pattern: every LLM call in the chain should be earning its keep. If a step rejects only a small fraction of inputs and the alternative is a deterministic check, the deterministic check is faster and usually as good.
Change 3: Stream the response back to the user
The original agent waited for the full response to be generated before returning anything to the user. The user saw a loading spinner for 31 seconds, then the full response appeared.
Streaming changes the perceived latency dramatically. The user starts seeing tokens within 1-2 seconds, even if the full response takes 8. The agent is not actually faster, but the UX feels faster, and abandonment rate drops because users get visible feedback.
The implementation is a few lines of code. The OpenAI and Anthropic APIs both support streaming natively. The frontend needs to handle server-sent events (SSE) or websockets. The user experience improvement is large and immediate.
For multi-step agents, you can also stream intermediate progress: "Looking up your account... Checking order history... Generating response..." This is qualitatively different from a spinner. Users will wait 8 seconds for visible progress; they will not wait 8 seconds for a spinner.
Change 4: Cache aggressively
Some queries repeat. Some intermediate steps repeat across queries. Both are cacheable.
The founder's agent had a "policy lookup" tool that retrieved from a small KB of company policies. The policies changed maybe once a month. The agent was hitting the KB and running a vector search every time. We added a simple in-memory cache with a 5-minute TTL. The cache hit rate was 35%, saving 0.8 seconds per cached request.
Semantic caching is the more advanced version: cache LLM responses keyed on the embedding of the query. If a new query is semantically similar to a recent one, return the cached response. We use this carefully (only for queries with high confidence and low staleness risk), but it can save 5+ seconds on cache hits.
Aggressive caching: every tool call and every LLM call should be evaluated for cacheability. Most are. Most teams do not bother. The latency improvement compounds.
What the founder shipped
After all four changes:
- Parallel tool calls: -3 seconds
- Removed critic LLM: -3 seconds
- Streaming: perceived latency dropped from 31s to 1-2s for first token
- Caching: -2 seconds average
- Total p95: 31s → 8s
User abandonment rate dropped 70% in the first two weeks. Customer satisfaction scores went up. The model and the agent's core logic did not change.
The team's original instinct (switch to a smaller model) would have saved maybe 5 seconds while degrading quality. The structural changes saved 23 seconds while keeping quality the same.
The latency optimization checklist
Before approving "switch to a faster model" as the latency fix, walk through:
- Are tool calls running sequentially when they could run in parallel? Almost always at least one set is serial that could be concurrent.
- Is every LLM step earning its keep? Critics, validators, refiners that fire on every request might be removable in favor of deterministic checks.
- Is the user seeing intermediate progress, or just a spinner? Streaming alone changes the perceived latency more than any model swap.
- Are repeated queries cached? Tool calls, LLM calls, retrieved chunks: most are cacheable with TTL.
- Are smaller models being used for non-critical steps? Use GPT-4o for the final answer if quality matters; use GPT-4o-mini for the planner, the router, the validator.
Most agent systems have 50-70% latency improvement available without touching the primary model. Audit before you swap.
The model swap is the last resort
Switching to a smaller, faster model can be the right call for some agents. But it is the last resort, not the first. The cost is response quality, and that cost is permanent. The structural improvements are free.
The pattern Sapota recommends: optimize the structure first, measure the new latency, then decide whether a model swap is still needed. Often it is not. When it is, you have a much smaller gap to close, and the trade-off is more justified.
If your agent's latency is killing UX
If your team has launched an AI agent and users are abandoning conversations before the response arrives, the issue is rarely the model. It is usually the structure.
Sapota offers a one-week latency audit that traces your agent's actual time spent, identifies the parallelizable tool calls, the removable LLM steps, and the missing caching opportunities, and ships the optimizations as working code. We have done this for chat products, customer support tools, and research assistants. The typical improvement is 60-75% latency reduction without changing the model.
Reach out via the AI engineering page with your current p95 latency and a sample trace if you have one. If you do not have traces, we will install observability first and audit second.
Top comments (0)