Average response time is the wrong number to optimize for AI agents because it hides exactly the requests that break trust: the slow tool call, the retried LLM step, the request that timed out and silently fell back. Track p95 and p99 latency per step instead, and ask any vendor for the same before you sign a contract.
The problem with averaging
Say your agent responds in 800ms on average. That sounds fine. But if 90% of requests finish in 400ms and the remaining 10% take 6 seconds because they hit a retry, a rate limit, or a slow downstream API, the average buries the part of the distribution your users actually feel. Nobody experiences the average. They experience their own request, and for one in ten users, that request is 15x slower than what your dashboard implies.
This is not a new idea in distributed systems generally, it's why the Google SRE book treats percentile latency (p50, p95, p99) as the standard for monitoring, not the mean. AI agents make this worse than typical web services because the tail is fatter: LLM inference time is itself variable, tool calls can hang, and agents often chain multiple calls where one slow link stalls the whole request.
Where the tail comes from in agent systems
A single well-scoped LLM call has one source of latency variance: the model's inference time, plus network. An agent that chains steps, retrieve context, call a tool, call the model again, validate output, call the model a third time, multiplies that variance at every hop. Each step also carries its own failure and retry probability, and retries don't just add latency, they add it unevenly. A request that needs one retry on step 3 might take twice as long as one that sails through, and that unevenness is exactly what averages erase.
We saw this directly building our own outreach tool, which scrapes a prospect's site and drafts a tailored email. Early versions used a three-step chain: extract facts from the scraped page, draft the email, then refine it. The extraction step was the one at the mercy of someone else's server, some prospect sites loaded fast, some were slow or bloated with tracking scripts, and however long that page took to fetch and parse became a floor under everything after it, since drafting couldn't start until extraction finished. The average looked fine because most prospect sites are fast. The tail was rough because the slow sites weren't rare, they were just unevenly distributed, and each one produced a full-length stall that a fast average never showed. Collapsing extraction and drafting into a single well-designed call didn't make individual sites load faster, but it removed the sequential dependency, there was one less handoff for a slow fetch to block. If you're deciding between an agent chain and a single call for your own product, that tradeoff is worth thinking through before you build, see our breakdown of single-call vs agent chains.
What to measure instead
For any AI agent, whether you're building it or evaluating a vendor's, ask for these numbers broken out by step, not just for the request as a whole:
- p50 (median): what a typical request feels like.
- p95: what one in twenty users experiences. This is usually where the real product complaints start.
- p99: your worst-case tail. For a support bot handling thousands of conversations a day, p99 is not an edge case, it's dozens of real conversations every day.
- Time to first token vs. total completion time: if the agent streams output, users tolerate a slower total time much better than a slow start. Measure both separately.
- Retry rate and where retries happen: a 2% retry rate on one step sounds small until it's the step every request depends on.
Tools like OpenTelemetry exist specifically so you can trace latency through each step of a distributed call, including agent chains, rather than only seeing the total. If your current stack (or a vendor's) can't show you per-step traces, that's itself a useful data point.
How this shows up in vendor conversations
If you're evaluating an outside team to build or operate an AI agent for you, latency claims are one of the easiest places for a pitch to overstate reality. "Sub-second responses" is a claim about the average, almost always, and it's exactly the kind of number that hid the stall in our own outreach tool until we broke it down by step. The right follow-up questions are:
- What's the p95 and p99 under realistic concurrent load, not a single warm request in a demo?
- Which step in the pipeline is slowest, and how does that change under load?
- What happens when a downstream call (a database, a search index, another API, or in our case a prospect's own website) is slow? Does the agent degrade gracefully or hang?
These questions belong in the same conversation as pricing and support terms. We cover the rest of that evaluation, including questions that have nothing to do with speed, in our AI agent vendor evaluation checklist.
Latency and cost are the same conversation
Worth noting: the same chain that produces a bad p99 usually also produces a bad bill. Extra steps mean extra tokens, extra retries mean paying twice for the same work, and extra model calls mean paying for coordination overhead that adds no value to the output. If you're already auditing latency, audit spend at the same time. Our guide on LLM cost optimization walks through the same trimming exercise from the cost side, and in practice the fixes overlap: fewer, better-scoped calls beat more, smaller ones on both dimensions.
The practical takeaway
Don't let "it feels fast in the demo" or "average response time: 800ms" stand in for real measurement. Before you ship an agent, or before you sign off on one someone else built for you, get the p95 and p99 numbers under load, broken down by step. If nobody can produce those numbers, that's your answer about how much testing has actually happened.
If you're building an AI agent and want a second set of eyes on the architecture before latency becomes a production problem, let's talk.
Originally published on the Pykero blog.
Top comments (0)