The problem
A service humming along at 40ms average latency. CPU sits at 60%. Dashboards green. Then the pager goes off: p99 is 900ms, and it's been that way for weeks — nobody noticed because the average never moved. On-call pulls up flame graphs, finds nothing pathological, and closes the ticket as "noise."
It isn't noise. It's queueing.
Why it happens
Average latency measures the common case. Tail latency measures what happens when several unlikely things line up at once — and unlikely things line up more often than intuition suggests. Two mechanisms do almost all the damage.
1. Utilization is not linear with latency. For a simple queue (the M/M/1 approximation), expected wait time scales roughly with ρ/(1-ρ), where ρ is utilization. At 50% utilization, that multiplier is 1x. At 80%, it's 4x. At 95%, it's 19x. At 99%, it's 99x. The curve stays flat for a long time and then goes vertical. A service that "looks fine" at 60% CPU isn't 60% away from saturation — it's most of the way up a curve that's about to explode.
2. Fan-out amplifies tails. If a request fans out to 20 backend calls and waits on all of them, your response time is the max of 20 samples, not the average of 20. Even if every backend has a p99 of 100ms (99% chance of finishing under 100ms), the odds that all 20 finish under 100ms is 0.99^20 ≈ 82%. Close to 1 in 5 requests gets dragged down by whichever one of the twenty had a bad moment. That's not a bug — it's arithmetic. Add naive retries on top and you can amplify load at exactly the moment a dependency is already struggling.
The two mechanisms compound: fan-out multiplies your exposure to tail events, and rising utilization on any one dependency makes tail events on that dependency more frequent.
What to do about it
Stop planning capacity around average utilization. Plan around the utilization at which queueing delay becomes unacceptable, then keep steady-state usage well under that line — most latency-sensitive paths want something like 65-75%, not 90%+.
Bound fan-out instead of letting it grow. Every extra parallel call you wait on is another chance to hit someone else's bad millisecond. If you don't need all 20 results, don't wait for all 20.
Use hedged (backup) requests instead of blind retries. Fire a second request to a different replica if the first hasn't returned by roughly your own p95, and take whichever comes back first, cancelling the loser. This trims the tail without the pile-on effect of retry-on-timeout, which just adds load to an already-struggling dependency.
Add jitter to timeouts and retries. Synchronized retries across many clients are how a small blip becomes a cascading outage — everyone times out at the same millisecond and hits the backend at once.
Isolate noisy neighbors. A single slow dependency behind a shared thread or connection pool can drag down requests that never even touch it, because everything is waiting in the same queue for a free worker. Bulkhead pools per-dependency.
Key takeaways
- Average and tail latency are different problems with different causes. A flat average tells you nothing about how close you are to the queueing cliff.
- Utilization near saturation makes latency blow up nonlinearly — plan capacity against that curve, not against "CPU still has headroom."
- Fan-out to N parallel calls means your latency is governed by the worst of N, not the average of N. Bound N wherever you can.
- Hedged requests with jittered timeouts beat blind retries for taming tails without triggering retry storms.
- If your p99 is bad and your average looks fine, check utilization and fan-out width before you check code.
Top comments (0)