Ten people load the same webpage. Nine of them see it in about 12 milliseconds, basically instant. The tenth person hits a slow server and waits 4 full seconds.
Now someone asks: "So, how fast is our page?"
You do what feels obvious. You average the ten numbers.
410.6 milliseconds.
Write that in a report and it looks fine, even respectable. But sit with it for a second. Whose experience does 410ms actually describe? Not the nine people who got 12ms. Not the one person who got 4000ms. It describes a person who doesn't exist. One outlier dragged a "typical" number to 30 to 40 times higher than what 90% of your users actually felt.
That's the problem I want to talk about, and it's hiding in plain sight in most dashboards.
Why the average betrays you
The average works fine when data is evenly spread out, things Latency almost never looks like that. It's skewed: a big, dense pile of fast requests, then a long thin tail of slow ones stretching out to the right. A server hiccups, a connection pool runs dry, a cache misses, and now you've got outliers that are 100x, 300x, sometimes 1000x worse than normal.
The average lets those outliers hijack the whole summary. One bad request can outweigh a hundred good ones in the final number. If you're making decisions off the average, you're making decisions based on something that isn't real.
So what's the honest alternative?
Percentiles: sort it, then ask "where does X% end?"
A percentile asks a different question than an average does. Instead of "what's the mean of all these numbers," it asks:
If I sort every result from fastest to slowest, what value sits X% of the way through?
- P50 (the median): the middle value. Half your requests were
- P90: 90% of requests were at or below this. Only the worst 10% were slower.
- P99: 99% were this fast or faster. Only the unluckiest 1% saw worse.
The math is simple enough that you don't need a stats course for it:
rank = (P / 100) × n
Then you take the value sitting at that rank in your sorted list.
Let's run it on our ten webpage loads, sorted:
10, 10, 11, 11, 12, 12, 12, 13, 15, 4000
- P50: rank = 0.50 × 10 = 5, so average the 5th and 6th values: (12+12)/2 = 12ms
- P90: rank = 0.90 × 10 = 9, so the 9th value: 15ms
- P99: rank = 0.99 × 10 = 9.9, which lands basically on the la
Look at what just happened. Instead of one number papering over the situation, you now get two honest statements. A typical user sees 12ms, and 90% of users see under 15ms: the experience is genuinely good. Meanwhile, the worst 1% of requests are getting hit with 4-second waits: there's a real problem to go fix.
Nothing gets hidden and nothing gets averaged away. The good news and the bad news stay as two separate, true statements instead of blending into one misleading one.
Here's roughly what that distribution looks like if you tried to sketch it:
Latency distribution, 10 requests
0-20ms █████████ (9 requests) what "typical" actually feels like
. (a long, empty gap, nothing happens here)
.
3980-4000ms █ (1 request) the tail: small in count
That gap in the middle is the whole story. The average pretends it isn't there.
"Percentage" alone doesn't mean anything
This is worth pausing on, because people mix the words up constantly. "Percentage" by itself is meaningless: percentage of what? A percentile is a percentage paired with a value, and that pairing is exactly what makes it a complete statement. "P99 = 500ms" tells you precisely that 99% of requests finish in under 500ms. That's why engineering teams write SLAs like:
P99 latency must stay under 300ms.
That's a promise about the worst realistic case most users could hit. Compare it to "average latency is 50ms," which tells you nothing about how bad the
bad days get.
The deeper reason the tail matters
This is the part that changed how I think about it, and it's just middle-school probability.
Say a webpage needs 100 small backend calls to fully render. That's not unusual at all, think of a page pulling in dozens of tiny images, widgets, or API responses. Suppose each individual call is fast 99% of the time. Sounds great. P99 is "good."
What's the chance all 100 calls come back fast, so the whole p
0.99^100 ≈ 0.366
36.6%. Even though every single piece is "good" 99% of the time, the page as a whole is slow almost two out of three times, purely because it depends on
so many moving parts, and any one slow piece drags the whole e
Watch how fast this compounds as the number of dependent calls grows:
| Number of dependent calls | Chance the whole thing is fast |
|---|---|
| 1 | 99.0% |
| 10 | 90.4% |
| 50 | 60.5% |
| 100 | 36.6% |
| 200 | 13.4% |
This is called tail latency amplification, and it's the real reason companies operating at Google or Amazon scale obsess over P99 and even P99.9 instead of the average. At that scale, the "rare" bad case stops being rare. It becomes the default experience of the overall system, because there are so many chances for something in the chain to be the one unlucky piece.
Quick summary
| Metric | What it tells you | Weakness |
|---|---|---|
| Average | The mathematical mean of all values | Easily distos, describes no real user |
| Percentage (alone) | Nothing, without a paired value | Not a complete statement by itself |
| Percentile (P50, P90, P99...) | The value below which X% of results fall | Requires sorting the data, more work but far more honest |
The one-sentence takeaway
The average tells you a number nobody experienced. Percentiles tell you what percentage of real people had a good experience, and exactly how bad it gets for the unlucky few. That second part is what actually decides how a system feels to use.
I keep coming back to this: the average isn't wrong because the math is wrong. It's wrong because it answers a question nobody actually asked. Nobody wants to know the mean of all outcomes. They want to know what their own experience will probably be, and how bad it can realistically get. Percentiles
answer that. Averages just sound like they do.
Next time someone hands you a dashboard with a single "avg lathe P99 next to it. That one extra number is usually where thereal story is hiding.
Do you track P99 or P99.9 in your own systems, or has "average latency" been quietly misleading your team too? I'd genuinely like to hear your version of the 4000ms outlier in the comments.
If this was useful, I write about problem-solving, AI/ML, and the thinking behind the tools we build with. Feel free to connect with me on LinkedIn.
Top comments (0)