Your dashboard says the average response time is 200 ms. Everything looks healthy. But users keep saying, "the app sometimes freezes."
Both things can be true at once. The dashboard isn't broken. The problem is that the average hides the users who are having a bad time.
In this article, we'll learn how to see those users using p50, p95 and p99. We'll start from the basics and build up step by step.
1. What is latency?
Latency is the time a request takes to get a response. You tap a button, the app asks the server for data, and the server replies. The waiting time in between is latency, usually measured in milliseconds (ms).
Lower latency means a faster app. But how do we measure it for thousands of users at once? The obvious answer is the average. Let's see why that goes wrong.
2. Why the average is misleading
Say your app handled 100 requests:
- 90 requests took 100 ms
- 8 requests took 500 ms
- 2 requests took 5,000 ms (5 seconds)
The average comes out to 230 ms, which sounds fine. But look at what really happened: 10 users had a slow experience, and 2 of them stared at a loading spinner for 5 seconds.
The fast majority pulled the average down and covered up the unlucky few. We need a way to look at the slow requests directly. That's where percentiles come in.
3. What is a percentile?
Imagine all 100 requests lined up from fastest to slowest. A percentile simply tells you how fast the request at a certain position in that line was.
For example, if p90 is 300 ms, it means 90% of requests finished in 300 ms or less.
Now let's look at the ones you'll see most often.
p50: the typical experience
p50 is also called the median. Half of the requests are faster than this, and half are slower.
In our example, p50 is 100 ms. This is what a typical user experiences.
p95: the slower experience
p95 means 95% of requests are faster than this value. Only 5% are slower, which is 1 in every 20 requests.
In our example, p95 is 500 ms. So 1 in 20 users waits this long or more.
p99: the worst-case experience
p99 means 99% of requests are faster than this value. Only 1% are slower, which is 1 in every 100 requests.
In our example, p99 is 5,000 ms. The unluckiest 1% of users wait 5 seconds or more.
p99.9 and beyond
p99.9 means 99.9% of requests are faster, so only 1 in 1,000 is slower. Large companies track this too. If you serve millions of requests a day, 1 in 1,000 still means thousands of unhappy users.
All together
| Metric | Value | What it tells us |
|---|---|---|
| Average | 230 ms | Looks fine, but hides the problem |
| p50 | 100 ms | The typical user is happy |
| p95 | 500 ms | 1 in 20 users waits 5x longer |
| p99 | 5,000 ms | 1 in 100 users waits 50x longer |
Now we see the real picture. The slow end of this list has a name: tail latency. When engineers say "we need to fix our tail," they mean the p99 and p99.9 requests.
4. "But it's only 1% of users. Why care?"
It's a fair question. There are two good answers.
Answer 1: Your slowest users are often your best users
In the book Designing Data-Intensive Applications, Martin Kleppmann shares an example from Amazon. Amazon describes the performance requirements of its internal services using the 99.9th percentile, even though that affects only 1 in 1,000 requests.
Why? Because the customers with the slowest requests are usually the ones with the most data on their accounts: the longest order history and the most purchases. In other words, they are often Amazon's most valuable customers.
Amazon has also observed that even a 100 ms slowdown can reduce sales by around 1%. For a company that size, tiny delays cost real money.
Answer 2: Small problems add up
When you open one page of a modern app, it doesn't make just one request. Behind the scenes, it may call dozens of services for your profile, prices, recommendations, stock availability, and more.
Suppose each service is slow only 1% of the time. That feels harmless. But if one page needs 100 calls, the chance that at least one of them is slow is:
1 - (0.99)^100 ≈ 63%
So a "rare" 1% problem becomes a problem for almost two out of three page loads. This is called tail latency amplification. The more services you have, the more your p99 starts to feel like your users' normal experience.
5. Why do some requests become slow?
If your code is the same for every request, why are a few so much slower? Usually the cause is something outside your business logic:
- Garbage collection pauses that briefly freeze a program
- Cache misses, where data has to be fetched from a slow database
- Lock contention, where many requests fight over the same resource
- Network hiccups like packet loss and retries
- Queueing, where a few slow requests make faster ones wait behind them
- Busy neighbors, where other apps on the same server use up the resources
The key point is that these problems are random and rare. That's exactly why they show up in p99 and not in the average.
6. How companies use these metrics
Teams turn percentiles into clear goals, called SLOs (Service Level Objectives). For example:
"99% of checkout requests should finish in under 300 ms."
This is much more useful than "average latency should be low," because it describes what real users feel, and everyone can measure it.
A simple way to remember which metric answers which question:
- p50: is the app healthy in general?
- p95: is a noticeable group of users struggling?
- p99 / p99.9: are there hidden problems like slow queries or overloaded servers?
Companies often go one step further and write these goals into contracts with customers. These are called SLAs, and breaking them can mean refunds or penalties.
7. Common mistakes when measuring
Once you start tracking percentiles, avoid these traps:
- Don't average percentiles. The average of the p99 from 10 servers is not your real p99. Use histograms instead, which tools like Prometheus, Datadog and OpenTelemetry support.
- Don't chase perfection. Improving p99.99 gets very expensive. Amazon reportedly decided it wasn't worth the cost. Pick the percentile that fits your business.
- Use enough data. A p99 calculated from 50 requests isn't reliable. You need enough traffic for the number to be stable.
Summary
- Latency is how long a request takes.
- The average hides slow requests, so we use percentiles instead.
- p50 is the typical experience, p95 shows slower users, and p99 / p99.9 shows the worst cases.
- The slow end is called tail latency, and it often affects your most valuable users.
- In systems with many services, rare slowness becomes common.
- Companies set SLOs on percentiles, not averages.
Next time someone says "our average latency is fine," ask them:
"What's our p99?"
Top comments (0)