Percentiles in System Performance

Understanding Percentiles in System Performance

When evaluating the performance of an API or distributed system, looking only at the average response time can be misleading. A system may have a good average response time but still be extremely slow for many users.

This is why percentiles are an important concept in backend engineering and system design.

1. Why Average Response Time Is Not Enough

Imagine an API receiving 1,000 requests.

The response times might look like:

Most requests → 100 ms
Some requests → 500 ms
Some requests → 2 seconds
A few requests → 10 seconds

If we calculate the average and get:

Average = 200 ms

We might conclude that the API is fast.

However, the average does not tell us how slow the worst requests are. A small number of extremely slow requests can be hidden by a large number of fast requests.

This is where percentiles become useful.

1. P50: The Median

The 50th percentile (P50) is the same as the median.

Suppose we sort all requests from fastest to slowest:

Fast ------------------------------------ Slow

1 2 3 4 ... 50 "P50" ... 100

If:

P50 = 200 ms

it means that approximately:

50% of requests take 200 ms or less
50% take 200 ms or more

P50 therefore represents the experience of a typical user.

However, P50 alone is not enough to understand the overall performance of a system.

1. P95: Looking at Slower Requests

The 95th percentile (P95) tells us the response time below which approximately 95% of requests fall.

For example:

P95 = 1.5 seconds

means approximately:

95% of requests → ≤ 1.5 seconds
5% of requests → > 1.5 seconds

P95 is useful because it begins to reveal the slower experiences that P50 can hide.

1. P99: Looking at the Very Slow Requests

The 99th percentile (P99) goes even further.

If:

P99 = 3 seconds

then approximately:

99% of requests → ≤ 3 seconds
1% of requests → > 3 seconds

This 1% may sound insignificant, but at large scale it can represent a huge number of users.

For example, with:

1,000,000 requests

1% represents:

10,000 requests

So a small percentage can still have a significant real-world impact.

1. P99.9: The Extreme Tail

The 99.9th percentile (P99.9) focuses on an even smaller portion of the slowest requests.

For example:

P99.9 = 8 seconds

means approximately:

99.9% → ≤ 8 seconds
0.1% → > 8 seconds

With 10,000 requests, 0.1% is approximately:

10 requests

At very large scale, these "tiny" percentages can represent thousands or even millions of requests.

1. Why Tail Latency Matters

The slowest requests are often called tail latency.

Consider an e-commerce checkout API:

POST /checkout

Suppose its performance is:

P50 = 200 ms
P95 = 500 ms
P99 = 2 seconds
P99.9 = 8 seconds

If we only look at P50, we might say:

"Checkout is very fast."

But some users are waiting 8 seconds.

From the user's perspective, the experience may look like:

User clicks "Pay"
↓
Waiting...
↓
Waiting...
↓
"Is the website broken?"
↓
User leaves

Therefore, a backend engineer should not ask only:

"What is the average response time?"

A better question is:

"What are the P95 and P99 response times?"

1. Percentiles and User Experience

A useful way to remember the different percentiles is:

P50
↓
Typical user

P95
↓
Slower users

P99
↓
Very slow users

P99.9
↓
Extremely slow users

The higher the percentile, the more we are focusing on the tail of the latency distribution.

This is particularly important for systems with millions of requests, where even a very small percentage can affect a large number of users.

1. SLO and SLA

Percentiles are also closely related to SLOs and SLAs.

SLO — Service Level Objective

An SLO is an internal or operational performance target.

For example:

P99 < 1 second

This means the goal is for approximately 99% of requests to complete within one second.

SLA — Service Level Agreement

An SLA is a formal agreement with customers or clients about the level of service they can expect.

An SLA might define specific availability or performance requirements. If those requirements are not met, the agreement may specify consequences such as:

• Credits
• Refunds
• Penalties

depending on the contract.

1. Queueing Delay

Another important source of latency is queueing.

Imagine a server can process only:

10 requests simultaneously

but suddenly receives:

100 requests

The server cannot process all 100 at the same time.

Some requests must wait:

Request
↓
Queue
↓
Available capacity
↓
Processing
↓
Response

Even if the actual processing takes only 100 ms, the user may experience a much longer response time because the request spent time waiting in the queue.

This becomes especially problematic when a system operates close to its maximum capacity.

As utilization increases, queueing delay can increase significantly, which can cause P95, P99, and P99.9 latency to rise sharply.

1. Applying This to Backend Development

Imagine you have an API:

GET /products

and your monitoring shows:

P50 = 120 ms
P95 = 300 ms
P99 = 1.2 s
P99.9 = 5 s

Looking only at P50 gives you:

"The API is excellent — only 120 ms."

But P99.9 tells you:

"Some requests are taking 5 seconds."

Now you have a problem worth investigating.

Possible causes include:

• Slow database query?
• Missing database index?
• Too many database queries?
• External API?
• Redis?
• CPU saturation?
• Memory pressure?
• Queueing?

This is where performance monitoring becomes more than simply measuring average response time.

1. The Main Lesson

The most important idea is:

Don't look only at the average or median. Look at the slow requests.

You can think about it like this:

Average / P50
↓
Typical user experience

P95 / P99 / P99.9
↓
Tail user experience
↓
Problems hidden from the average

A system is not necessarily healthy just because its average response time is low.

Understanding tail latency, percentiles, and queueing helps you think about systems from the user's perspective and identify performance problems that simple averages hide.

Final Takeaway

The key concepts to remember are:

Concept Meaning
P50: Median/typical request
P95: Slower 5% of requests
P99: Slower 1% of requests
P99.9: Slowest 0.1% of requests
Tail latency: Latency of the slowest requests
SLO: Target performance level
SLA: Formal service commitment
Queueing: Time a request spends waiting for available capacity

This is one of the important shifts from being a developer who mainly focuses on writing code to becoming an engineer who thinks about how the entire system behaves under real-world load.