DEV Community: Subeg Singh

Why 99% Fast Can Still Feel Completely Broken. That's where averages lie and percentile dont.

Subeg Singh — Fri, 07 Aug 2026 19:14:36 +0000

Ten people load the same webpage. Nine of them see it in about 12 milliseconds, basically instant. The tenth person hits a slow server and waits 4 full seconds.

Now someone asks: "So, how fast is our page?"

You do what feels obvious. You average the ten numbers.

410.6 milliseconds.

Write that in a report and it looks fine, even respectable. But sit with it for a second. Whose experience does 410ms actually describe? Not the nine people who got 12ms. Not the one person who got 4000ms. It describes a person who doesn't exist. One outlier dragged a "typical" number to 30 to 40 times higher than what 90% of your users actually felt.

That's the problem I want to talk about, and it's hiding in plain sight in most dashboards.

Why the average betrays you

The average works fine when data is evenly spread out, things Latency almost never looks like that. It's skewed: a big, dense pile of fast requests, then a long thin tail of slow ones stretching out to the right. A server hiccups, a connection pool runs dry, a cache misses, and now you've got outliers that are 100x, 300x, sometimes 1000x worse than normal.

The average lets those outliers hijack the whole summary. One bad request can outweigh a hundred good ones in the final number. If you're making decisions off the average, you're making decisions based on something that isn't real.

So what's the honest alternative?

Percentiles: sort it, then ask "where does X% end?"

A percentile asks a different question than an average does. Instead of "what's the mean of all these numbers," it asks:

If I sort every result from fastest to slowest, what value sits X% of the way through?

P50 (the median): the middle value. Half your requests were
P90: 90% of requests were at or below this. Only the worst 10% were slower.
P99: 99% were this fast or faster. Only the unluckiest 1% saw worse.

The math is simple enough that you don't need a stats course for it:

rank = (P / 100) × n

Then you take the value sitting at that rank in your sorted list.

Let's run it on our ten webpage loads, sorted:

10, 10, 11, 11, 12, 12, 12, 13, 15, 4000

P50: rank = 0.50 × 10 = 5, so average the 5th and 6th values: (12+12)/2 = 12ms
P90: rank = 0.90 × 10 = 9, so the 9th value: 15ms
P99: rank = 0.99 × 10 = 9.9, which lands basically on the la

Look at what just happened. Instead of one number papering over the situation, you now get two honest statements. A typical user sees 12ms, and 90% of users see under 15ms: the experience is genuinely good. Meanwhile, the worst 1% of requests are getting hit with 4-second waits: there's a real problem to go fix.

Nothing gets hidden and nothing gets averaged away. The good news and the bad news stay as two separate, true statements instead of blending into one misleading one.

Here's roughly what that distribution looks like if you tried to sketch it:

Latency distribution, 10 requests

0-20ms      █████████  (9 requests)   what "typical" actually feels like
   .              (a long, empty gap, nothing happens here)
   .
3980-4000ms █          (1 request)    the tail: small in count

That gap in the middle is the whole story. The average pretends it isn't there.

"Percentage" alone doesn't mean anything

This is worth pausing on, because people mix the words up constantly. "Percentage" by itself is meaningless: percentage of what? A percentile is a percentage paired with a value, and that pairing is exactly what makes it a complete statement. "P99 = 500ms" tells you precisely that 99% of requests finish in under 500ms. That's why engineering teams write SLAs like:

P99 latency must stay under 300ms.

That's a promise about the worst realistic case most users could hit. Compare it to "average latency is 50ms," which tells you nothing about how bad the
bad days get.

The deeper reason the tail matters

This is the part that changed how I think about it, and it's just middle-school probability.

Say a webpage needs 100 small backend calls to fully render. That's not unusual at all, think of a page pulling in dozens of tiny images, widgets, or API responses. Suppose each individual call is fast 99% of the time. Sounds great. P99 is "good."

What's the chance all 100 calls come back fast, so the whole p

0.99^100 ≈ 0.366

36.6%. Even though every single piece is "good" 99% of the time, the page as a whole is slow almost two out of three times, purely because it depends on
so many moving parts, and any one slow piece drags the whole e

Watch how fast this compounds as the number of dependent calls grows:

Number of dependent calls	Chance the whole thing is fast
1	99.0%
10	90.4%
50	60.5%
100	36.6%
200	13.4%

This is called tail latency amplification, and it's the real reason companies operating at Google or Amazon scale obsess over P99 and even P99.9 instead of the average. At that scale, the "rare" bad case stops being rare. It becomes the default experience of the overall system, because there are so many chances for something in the chain to be the one unlucky piece.

Quick summary

Metric	What it tells you	Weakness
Average	The mathematical mean of all values	Easily distos, describes no real user
Percentage (alone)	Nothing, without a paired value	Not a complete statement by itself
Percentile (P50, P90, P99...)	The value below which X% of results fall	Requires sorting the data, more work but far more honest

The one-sentence takeaway

The average tells you a number nobody experienced. Percentiles tell you what percentage of real people had a good experience, and exactly how bad it gets for the unlucky few. That second part is what actually decides how a system feels to use.

I keep coming back to this: the average isn't wrong because the math is wrong. It's wrong because it answers a question nobody actually asked. Nobody wants to know the mean of all outcomes. They want to know what their own experience will probably be, and how bad it can realistically get. Percentiles
answer that. Averages just sound like they do.

Next time someone hands you a dashboard with a single "avg lathe P99 next to it. That one extra number is usually where thereal story is hiding.

Do you track P99 or P99.9 in your own systems, or has "average latency" been quietly misleading your team too? I'd genuinely like to hear your version of the 4000ms outlier in the comments.

If this was useful, I write about problem-solving, AI/ML, and the thinking behind the tools we build with. Feel free to connect with me on LinkedIn.

The Greed of Low Bias and Low Variance

Subeg Singh — Fri, 19 Jun 2026 18:04:03 +0000

Every model is a little bit greedy

I will start with a example of two archers. The first one always aims slightly left of the bullseye i.e. to the left of the centre.
Every single arrow, same mistake, same direction. The second one aims dead
center on average, but for both of them, their hand shakes, so the arrows land all over the board.

Neither archer is "bad". They're both being greedy in their own way. The first archer is greedy for consistency. He has locked onto one strategy and won't budge, even though it's wrong. He just wants the result with same input even though it is wrong. The second archer is greedy for flexibility. He will react to every gust of wind, every twitch, every tiny signal, even the ones that don't matter.

That's bias and variance. The maddening part of machine learning is that you
can't fix both at once. Reduce one, and the other usually grows. Once you
understand why it happens, and how we actually measure it, it stops being a
vague textbook warning and starts feeling like common sense.

Bias: the model that won't change its mind

Bias is systematic error. It's what happens when your model is too simple to capture the real pattern in the data. Think a straight line trying to fit a curve. No matter how much data you throw at it, it keeps making the same kind of mistake, because the shape it's allowed to take is fundamentally wrong.
High-bias models underfit: they're stable, but stably wrong.

Variance: the model that overreacts

Variance is instability. It's what happens when your model is so flexible that it doesn't just learn the underlying pattern, it learns the noise too. Train it on one sample of data and it draws one curve. Train it on a slightly different sample and it draws a wildly different curve. High-variance models overfit: they're accurate on the data they've seen, but unreliable everywhere else.

So how do you actually measure variance?

"How much does my model swing around" sounds vague until you turn it into a
concrete procedure. The trick: fix the test point, vary the training data.

Take your training set and resample it. The standard way is bootstrapping, where you randomly draw examples with replacement to create a new training set of the same size. Do this B = 100 times and you get 100 slightly different training sets, all pulled from the same pool.
Train your model fresh on each of those 100 training sets. Same algorithm, same hyperparameters. The only thing that changes is which examples it saw.
Now take one single test point, something none of those models trained on, and ask all 100 models to predict it.
You now have 100 different predictions for the exact same input. The spread of those 100 numbers is your variance.

Note what stayed constant: the test point. If you let the test data change too, like in plain k-fold cross-validation, you're no longer isolating variance.
You're mixing it with "this fold happened to be harder." Keeping the test point fixed is what makes the isolation real.

Why do we square the deviations?

Once you have your 100 predictions and their average, why not measure "how far
off" each prediction is with plain distance instead of squaring it?

The first problem is cancellation. A prediction that's +5 above the average
and one that's -5 below cancel out to zero when summed, even though both
represent real instability. Squaring makes every deviation positive, so errors in opposite directions can't hide each other.

The second is that squaring punishes big swings harder. A model that's
occasionally wildly wrong is more dangerous than one that's consistently a little off. Squaring grows faster than the deviation itself, so an error of 10 contributes 100x more than an error of 1. That's by design. You want the measurement to be sensitive to extreme instability, not just average shakiness.

Third, it matches the actual statistical definition. Variance, in statistics, is the expected squared deviation from the mean. This isn't something machine learning invented. It's the same definition you'd use to describe how spread out any dataset is, so using it here isn't a choice, it's just staying
consistent.

And fourth, squared functions are smooth and differentiable everywhere, which
matters once you start optimizing models. This is also why the famous identity
Total Error = Bias² + Variance + Irreducible Noise uses squared bias. The decomposition only works cleanly when both terms are squared quantities.

The actual goal

You're not trying to make bias zero or variance zero. That target doesn't exist outside of toy examples with infinite noiseless data. You're looking for the point where their combined greed costs you the least: a model flexible enough to be roughly right, but stable enough not to chase noise.

That means giving up a little of each archer's stubbornness. Add just enough
flexibility to fix the systematic miss, and just enough constraint to steady the shaky hand. Regularization, more training data, ensembling are all different tools for the same goal: tell your model to be less greedy.