Your average response time is lying to you (use p95 and p99)

Your dashboard says average response time: 180 ms. Looks healthy. Meanwhile one in ten users waits almost a second. Here's why averages hide that, and what to use instead.

The math trick hiding in your dashboard

100 requests:

• 90 take 100 ms
• 10 take 900 ms

average = (90 × 100 + 10 × 900) / 100 = 180 ms

Nobody experienced 180 ms. Most people got a fast answer, some got a slow one, and the average describes neither group.

Now double the slow group to 20 requests: the average only moves to 260 ms. Still "fine" on a dashboard, while 1 in 5 users is having a bad time.

Percentiles say what users actually experience

A percentile answers: "how slow were the slowest X%?"

• p50 (median): half of requests were faster. The typical experience.
• p95: 1 in 20 was slower. Where people start to notice.
• p99: 1 in 100 was slower. Sounds rare, isn't (see below).

For the example above: p50 = 100 ms, p95 = 900 ms. Two numbers that tell the real story.

Why p99 matters more than it sounds

A single page often makes many requests: API calls, assets, fragments. If a page makes 20 requests, the chance that at least one hits the p99 is:

1 − 0.99²⁰ ≈ 18%

Nearly one page view in five includes a p99-slow request. At scale, the tail is the user experience.

Compute them from your own data

You don't need an APM to start. If you log request durations, SQL does it:

-- PostgreSQL
SELECT
  route,
  count(*)                                                   AS requests,
  round(avg(duration_ms))                                    AS avg_ms,
  percentile_cont(0.50) WITHIN GROUP (ORDER BY duration_ms)  AS p50_ms,
  percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms)  AS p95_ms,
  percentile_cont(0.99) WITHIN GROUP (ORDER BY duration_ms)  AS p99_ms
FROM request_log
WHERE created_at > now() - interval '1 hour'
GROUP BY route
ORDER BY p95_ms DESC;

Or in a few lines of JavaScript:

function percentile(values, p) {
  const sorted = [...values].sort((a, b) => a - b);
  const index = Math.ceil((p / 100) * sorted.length) - 1;
  return sorted[Math.max(0, index)];
}

const durations = [...Array(90).fill(100), ...Array(10).fill(900)];
console.log(percentile(durations, 50)); // 100
console.log(percentile(durations, 95)); // 900

⚠️ Never average percentiles across servers or time windows. The mean of two p95s is not a p95. Recompute from raw data, or use histograms that can be merged.

Why the tail moves first under load

As a system approaches capacity, the median often stays flat while the tail grows: requests start waiting in queues for a DB connection, a worker or a lock. A rising p95/p99 is usually the first warning that you're near the limit, well before the average or error rate changes.

Setting latency goals that work

• Use p95 as the main goal: "p95 < 500 ms".
• Add p99 for critical paths: login, checkout, search.
• Always pair it with an error-rate goal. A fast 500 is still a failure, and a test that "passes" latency because errors return instantly is not a pass.
• Set goals per endpoint: search can be allowed more than a page view.

In k6, that's:

export const options = {
  thresholds: {
    'http_req_duration{name:checkout}': ['p(95)<500', 'p(99)<1200'],
    http_req_failed: ['rate<0.01'],
  },
};

Compare runs, not single numbers

p95 = 420 ms alone means little. Against last week's identical run at 310 ms, it's a 35% regression worth catching before users do. Keep a baseline per test and compare every run with it.

TL;DR

• Averages blend fast and slow users into a number that describes nobody.
• Use p50 for typical, p95 for "noticeable", p99 for the tail.
• Never average percentiles.
• The tail rises first under load: it's your early warning.

---

I build Hoofbeat, which reports p50/p95/p99 per step and flags regressions against a baseline run. Original post on the Hoofbeat blog.