If your dashboard says p99 = 250ms and support says pages hang for seconds, the metric is usually wrong before the system is. Two mistakes cause most of it: computing a percentile per instance and then averaging those numbers, and using histogram buckets too coarse to represent your tail. Both fail in the same direction — they pull the reported number toward the middle and hide exactly the requests you care about.
This is a measurement problem, not a tuning problem. Fixing the query and the buckets usually changes the number more than any code change you were about to make.
Why does the dashboard show p99 = 250ms while users report multi-second waits?
There are three independent places the number can go wrong, and they stack:
- Aggregation: percentiles were computed per instance (or per minute) and then averaged.
- Resolution: the histogram's bucket boundaries are too coarse or top out too low, so the quantile is interpolated or clamped.
- Boundary: the timer starts after the slow part — queueing, TLS, request body upload, or an event loop that was blocked before your handler ever ran.
You can chase a "slow endpoint" for a day and find nothing, because the endpoint isn't slow; the wait happens somewhere your timer isn't watching. Before optimizing anything, confirm your latency metric can even represent the number users are experiencing.
The math: why averaging p99s is not a p99
Here's a constructed but entirely ordinary case. Three instances, 1,000 requests each in the window. Instances A and B serve everything in 50ms. Instance C — bad disk, noisy neighbor, a connection pool stuck on one shard — serves 900 requests at 50ms and 100 at 4,000ms.
| Metric | Value |
|---|---|
| p99 on instance A | 50ms |
| p99 on instance B | 50ms |
| p99 on instance C | 4,000ms |
| Average of the three p99s | ~1,367ms |
| True p99 across all 3,000 requests | 4,000ms |
The true p99 is 4,000ms: 100 of 3,000 requests are slow, which is more than 1%, so the 99th percentile lands squarely in the slow group. The averaged figure understates it by roughly 3x — and if you have twenty healthy instances instead of two, the average slides down toward 50ms while a twentieth of your users still wait four seconds.
Percentiles are not linear, so no arithmetic on them recovers the real one. You have to aggregate the underlying distribution first, then compute the quantile once. In Prometheus that distinction is one function call deep:
# Wrong: a quantile per instance, averaged afterwards
avg(histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])))
# Right: sum the bucket counters across instances, then take one quantile
histogram_quantile(
0.99,
sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)
The sum by (le) is the whole point: it merges the histograms, keeping the bucket boundaries as the grouping key. Anything that aggregates after histogram_quantile is reporting a number with no statistical meaning.
Takeaway: if a percentile appears inside avg(), max(), or a spreadsheet column of per-server values, the number on your dashboard is not a percentile.
How do histogram buckets decide what your p99 can even say?
A Prometheus-style histogram doesn't store latencies. It stores counters: how many requests were ≤ 5ms, ≤ 10ms, ≤ 25ms, and so on. histogram_quantile finds the bucket containing the quantile and interpolates linearly inside it, assuming observations are spread evenly across that bucket. They aren't. If your p99 falls in the [2.5s, 5s) bucket, the reported value can be off by a second or more purely from that assumption.
Worse is the top of the range. When the quantile falls into the +Inf bucket, Prometheus returns the highest finite bucket bound instead. That's why a p99 pinned at exactly 10 (or 5, or whatever your largest bucket is), flat for hours, is not a real measurement — it's the metric hitting the ceiling.
Pick buckets that bracket both your SLO and your realistic worst case. Exponential buckets cover a wide range cheaply:
httpDuration := prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
// 5ms doubling 12 times → top finite bucket ≈ 10.24s
Buckets: prometheus.ExponentialBuckets(0.005, 2, 12),
}, []string{"route", "method"})
The honest cost: every bucket is a time series, multiplied by every label combination. Twelve buckets across fifty routes and four methods is thousands of series from one metric, and route labels with IDs in them will bankrupt your storage. Keep the label set small and bounded, and put the high-cardinality detail in traces instead.
As of mid-2026, Prometheus native histograms — exponential buckets with automatic resolution, stored far more compactly — remove most of this bucket-choosing work, but they've been an opt-in feature rather than the default path, so check what your Prometheus and client library versions actually support before you design around them.
Takeaway: a flat p99 sitting exactly on a round number is almost always your top bucket, not your service.
Where should the timer start?
Handler-level instrumentation measures the part of the request your application already knows about. It cannot see time spent in the kernel accept queue, in the load balancer's connection pool, or waiting for a runtime that was busy elsewhere.
Node is the clearest example: if the event loop is blocked for 800ms by a synchronous JSON parse, requests that arrived during that window get their timers started after the block clears. The handler honestly reports 20ms. The user waited 820ms. Measure the loop itself to catch it:
const { monitorEventLoopDelay } = require('node:perf_hooks');
const h = monitorEventLoopDelay({ resolution: 10 });
h.enable();
setInterval(() => {
// percentile() returns nanoseconds
console.log('event loop delay p99 (ms):', h.percentile(99) / 1e6);
h.reset();
}, 10_000).unref();
The same gap exists in load testing. A closed-loop generator that waits for each response before sending the next one stops applying pressure exactly when the system stalls, so the stall never appears in the results — the coordinated omission problem. Tools designed around it, like wrk2, hold a constant request rate and account for the requests they should have sent.
Takeaway: instrument at the edge and inside the app; the difference between the two numbers is the queueing your handler metrics will never show.
Which aggregation approach should you use?
| Approach | Aggregates correctly across instances | Arbitrary percentiles after the fact | Main drawback |
|---|---|---|---|
| Per-instance percentile, averaged | No | No | Statistically meaningless |
| Client-side summary quantiles | No | No | Cannot be merged across hosts |
| Explicit-bucket histogram | Yes | Only to bucket resolution | Bucket choice is baked in; series cardinality |
| Sketch-based distribution metrics | Yes | Yes, within error bounds | Vendor-specific storage and billing |
| Raw events or traces | Yes | Yes, exactly | Sampling design and cost at volume |
If you want histograms without choosing bucket boundaries by hand, Datadog's distribution metric type aggregates sketches server-side so global percentiles stay correct across every host reporting the metric — with the caveat that percentile queries on distributions are billed separately from the base metric. If you'd rather keep raw events and decide the percentile and grouping at query time, Honeycomb stores wide events instead of pre-aggregated buckets, which means the tail is still there to slice, though you take on designing a sampling strategy once traffic grows. For self-hosted stacks, Prometheus with correctly summed buckets plus trace exemplars gets you most of the way at no license cost, provided you accept the bucket resolution you chose up front.
FAQ
Can you average percentiles across servers?
No. A percentile is a position in a distribution, not a quantity, so the average of several p99s has no defined meaning. Merge the underlying histograms or raw observations first, then compute one percentile over the combined data.
Why is my p99 latency always exactly 10 seconds?
Because the 99th percentile has fallen into your +Inf bucket and Prometheus is returning the highest finite bucket bound. Add larger buckets — or switch to native histograms — so the tail has somewhere to land.
What histogram buckets should I use for HTTP request latency?
Use exponential buckets spanning from below your fastest realistic response to above your worst acceptable timeout, and make sure one boundary sits exactly on your SLO threshold so you can compute SLO compliance directly from the bucket counter.
Bottom line
Fix aggregation first: put the quantile function outermost and sum buckets by le underneath it. Then check your bucket range — if the reported p99 is a flat, round number, you're reading the ceiling, not the latency. If you're on a self-hosted stack, correctly summed Prometheus histograms are enough for SLO work; if you need to ask questions you didn't anticipate when you defined the metric, keep raw events somewhere and pay for the storage instead of the guessing. Whatever you choose, compare an edge-level timer against the in-process one at least once, because the gap between them is the latency your users feel and your dashboard never showed.
Top comments (0)