48-Hour Field Notes: My p95 Looked Healthy Because I Averaged Four Percentiles

The number that cost me two days

Have you ever shipped a change that made a dashboard look better and a user feel worse? That is roughly how this week started for me. The export screen took several seconds to answer, while the latency panel I check every morning sat comfortably inside its target. I spent the first six hours blaming the export code and the next six blaming the panel, and only one of those instincts turned out to be wrong.

The panel was not lying, which is the uncomfortable part of the story. It was answering a question about the whole service, and I kept reading it as an answer about one screen. Once I accepted that distinction, the real work started: finding out which of my aggregation shortcuts had quietly made that panel useless for the question I actually had.

What the panel was showing, and what I was asking it

Every request in that service carries a route label, but only four of them matter for latency. Three are high-volume API routes, and one is an export route that runs a few hundred times a day and occasionally takes seconds. The panel merged all four into a single traffic-weighted p95, which is a perfectly reasonable thing for a global SLO and a terrible thing for a question like why is export slow. Export is roughly two percent of requests, so it cannot move a global p95 by much, no matter how badly it behaves.

Does that make the panel wrong? No. It means I had built one number for two audiences and then acted surprised when it served neither of them. The fix I needed was per-route visibility. The fix I shipped first, at around 1 a.m., made everything worse.

Then I made it worse by averaging percentiles

My first patch was the kind of code that looks obvious when you are tired. I had per-shard p95 values from four workers, so I averaged them, weighting each shard by its request count, and plotted the result as a service-level p95. It moved when traffic moved, it looked plausible on a chart, and it was not the p95 of anything at all. The average of percentiles is not a percentile, and the gap is not a rounding difference you can wave away.

Here is the harness I wrote to settle the argument with a colleague, and then with myself. It uses only the standard library, so it runs anywhere you have Python 3.10 or newer.

# merge_check.py -- stdlib only
import math
import random
from dataclasses import dataclass

random.seed(7)

@dataclass
class Shard:
    name: str
    samples: list[float]

def synth(name: str, n: int, base: float, tail_rate: float) -> Shard:
    out = []
    for _ in range(n):
        if random.random() < tail_rate:
            out.append(random.gauss(base * 6, base * 2))   # slow tail
        else:
            out.append(random.gauss(base, base * 0.2))    # body
    return Shard(name, [max(v, 0.1) for v in out])

def pct(values: list[float], q: float) -> float:
    # nearest-rank percentile, q in (0, 100], no interpolation
    if not values:
        return float('nan')
    s = sorted(values)
    k = max(1, math.ceil(q / 100 * len(s)))
    return s[k - 1]

shards = [
    synth('edge-a', 4000, base=12, tail_rate=0.02),
    synth('edge-b', 4000, base=13, tail_rate=0.02),
    synth('api',    9000, base=20, tail_rate=0.01),
    synth('export',  300, base=400, tail_rate=0.15),  # small and slow
]

def naive_weighted(shards, q):
    total = sum(len(s.samples) for s in shards)
    return sum(pct(s.samples, q) * len(s.samples) for s in shards) / total

def merged(shards, q):
    return pct([v for s in shards for v in s.samples], q)

for q in (50, 95, 99):
    print(f'p{q:<3} naive={naive_weighted(shards, q):9.2f} merged={merged(shards, q):9.2f}')

# the accident I actually had: one small shard missing from the merge
subset = [s for s in shards if s.name != 'export']
for q in (95, 99):
    print(f'p{q:<3} without export={merged(subset, q):9.2f}')

Run it with python3 merge_check.py and read the columns, not the absolute numbers. The exact values depend on the seed and on the traffic shapes I picked, but there are three things to look for, and all three held for every seed I tried.

• The naive weighted p95 and the true merged p95 disagree, and the size of the disagreement grows with how different the shards are.
• Dropping the small slow shard changes the naive number a lot and the merged number barely at all, because 300 requests out of 17,300 cannot move a rank.
• The direction of the naive error is not a law of nature. In this mix it runs high; flip the shape, with many small fast shards and one large slow shard, and the same code errs low. The sign follows your traffic mix, not your math.

That last bullet is the part I keep repeating to people. If an approximation is wrong in a direction you cannot predict, you cannot correct for it later with a fudge factor. You can only stop calling it a percentile.

Histograms do not save you, they just fail differently

After the averaging patch came the obvious second idea: keep histograms instead of raw samples, since histograms merge by adding bucket counts. That is true, and it is why most metric backends work this way. It also introduces a failure mode that returns a confident number with no warning, which is worse than an error.

BUCKETS = [5, 10, 25, 50, 100, 250, 500, 1000]

def histogram(samples):
    counts = [0] * (len(BUCKETS) + 1)   # last slot is the overflow bucket
    for v in samples:
        for i, edge in enumerate(BUCKETS):
            if v <= edge:
                counts[i] += 1
                break
        else:
            counts[-1] += 1
    return counts

def p95_from_hist(counts):
    total = sum(counts)
    if total == 0:
        return None
    rank = math.ceil(0.95 * total)
    seen = 0
    for i, c in enumerate(counts):
        seen += c
        if seen >= rank:
            if i == len(counts) - 1:
                return None      # the answer lives above our last edge
            return float(BUCKETS[i])
    return None

# hand-built: about 8% of requests are slower than the last edge (1000 ms)
counts = [0, 0, 200, 3000, 4000, 2000, 600, 100, 900]
print(p95_from_hist(counts))                    # None -- the honest answer
print(sum(counts), 'samples,', counts[-1], 'above the last edge')

The function returns None on purpose when the target rank lands in the overflow slot, and I now treat that None as a feature rather than a bug. A version that silently returns the last finite edge would report 1000 ms for a distribution whose tail is much slower, and nobody downstream would ever question it. Two habits came out of this: assert that the overflow count is small relative to your SLO threshold, and export that count as its own series so an empty tail is visible before it becomes a mystery.

The decision table I now keep next to the dashboard

Approach	Merges exactly?	Cost	Fails when
Raw samples	Yes, concat then rank	storage scales with requests	volume or retention too low to keep them
Fixed histograms	Yes, if bucket edges match and overflow is kept	bytes per series	tail lives beyond the last edge, or edges differ per client
Sketches (t-digest and friends)	Approximately	small and tunable	you treat the approximation as exact under audit
Average of per-shard percentiles	No	free	always, in a direction you cannot predict

My rule after this week is simple and slightly boring. If a number cannot be recomputed from its parts, it does not go on a panel that other people trust. If it can be recomputed but the population is mixed, it gets split by route before anyone is allowed to read it.

Where the free server and the free models actually fit

Two parts of this needed something other than my laptop, so here is the honest accounting. I ran the seed sweep, the same script over a few thousand seeds checking whether naive merging ever matches the true percentile, on MonkeyCode's free server option; a CPU-bound throwaway loop is exactly the kind of work I do not want pinned to my machine for an hour. MonkeyCode also offers free model access, which I used to red-team the table above by asking for traffic shapes where my ordering claim breaks. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The model found two shapes I had missed, both involving a large slow shard, and one of them produced a naive merge that landed within a few percent of the truth. That was a useful reminder that a wrong method can look fine on a lucky day, which is precisely how it survives code review for a year.

What still does not work, and who should skip this

The harness is synthetic, so it proves the arithmetic and nothing about your production traffic. Nearest-rank percentiles also differ from the linear-interpolation definition some tools use, and on a few hundred samples those two definitions can disagree enough to reopen the same argument. If your volume is low, do not build any of this: keep raw samples, compute exact percentiles, and spend the saved afternoon on something else.

This approach is also the wrong tool if you need per-user or per-tenant latency, because aggregation destroys exactly the identity you need. And none of it fixes head sampling, clock skew between collectors, or a client library that quietly drops the overflow bucket before the data ever reaches your backend.

What I would repeat on Monday

Write the reproduction harness before the patch, and put the percentile definition in a comment where the next person will trip over it. Split populations before you compare them, test the missing-shard case on purpose, and let a function return None rather than a number it cannot defend. The 48 hours were expensive, but the harness took twenty minutes and would have ended the argument in the first one.

If you want to run the seed sweep yourself, the free server option is the least intrusive way I found to do it without pinning a laptop. Send me the traffic shape where the naive merge comes closest to the truth, because I suspect small fast shards are the worst offenders and I would like to be proven wrong.