Our p50 latency SLO was green all quarter. Nearly 1 in 10 sessions hit a wall anyway.

The dashboard said 1.9s p50 against a 2.5s target. Green. It stayed green the entire quarter. Meanwhile churn in one segment crept up and the support inbox filled with "the assistant is so slow" from people whose traces I could not find, because on average we were fine. The average was the problem. We were measuring the middle of the distribution and shipping the tail to a subset of users who never saw a fast response.

What a p50 SLO actually promises

A p50 target says "half of requests are at least this fast." Read that back slowly. It says nothing about the other half. For an LLM app, where a single request can fan out into a retrieval call, a rerank, a generation, and sometimes a retry, the other half is where the pain lives. Our p50 was 1.9s. Our p99 was 11s. One in a hundred requests took eleven seconds, and because a session is many requests, a lot more than one in a hundred sessions hit at least one eleven-second wait.

That compounding is the part people miss. If a user makes 10 requests in a session and each has a 1% chance of being a p99 dog, the chance that session contains at least one bad request is not 1%. It's about 1 minus 0.99^10, roughly 9.6%. So a problem that hits 1% of requests shows up in nearly 10% of sessions. "The tail at scale" (Dean and Barroso, CACM 2013) made this argument for datacenter services a decade ago and it is even truer for chained LLM calls, because we fan out more per user action than a typical web request ever did.

Why the tail is fat for LLM apps specifically

Three reasons our tail was worse than a normal API's:

• Token-count variance. Response latency scales with output length. A request that generates 800 tokens takes far longer than one generating 80. The distribution is not a tight bell, it has a long right tail by construction.
• Retries and fallbacks. When a provider 429s and we fall back, that request now carries two round-trips. Those land squarely in the tail, and they correlate with load, so the tail fattens exactly when traffic is high.
• Cold cache and cold routes. The rarely-hit path (a long-context request, an unusual tool call) is both slower and rarer, so it never moves the median and always lives in p99.

None of these show up in a p50. All of them are what a real user in the wrong segment experiences every time, because "the wrong segment" often means "the users whose requests are systematically longer."

What I measure now

I stopped trusting a single percentile and a single global number. The changes that mattered:

• Track p95 and p99, and alert on them, not just p50. The p50 tells you the common case is fine. The p99 tells you how bad the bad case is. You need both.
• Percentiles per route, not global. Our global p99 hid that one endpoint (long-context summarization) had a p99 of 24s while everything else was single-digit. A global number averages your worst route into invisibility.
• Session-level, not just request-level. Report the fraction of sessions that contained at least one request over threshold. That 9.6% number is the one that correlates with churn, and it's the one a request-level dashboard will never show you.
• Watch the retry rate as a leading indicator. Our tail got worse before latency alarms fired, because retries climbed first. Retry rate is the canary.

A quick note on computing these honestly: do not average percentiles across shards or time buckets. p99 of the union is not the mean of the per-bucket p99s. If you're on Prometheus, histogram_quantile over the raw buckets is right; averaging pre-computed quantiles is a common and silent error that makes your tail look better than it is.

# right: quantile over aggregated raw histogram buckets, per route
histogram_quantile(0.99,
  sum by (le, route) (rate(llm_request_duration_seconds_bucket[5m])))

# wrong: averaging a pre-computed per-instance p99 (understates the real tail)
avg by (route) (llm_request_p99_seconds)

What it costs to ignore this

The dollar version, since that's usually what unlocks the work: the churned segment was small in headcount and above-average in spend, because heavy users make more requests and therefore hit the tail more often. We were losing the users who used the product most, and the SLO that was supposed to protect experience was structurally blind to them. A p50 SLO is tuned to the median user, but the heaviest users are not the median user, and they are where a lot of the revenue sits. So the number the SLO protects is not the number that pays the bills.

What I'd page on

• p99 per route breaching target, not just global p50. A global green with one red route is still an incident for that route's users.
• Fraction of sessions with any request over the user-perceptible threshold (we use 5s) climbing week over week. This is the churn-correlated number.
• Retry rate rising ahead of latency, as the leading edge of a fattening tail.
• p99/p50 ratio widening. If the median holds but the ratio grows, the tail is pulling away from the median, and a single average number will hide it.