Why Chaining Five Fast Services Gives You One Slow One

The problem

Five services, each with a P99 latency of 200ms and a comfortable P50 of 20ms. Every team's dashboard is green. Every individual SLO is met. And yet the end-to-end request that touches all five has a P99 well over a second, and nobody can find the "slow service" causing it — because on any given slow request, a different service is the culprit.

This is the conversation that repeats in incident review after incident review: "It's not us, our P99 is 200ms, look at the graph." Five teams, five graphs, five clean dashboards, one furious product owner staring at a checkout flow that blows its SLA on 1 request in 20.

Why it happens

Percentiles don't add. That's the part almost nobody's intuition gets right on the first try.

If five services are called in sequence and each is independently slow (above its own P99 threshold) 1% of the time, the probability that the whole chain stays fast — every single hop landing below its own P99 — is 0.99 raised to the fifth power, which is about 95.1%. Flip it around: roughly 4.9% of end-to-end requests will hit at least one hop having a bad day. What was a 1-in-100 event for any single service becomes something close to a 1-in-20 event for the request your user actually experiences. You didn't add latency. You added opportunities to be unlucky, and tail latency is entirely about how often you get unlucky.

Fan-out makes this worse, not better. A sequential chain waits on one slow link; a scatter-gather call — hitting N shards or N replicas in parallel and waiting for all of them — waits on the slowest of N independent draws from the same distribution. Jeff Dean and Luiz André Barroso's 2013 paper "The Tail at Scale" has the sharpest framing of this I've seen: at large fan-out (hundreds of parallel calls, common in search or ad-serving backends), the probability that at least one call lands in the tail approaches 1. Almost every request ends up waiting on somebody's P99, even though no individual server is slow very often.

Two things quietly make this worse in real systems: hops usually aren't fully independent (a shared resource — a connection pool, a downlink, a noisy-neighbor node — correlates slowness across calls that "shouldn't" be related), and a single very slow hop (a GC pause, a retry) can dominate the whole chain's latency by itself.

What to do about it

Measure the critical path, not per-service dashboards. Distributed tracing (OpenTelemetry span data is enough) shows you which hop is actually on the critical path for slow requests — not which service has the worst P99 in isolation, which is usually a different, less useful ranking.

Use hedged (backup) requests for latency-critical fan-out. Dean and Barroso's mitigation, and it's simple: if a request hasn't returned within, say, the 95th-percentile expected time, fire a second, identical request to a different replica and take whichever comes back first, cancelling the loser. This trades a small amount of duplicate work for a large cut in tail latency, because now both attempts have to be slow, not just one.

Set per-hop deadlines that shrink as they propagate. A 2-second end-to-end budget split across five sequential hops shouldn't give each hop 2 seconds to retry in — propagate a shrinking deadline so a hop that's already burned 1.5 seconds doesn't get to spend another 2 trying again.

Reduce fan-out width before you optimize per-node latency. Cutting a scatter-gather from 50 shards to 10 often does more for tail latency than shaving 20ms off each shard's P99, because you're directly attacking the exponent, not the base.

Budget for P99, not P50, at every hop you add. Each additional synchronous hop taxes your tail latency measurably even if it barely moves your average — that tax is invisible until you do the math above.

Key takeaways

• Percentiles don't add across hops — probabilities of badness compound, and tail latency is a probability-of-badness problem.
• A chain of N services, each bad p% of the time independently, fails end-to-end at roughly 1 - (1-p)^N of requests — far more often than any single hop's own SLO suggests.
• Parallel fan-out is worse than sequential chaining for tail latency: you wait on the slowest of N draws, not just one.
• Hedged/backup requests, shrinking per-hop deadlines, and narrower fan-out all directly attack this multiplication; shaving per-service P50 mostly doesn't.
• Trace the critical path per slow request instead of trusting per-service dashboards — they can all be green while the request isn't.