How to Read a Latency Budget

Gangadharan Venkataraman — Mon, 02 Nov 2026 05:41:08 +0000

Gangadharan Venkataraman

Your average response time is 80ms and a tenth of your users are having a bad day.

Someone hands you a requirement: "the endpoint should respond in under 200 milliseconds."

That sentence looks like a specification. It isn't one yet. It's missing the two things that make a latency target actionable — which percentile, and how the budget is divided. Without those, you'll ship something that averages 90ms and still generates complaints.

Averages hide the thing you care about

Here's a distribution that averages 80ms:

Percentile	Latency
p50	45ms
p90	95ms
p99	340ms
p99.9	2,100ms

Mean: about 80ms. Looks fine on a dashboard.

Now count users. One in a hundred waits a third of a second. One in a thousand waits over two seconds. If you're serving 10 million requests a day, that's 10,000 people a day on the two-second path.

And it's worse than it looks, because slow requests aren't randomly distributed across users. They correlate with cache misses, which correlate with new users, large accounts, and unusual queries. The tail isn't a random tenth of a percent. It's often your most valuable customers, consistently.

So: always state a percentile. "p99 under 200ms" is a specification. "Under 200ms" is a wish.

Draw the budget before you write the code

A latency target is a budget, and a budget gets spent. Write down where before you start, because you'll discover the allocation was never going to work far more cheaply on a whiteboard than in load testing.

A worked example for a 200ms p99 on a recommendation endpoint:

Stage	Budget	Notes
Network in + TLS	15ms	Mostly out of your control
Auth + request validation	5ms	Cache the token check
Feature fetch	40ms	The risky one. Network call to a store
Model inference	60ms	Measure it, don't guess
Business rules + filtering	20ms	Grows quietly over time
Response serialization	10ms	Larger than people expect for big payloads
Network out	15ms
Headroom	35ms

That headroom line is not padding you can reclaim later. It's what absorbs a GC pause, a slow DNS resolution, a noisy neighbour on the host. A budget that sums exactly to target is a budget that misses target.

Rule of thumb: allocate about 80% and leave the rest.

Sequential stages add. This is the whole problem.

If your request calls three services one after another, your p99 isn't the p99 of the slowest one. It's worse than that, because you need all three to be fast simultaneously, and they fail to be independently.

Three services, each with a 1% chance of being slow, chained sequentially: the probability that none is slow is 0.99³, about 97%. You've turned three 1% tails into a 3% tail just by calling them in order.

Two fixes, and you want both.

Call concurrently where the dependency graph allows it. If feature fetch and user profile lookup don't depend on each other, run them together. Sequential cost is the sum; concurrent cost is the maximum. With a 40ms and a 30ms call, that's 70ms versus 40ms, for the price of a Promise.all or a CompletableFuture.

Most request paths have more available concurrency than they use, because they were written in the order someone thought about the problem rather than in the order the data requires.

Set per-stage timeouts, not just an overall one. An overall 200ms timeout means a stage that hangs consumes the entire budget before anything else gets to run. Give each stage its own, sized from its own p99 plus margin, and make sure the sum of your timeouts is less than your overall budget. Otherwise a single slow dependency guarantees a missed SLA rather than a degraded response.

Decide what to do when the budget runs out

This is the design decision most teams skip, and it's the one that determines whether degradation is graceful.

When the feature store is slow, do you:

Wait, and blow the latency target
Return an error
Serve a cached or default result and mark the response as degraded

Option 3 is almost always right for recommendation, ranking and personalization. A slightly worse recommendation delivered in time beats a perfect one that arrives after the user has scrolled past.

Option 2 is right for anything where a wrong answer costs more than no answer — payments, authorization, fraud decisions above a threshold.

Option 1 is never right, but it's the default in most codebases, because nobody chose. Whatever you pick, emit it. A degraded: true field in the response and a counter on the dashboard will tell you how often you're falling back, which is a number most teams don't have and would be alarmed by.

Measure where it hurts

Client-side, not server-side. Your server reports the time from "request parsed" to "response written." The user experiences DNS, connection setup, TLS negotiation, transfer and rendering. Those gaps are frequently larger than everything you've optimized.

Also: measure at the edge, under real concurrency. A latency number from an unloaded staging environment with a warm cache is a number about your staging environment.

The short version

Always say which percentile. p99 or it isn't a target.
Write the budget down by stage before you build.
Keep 20% as headroom and don't spend it.
Run independent calls concurrently. Sequential stages add up fast.
Per-stage timeouts, summing to less than the overall budget.
Decide the fallback explicitly, and emit a flag when you take it.

The 200ms wasn't the hard part. Deciding what happens at 201 is.

About the author

Gangadharan Venkataraman works on architecture for real-time machine learning and decision platforms. He holds an M.S. in Computer Science from the Georgia Institute of Technology and has spent over a decade building large-scale distributed systems and production ML infrastructure.

LinkedIn: linkedin.com/in/ganga5v

DEV Community: Gangadharan Venkataraman