Why utilisation is not capacity, with the numbers.
Every capacity conversation I have been in goes the same way. Someone runs a load test, finds the number where the service falls over, and writes it down: "each instance handles 5,000 requests a second". From then on the sizing is arithmetic. 80,000 a second at peak, divide by 5,000, that's 16 instances, round up to 18 for safety.
Then the service falls over at 70,000.
Nothing was wrong with the load test. What was wrong is the idea that a service has a single number called capacity, and that it is fine right up to that number. It isn't. It gets slow long before it gets full, and the slowness is what kills you.
A service is a queue
Picture one instance of a redirect service. A request arrives, it takes 2 ms of actual work, it leaves. If requests arrived in perfect rhythm, one every 2 ms, the instance would be 100% busy and every request would take exactly 2 ms.
Traffic does not arrive in rhythm. It arrives in clumps. Two requests land in the same millisecond, one has to wait for the other. Three land, the third waits for both. The busier the instance, the more often this happens, and the longer the clumps take to clear.
The textbook result for this (the M/M/1 queue, if you want to look it up) is short:
time in system = own work ÷ (1 − utilisation)
At 50% busy a 2 ms request takes 4 ms. At 70%, 6.7 ms. At 90%, 20 ms. At 96%, 50 ms. At 99%, 200 ms. The curve does not rise, it turns vertical.
That denominator is the whole story. Capacity is the point where it hits zero. Everything interesting happens before that.
The same service, four ways
Here is a URL shortener, the classic warm-up problem. Redirects are the read path: a lookup in a cache, occasionally a miss to the store. 23 copies of the redirect service, 5,000 requests a second each, 2 ms of work per request.
At the design's peak of 80,000 redirects a second it sits at 70% busy. This is the number everyone would call "comfortable". Look at what a request actually costs:
| Traffic | Busy | Own work | Waiting | Total |
|---|---|---|---|---|
| 80K/s | 70% | 2 ms | 4.6 ms | 6.6 ms |
| 90K/s | 78% | 2 ms | 7.2 ms | 9.2 ms |
| 100K/s | 87% | 2 ms | 13 ms | 15 ms |
| 110K/s | 96% | 2 ms | 44 ms | 46 ms |
The work never changes. Two milliseconds, every row. The waiting goes from twice the work to twenty-two times the work, and traffic only grew by a third.
At 120,000 a second the service is over. Not slow, over: more is arriving than it can serve, the queue grows without bound, and there is no steady answer for how long a request takes.
Why 70% is the number
This is where the "70% headroom" rule of thumb comes from, and it is not arbitrary. Below 70% the curve is nearly flat: adding load costs you little latency. Above it, every extra point of utilisation costs more than the last one. Running at 70% means you are buying the flat part of the curve and leaving the vertical part for the day the marketing email goes out.
It also means something more concrete. At 70% busy with 23 copies, losing one copy puts the other 22 at 73%. Fine. At 90% busy, losing one puts them at 94%, and you are in the vertical part with a deploy half done.
What to do with this
Three things change once you see capacity this way.
Size for latency, not for throughput. The question is not "how many requests can it take" but "at how many requests does p99 cross what the caller will tolerate". For a 2 ms service and a 20 ms budget, that is about 90%, and you want to be well below it.
Measure own work separately from waiting. If your dashboards only show total latency, you cannot tell a slow service from a busy one. They need different fixes. Slow needs a profiler. Busy needs copies.
Look at the whole path. A request through a load balancer, a service and a cache pays the queue at each hop. Three components at 70% do not make a 70% system; the waits add up, and the slowest hop dominates.
Try it on your own numbers
The design above is a real one. Open it, change the traffic, watch the badges change colour:
Open the URL shortener in the playground
Put your own service in: own work in milliseconds, how many copies, what one copy handles. It will tell you where the curve turns vertical for you, which is a better number to write down than the one from the load test.
I built System Design 101 because I kept having this conversation. It is free, runs in your browser, and every number shows its working.
Top comments (0)