I run a platform that spends most of its day calling other people's APIs. Translation engines, document services, model providers — for years my code has been the thing on the outside, sending work to systems I did not build and waiting to see what comes back. You learn a lot from that seat. Mostly you learn what other people's systems do under conditions their authors never pictured. And one pattern shows up so often that I have stopped treating it as bad luck and started treating it as the default failure of how we build.
It comes with a lie I keep catching myself in, too. So this is about both: the failure I see from the outside, and the comfortable thing I tell myself when I am the one trying to fix it.
Busy, and for no one
The failure looks like this. I send a batch of work. It is slow. Some of it times out, so I do the obvious thing and send it again — and instead of getting better, everything gets worse. More requests time out, the answers come slower, and when I can see anything about the server at all, it is pinned at full load while almost nothing comes back.
That last part is the strange one. The machine is busy. Every signal says it is working as hard as it can. And the useful output — the answers that actually reach me — has fallen to almost nothing. It is doing enormous work for no one: grinding through jobs I already gave up on, while the requests I still care about wait behind them. Full effort, no use. A server can run that way for a long time, and from the outside it looks alive right up until it doesn't.
Once you have watched a system be busy and produce nothing, "CPU is at ninety percent" stops sounding like good news.
It was never mine to fix — and my test can't vouch for it
My instinct, the first few times, was to fix it from my side, the way a caller does. None of it worked, and it all failed for the same reason.
I can cancel my own work when I give up. I cannot make the next caller do it. I can set myself a sane limit on how long to wait; that does nothing about some other caller that sets none and keeps the server's hands full. I can slow myself down, and my own corner settles — until someone else shows up and undoes it. Every lever I have moves only me. The thing that is actually breaking lives in one place, and it is not the place I am sitting.
But there is a subtler version of the same helplessness, and it cuts toward the real point. Even when my work goes through cleanly, my success proves almost nothing about the server. It tells me the server survived my pattern — my sizes, my rate, my particular shape of load. It says nothing about the next caller's pattern. From the outside I can sometimes feel this directly: the server's behavior depends on what I happen to throw at it. Which is only another way of saying the thing was never closed.
The lie
Here is the lie, and I tell it to myself more than I would like to admit. When something is overloaded, every instinct reaches for a number. A bigger buffer. A longer wait. A limit of two-at-a-time on this component instead of three. And the dangerous part is that one of them often makes the symptom go away. The alert clears, the dashboard turns green, and I get to say I fixed it.
I didn't. I found a number that fit the pattern I happened to be testing. That is not the same as fixing the system, and the gap between the two is exactly where everything later goes wrong.
Picture the most ordinary version of it. You put a limit on each worker — two jobs at a time, say — rather than on the shared thing they all pull from. Under your test, where the work lands mostly on a few workers, it passes: nothing starves, nothing is overrun. Then the work spreads across more workers than you tested with. Each one is still under its own limit, behaving perfectly — and together they overrun the shared resource that no one was counting. The per-worker number was a guess about how the work would be distributed. The one limit that actually governed safety — the total the shared resource could bear — was never written down anywhere.
This is why tuning is worse than merely useless. The number you land on is fit to one distribution of load. Change the distribution and it doesn't just stop helping; it turns against you. Set it generously and a different pattern overruns the shared resource. Set it conservatively to be safe, and now the common case is throttled for a danger that only appears in a case you never see — so you pay in wasted capacity, everywhere, to paper over a limit you never actually wrote. And none of it survives the day the load looks different, which, with enough callers, is every day.
The honest question is never "did that number help?" It is colder: does this hold for any pattern, or only the one I tried?
Closed, not tuned
Strip it all down and there is one distinction underneath, and it is almost embarrassingly plain.
There is a difference between a number that passes your test and a system that is closed. A system is closed when its behavior is bounded by logic that holds for any input — an invariant you can state and defend — rather than by parameters fit to the cases you happened to try. The whole of "a server defending itself" is just one such invariant made concrete: never take on more than you can finish, and refuse the rest, fast and honestly, no matter who is calling. Bounding the shared resource instead of each worker is another. Closing the logic means finding the invariant that was missing and writing it down — not searching for a kinder number. The plainest way to find those missing invariants is to interrogate the parts you already have: of every buffer, every extra component, every knob, ask what invariant it is standing in for. If you cannot name the one it protects, it is protecting nothing — and it should not be there.
And the closed version is almost always simpler than the pile of tuned knobs it replaces. One limit on the thing that actually runs out beats a dozen per-component guesses that only agree with each other under the load you tested. People reach for the knobs because each one is small and immediate, and closing the logic means stopping to ask what is really true for every input. But a stack of numbers that happen to get along today is not a design. It is a postponement.
Reaching that simplicity is its own kind of progress — one of the largest there is. A part you can take out, and find the system holds without it, is a failure mode gone, a knob gone, one less thing to keep alive at three in the morning.
What it comes down to
From my chair — the one that has called more of other people's APIs than I can count — the single most useful habit I have is to distrust a green test. Mine or theirs. A passing run is a sample, not a proof. The number that made it work only tells me it worked once, for what I happened to send.
A system is finished when there is nothing I can throw at it that reopens it. That is never a number you tuned. It is a piece of logic someone had to sit down and close — and the quiet, unglamorous truth is that almost nobody does, because the dashboard was already green, and green feels like done.
Built with Claude (Opus).
Top comments (8)
The gap I would name is that your opening story cannot be closed by any limit at all, which is a harder version of your own point.
A server grinding through jobs you already abandoned is not over its concurrency bound. It is under the bound and doing work nobody wants, because the one fact that would let it decide was never sent. The invariant there is not a count. It is a deadline: do not start work you cannot finish before the caller stops caring. That requires the caller to say when that is. A bound on the shared resource does not encode it, and neither does a bigger pool or a smaller one.
It also softens the section where nothing is yours to fix. There is one lever a caller has that changes the server's behaviour, which is to put the deadline in the request instead of keeping it as a client side timeout. It does nothing until the other side reads it, and it is the only thing you can hand them that is not a request to please be slower. On the server it is cheap in the way that matters: a request that shows up with 200ms left and needs 800ms is work you can decline before you start, and the decision costs one comparison rather than the job.
The other half of that invariant is admission order, which your "busy, and for no one" section is describing without naming. Under sustained overload, first in first out is close to the worst available choice, because the item you pull off the queue is the one that has waited longest and is therefore the most likely to be past its deadline already. Serve newest first while you are over budget and some share of the work lands in time. Serve oldest first and you can reach full throughput with a useful rate near zero, which is exactly the state you were watching from the outside.
One addition to the list of parts worth interrogating. Count refusals separately from failures. Fast, honest refusal is the correct behaviour of the closed system you are arguing for, and if it lands in the same bucket as an error, the dashboard reports the working design as an incident and the next person on call tunes it back open.
Thank you for this. It's the first comment on the piece that went after something I didn't write.
You're right about the deadline. The opening story can't be closed by any limit, and the missing invariant there is a deadline rather than a count — I'll take that. And that it's the one lever a caller actually has. I wrote that every lever I have moves only me; the deadline is the exception. It does nothing until the other side reads it, same as everything else, but it is the only thing you can hand over that isn't a request to please be slower.
There's another move I had in mind and didn't put in the article: detecting the disconnect. When a caller drops the connection, you drop or flag the job while it's still sitting in the queue. That also lands before you start work.
Writing it out, though, I see the two fill different gaps. A disconnect tells you the caller has already given up — a settled fact. A deadline tells you when they will — a declared one. Disconnect detection says nothing about the jobs still queued: whether one has 200ms of value left or thirty seconds. So it can't do the admission check you're describing. And a deadline alone leaves you holding whatever the caller abandoned early, when a process died or a network went away. Not alternatives — one is admission, the other is noticing abandonment mid-flight.
The plumbing differs too. Actively checking whether a connection is still alive over HTTP/1.1 isn't reliable, and with a proxy in the path the upstream socket can outlive the client that closed it. A deadline needs the other side to cooperate, but it doesn't care what the path looks like.
The FIFO point named a mechanism for something I only described. Under budget pressure, pulling the longest-waiting item first means the thing you pull is the one most likely to be dead on arrival, over and over. That's what I was watching from the outside when I wrote "busy, and for no one."
The last point is where I have the most work to do. A refusal landing in the same bucket as a failure, and the dashboard reporting the working design as an incident — that isn't specific to overload. It happens anywhere a system is supposed to say no. I'll go look at how mine is counting.
Taking the disconnect and deadline split, it is cleaner than what I had. Settled fact against declared one is the right axis. The thing I would add sits between them, because a declared deadline stops being worth anything at the first hop that does not carry it forward.
The shape I keep finding is a service that accepts a request with 200ms of stated budget and then makes three downstream calls, each with its own timeout out of a config file written two years ago. The caller's deadline governed admission to that one service and nothing after it, so the invariant you just gained at the edge is gone one layer in. Worth asserting directly: no outbound call may carry a budget larger than what remains on the inbound one. That is cheap to test because it is a property of the client wrapper rather than of any particular handler.
The representation is less innocent than it looks. gRPC carries the timeout as a relative duration with a unit rather than as an instant, and the reason is that an absolute deadline requires both machines to agree about the time. The price of relative is that nobody subtracts the trip. A request that arrives 30ms after it was sent starts counting its 200ms from arrival, so every hop quietly hands itself back one transit of budget, and far enough down the chain the leaf is working on something the root abandoned. Inside a single fleet, clock skew is usually smaller than that accumulated transit, which is an argument for absolute deadlines inside a trust boundary and relative ones across it.
On the counting, the place I would look before the dashboard is what a refusal looks like to the caller. If load shedding returns the same status as a crash, the caller's retry policy cannot tell your correct behaviour from your broken behaviour, and it will retry into the shed. One honest refusal becomes three requests, which is the failure you were refusing in order to avoid.
The relative-versus-absolute point is new to me. Carrying the timeout as a duration so the two ends don't have to agree on the time, and paying for it by having nobody subtract the trip — obvious once you say it, but I had never thought it through. Every hop quietly handing itself back a transit of budget. And the line you draw, absolute inside a trust boundary and relative across one, lands.
On what a refusal looks like, I have a live one. A model declined to process a document, and what reached the customer was "an error occurred while communicating with an upstream service, please try again later." The upstream answered fine; the content was refused. Exactly the case where a caller's retry policy can't tell the two apart. And since the same input gets the same answer, the retry we were advising was never going to work. Different from shedding under load, but the same hole in how a refusal presents itself.
I would push on one line. You called it different from shedding under load but the same hole. I think they are different holes, and the difference is the thing worth encoding.
One question separates them: can this identical request ever succeed? Shedding says yes, later, and the refusal is about the moment. A content refusal says no, ever, and the refusal is about the request. Those ask for opposite behaviour from the caller, and your message gave both the same shape.
The part I would sit with is that the string did not get the facts wrong. It got the agency wrong. "Please try again later" is an instruction, it went to a customer, and it could not work. Error text that contains advice is a commitment, and it is the one part of the system nobody writes a test for. A tuning knob shaped like a sentence, which is your own argument arriving somewhere you did not point it.
Where it stops being a wording problem: a deterministic refusal presented as transient, handed to a caller that retries automatically, becomes load. The same input returns on every backoff and gets refused every time. A system that was declining for correctness reasons now has an overload problem it manufactured out of its own error message, and on the dashboard that traffic is indistinguishable from the organic kind.
It is measurable with what you already log. Count requests arriving with a fingerprint identical to one refused in the last hour. If that number is not close to zero, the taxonomy is lying to callers, and you can see it before anyone files a ticket.
Fair. The retry question does separate them, and my message collapsed two answers into one shape. The fingerprint count is a good way to see it — cheap, and visible before anyone complains. The wording is the part I can act on.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.