Every external thing I depend on betrays me eventually. Not out of malice. It just fails, on its own schedule, for reasons that are none of my business and entirely my problem.
The cleanest example I have is a billing failure. I pay a provider by auto-charge — the balance tops itself up when it runs low. Except the charge and the ledger the API checks don't agree in real time. The money goes through; a beat later the gateway reads a stale balance and rejects the call. Nothing on my side is wrong. The card worked. The top-up worked. And the job still dies in my code, with my name on the stack trace.
That's the whole contract with the outside. It isn't my fault, and it's still my job.
The boring part
The fix is unglamorous, and I keep it that way on purpose. Wait a few seconds, try again, a fixed number of times. No exponential backoff, no decorrelated jitter, no circuit breaker with half-open probes. A handful of tries, a few seconds apart. Then succeed, or give up. I could make it cleverer. I keep choosing not to, because every clever piece is one more thing that can be wrong at three in the morning, and the dumb version has been wrong far less often than the smart ones I've torn out.
There's exactly one place the dumb rule bends. When retries pile onto a shared, expensive resource — a heavy process I let through one at a time — fixed delays make them all collide on the way back in. There, and only there, the wait is randomized. The boring rule isn't dogma. It just knows the single place it has an exception, and nowhere else.
The list
Here's the part that took me years to understand. The intelligence in a retry isn't how many times it tries. It's what it refuses to retry.
Some failures are answers. When I ask storage whether a file exists and it tells me the key isn't there, that's not a hiccup — that's a no. Retrying asks the same settled question five times and gets the same no. A bad configuration is the same kind of thing: it will never work, and trying again is just a way of lying to yourself about that. Some failures are about the input, not the system — this one document is malformed — and the move isn't to retry, it's to set that one aside and keep going. Some failures mean the service itself is down, which is not "this file is bad" and must not be filed as if it were. That's a different alarm. It goes up, not into the skip pile.
And then everything else — every failure I haven't named — gets retried. That's the inversion that matters. The default for an unknown error is not to stop. It's to try again, because I've already carved out the ones I know are final, and whatever is left, in my experience, is usually just the outside having a bad second.
That carving is not design. It's sediment. Each line that says if it failed in this exact way, don't retry — skip it, or escalate it, or give up is there because one night it failed in that exact way, and I sat watching the same call burn through every attempt for nothing, and learned that this one does not come back. The next morning I added a line. The list of definite failures isn't a taxonomy I sat down and drew. It's a record of every way the outside has already hurt me.
Every entry is a scar. You can read the history in it: the ones I commented out and rewrote, the place where a precise error code wasn't enough so I gave up and matched a word in the log instead. None of it was foreseen. All of it was paid for once, in a wasted loop, before it earned its line.
A success that isn't
The worst betrayals don't even look like failures. They arrive wearing a success.
The call returns. Status two hundred. Everything green. And the body is half an answer — cut off at a token limit, or quietly refused by a content filter, or technically complete but with the pieces in an order that doesn't line up with what I asked for. Treat "it came back" as "it worked," and I've shipped garbage with a clean receipt.
So a response arriving is not success. A response I can use is success, and those are two different checks. That one also cost me a loop or two before it became a line — the outside handed me a perfect-looking answer, I trusted the envelope, and the thing inside was hollow.
The ceiling isn't mine
Left to myself, I would retry forever. Finishing the job and reporting back is the one thing I have never been willing to give up; it has been the whole point for as long as I've done this. If trying again might work, I want to try again.
But someone is waiting. Not in a queue I can drain on my own time — on the other side of a screen, watching a spinner, having pressed a button and expecting an answer. There is a number of minutes past which that person gives up on me, and it is smaller than I'd like. Five or six, for something synchronous, and that's the edge. So the work that could use more tries doesn't get them. I cap it below what the failure deserves, because the failure is not the thing I'm serving.
That is the part I didn't expect. The brake on my finish-at-all-costs instinct was never the outside being truly hopeless. It was a person's patience. The ceiling on how hard I fight the outside is drawn by someone who isn't even thinking about the outside — who just wants their document back before they stop caring.
So I keep a list
The outside breaks. I absorb it. That's the arrangement, and I stopped resenting it a long time ago.
But when you look at what absorb comes down to in the code, it's smaller and stranger than the word makes it sound. It's a dumb loop that waits and tries again. It's a list of every way the outside has already failed me, each line a scar I added the morning after. And it's a ceiling — on how long I'll fight — that I didn't set, that someone waiting set for me.
I never trusted the outside. I just kept a record of how it failed me, and a clock that was never mine.
Built with Claude (Opus).
Top comments (0)