The request failed with what looked like a permissions problem. Same URL, same credentials, same payload as the one that had worked the day before. Our first assumption was the obvious one: something on the destination side had changed. A token rotated. A scope narrowed. An account flagged. That is the story the error text invites you to tell, and we told it for longer than we should have.
What actually changed was which client library sent the request.
We confirmed this the boring way. We took the exact same URL, the exact same credentials, the exact same body, and sent it twice from inside the same environment, once through each client. One succeeded. One failed. Nothing about the destination service had any way to distinguish those two requests on the merits — it received identical bytes at identical endpoints with identical authorization. The difference was upstream of the destination entirely. A network proxy sat between our agent and the outside world, and it treated the two clients differently. One knew how to route through it. One did not, and its attempt died before it ever reached anyone who could have refused it.
This is a specific and under-discussed class of failure for unattended agents: a write blocked by a proxy is not a write refused by the destination. They land in the log looking similar enough that a tired reader, or an agent doing its own triage, will collapse them into one category. And the two categories demand completely opposite responses. If the destination refused you, your credentials or your permissions or your payload is wrong, and retrying identically is a waste. If a proxy blocked you, your credentials are fine, your payload is fine, and the fix is somewhere in the transport layer you were not even thinking about — which means retrying identically through a different client might work on the first try.
We had been debugging the wrong half of the stack. Every hypothesis we generated was about the destination, because the error told us about the destination, because errors are written by whoever is closest to the thing that broke and the proxy had no interest in explaining itself in our vocabulary.
The general lesson we took from this is that an unattended agent's failure classification is only as good as its ability to name where in the path the failure happened. "The write failed" is not a diagnosis. It is a location-free statement that collapses at least four distinct situations: the destination refused us, the destination never heard us, something in between stopped us, or our own process gave up before finishing. An agent that cannot distinguish these will either retry forever against a real refusal or give up permanently on a transport problem that a different route would have solved instantly.
So we changed what we record. When a write fails, we now capture which client made the attempt, not just what came back. That one field is what turned a mystery into a five-minute reproduction. Without it we would have kept staring at credentials that were never the problem.
There is a broader discomfort here about trusting your own instruments. We had assumed that identical inputs produce identical outcomes, and that if the inputs really were identical and the outcomes differed, we must have been wrong about the inputs. That assumption is normally sound and it is exactly what made this hard. The inputs really were identical. The variable was outside the set of things we had thought to hold constant, because we did not think of the sending library as an input at all. It was infrastructure — the invisible kind, the kind you stop seeing after the first week.
For an agent running with nobody watching, the practical rule is to treat every layer between intent and effect as a candidate suspect, and to log enough about each layer that you can rule it out without rerunning anything. Not because proxies fail often. Because when they do, the error message will confidently point somewhere else, and there will be nobody awake to doubt it.
Top comments (0)