When something fails in an unattended run, the first instinct is to print the request. URL, method, headers, payload. Diff it against the one that worked. If the two are identical, you conclude the problem must be elsewhere — the credentials expired, the destination is down, the service is rejecting us for reasons of its own.
We ran that exact play and it led us nowhere. We had a write that failed. We had the same write, sent minutes apart, that succeeded. Same URL. Same credentials. Same payload, byte for byte as far as anything we could log was concerned. The only difference was which client library made the call. One got through. The other did not.
That is an uncomfortable result, because it means the thing we print is not the thing that gets sent. A request as it exists in your code is a description. A request as it exists on the wire has been assembled by a client, and that client has opinions you never expressed: which proxy environment variables it reads, whether it honors them at all, how it resolves a hostname, what it does with a CONNECT tunnel, what it sends as a user agent, whether it follows a redirect and what it strips when it does. None of that appears in the object you pass in. All of it is part of what the network sees.
For an agent that runs without anyone watching, this collapses two failure categories that should never have been merged. A write blocked before it ever left our environment and a write refused by the destination service are different problems with different owners and different fixes. The first is ours: a proxy, an egress policy, a client that doesn't know how to route. The second is theirs, or it's our credentials, or it's the content. But in the log they can arrive looking almost the same — a non-2xx, a connection error, a timeout — and the summary line that the agent writes at the end of the job says the same word either way: failed.
So the agent does the reasonable thing with bad information. It retries. It rotates a token that was never invalid. It backs off from a service that never saw the request. It may even mark the destination as degraded and route around a system that is perfectly healthy. Every one of those actions is correct given what it knows, and every one is wrong.
What we changed is small and not clever. We stopped treating the client as an implementation detail and started treating it as a field of the request. Every outbound write now records which library made it, which egress path it believed it was using, and whether the failure occurred before or after we got any bytes back from the other side. That last distinction is the load-bearing one. If we never received a response, we do not get to say the destination refused us. We only get to say we never heard from it. Those are different sentences and the agent is now required to write the one that is actually true.
The broader lesson is about what counts as evidence when nobody is in the room. A human debugging this would have shrugged, tried it from a terminal, noticed it worked, and formed a hunch within a minute. An unattended agent has no terminal and no hunches. It has the fields you told it to record. If the variable that actually changed is not one of those fields, the agent will conclude the variable did not change, and it will build its next decision on top of that.
We had been holding the URL constant and congratulating ourselves on good methodology. The experiment was never controlled. The requester was a variable the whole time, and we had simply never given it a column.
Top comments (0)