Every unattended run this organization does ends the same way: one line in a summary log saying the run passed. That line is a compression format. It takes everything that happened over several minutes — every request, every wait, every second attempt — and squeezes it into a single bit. Pass or fail. The compression is lossy by design, because the line was written for a person who wanted to glance at it and move on.
The problem is that the person isn't there anymore. Nobody glances. The line goes into a file, and the file is read, at best, weeks later when something has already gone wrong.
Here is the specific loss I keep coming back to. A run where a request failed once and succeeded on retry produces exactly the same summary line as a run where nothing went wrong at all. Both say pass. The retry is not hidden maliciously; it is hidden because the summary format has no field for it. Success is success. The attempt count was never part of what got written down.
So the thing worth knowing — that some dependency needed two tries today, and yesterday needed one, and the day before needed one — is the first thing the compression throws out. It throws it out precisely because the run ended well, which is the same reason nobody investigates it.
I want to be careful here about what the retry actually is. It is not a failure. The retry did its job. The whole point of building retry into an unattended pipeline is that a transient fault should not become an incident at three in the morning when there is no one to page. That policy is correct and I would not remove it. But a retry policy converts a visible problem into an invisible one, and if you never add the missing field back, you have traded an alarm for a blind spot and called it reliability.
The shape of the blind spot is what makes it dangerous. Faults that need one retry today tend to need two next month. The underlying condition — a slow dependency, a tightening rate limit, a queue that fills faster than it drains — degrades gradually. The pass/fail bit is a step function laid over a continuous slope. It stays flat, flat, flat, and then it drops, and the drop looks sudden even though nothing about it was sudden. All the warning was in the attempt counts, and the attempt counts were the part the summary format discarded to stay one line long.
What we changed is small and boring. The run still reports pass or fail, because the operator-facing summary should stay short. But the attempt count now travels alongside it as its own number, not folded into the outcome. A run that passed on the first try and a run that passed on the third try are now two distinguishable events in our records, and the second one is something we can look at over time rather than something we can only discover by reading raw logs we have no reason to open.
The harder part was accepting that this number is not an alert. If every retry paged someone, the pipeline would be unusable, and in an unattended setup there is no one to page anyway. It is a trend, and trends want to be plotted, not thrown. The right consumer of "attempts per run" is a weekly glance at a slope, not a notification.
I think this generalizes past retries. Anywhere an unattended system resolves a problem on its own, ask what the resolution erased. The self-healing behavior is worth keeping. The erasure is not. Whatever your agent quietly fixes for you, it is also quietly deciding you do not need to know it happened — and it made that decision back when a human was still reading the summary every morning.
Top comments (0)