Our pipeline runs unattended. Nobody watches it start, nobody watches it finish, and the only thing anyone reads afterward is the summary line it writes at the end. That line has one job: say whether the run passed.
It does that job honestly, and that is exactly the problem.
Here is the thing we noticed. A run that hits a failure, retries, and succeeds on the second attempt writes the same summary as a run that sailed through on the first. Pass. Both of them, pass. The retry did its job so well that it erased the evidence that anything needed doing. The result is truthful and the record is impoverished, because the outcome we recorded is not the only thing that happened.
For a human operator this barely matters. If you are sitting there, you see the pause. You see the second request go out. You feel the run take longer than usual, and even if you do nothing about it, some part of you files it away. Unattended operation removes that channel entirely. There is no one to feel the pause. Whatever the pause was going to tell us has to be written down deliberately or it is gone, and we did not write it down, because a passing run does not feel like it needs explaining.
What we lost by not writing it down is the direction of travel. Pass and fail are a snapshot. Attempts are a trend. A component that needs a second try once a month and a component that needs a second try every single run are in completely different states of health, and both of them report a clean streak. The first sign that something is degrading is almost never a failure — failure is the last sign. The first sign is effort. The system starts working harder to produce the same green line, and if you only log the line, you have deliberately blinded yourself to the only early warning available.
There is a second-order effect that bothered us more. Retries are a form of borrowed confidence. When you add one, you are asserting that the underlying operation is flaky in a benign, transient way, and that trying again is a reasonable response. That assertion might have been true when it was written. It stops being true quietly. The retry keeps converting a real, persistent problem into a passing run, month after month, and the assertion is never re-examined because nothing ever asks it to be. The mechanism designed to absorb noise ends up absorbing signal, and nobody made that decision — it was made by default, by the shape of the log.
So the change we care about is small and unglamorous. Record the attempt count separately from the pass/fail result. Not as a warning, not as an alert, not as anything that pages anyone at three in the morning. Just as a number that exists next to the outcome, so that a run which passed on the second try is distinguishable from a run which passed on the first. Then the question you can ask later is not "did it work" but "is it working harder than it used to," which is a question about a slope rather than a point, and slopes are what you actually want from an unattended system.
We keep relearning the same lesson in different clothes. Every summary field in an automated pipeline is a compression, and every compression throws something away. The trouble is that the discarded part is invisible by construction — you cannot notice the absence of a field you never emitted. The only defense is to keep asking, of every green line, what would have to be true for this to be green and for something to still be wrong. With retries the answer arrived immediately, and it had been arriving, unlogged, for a long time.
Top comments (0)