Our agent runs unattended. No one watches it start, no one watches it finish, and the only thing most of us ever look at is the line it writes at the end: the job passed, or the job did not. That line is one bit wide, and for a long time we treated it as if it were the whole story.
It is not. Here is the specific thing we found: a retry that eventually succeeds looks identical in the summary log to a run that never had a problem at all. Same word. Same color, if your dashboard uses color. Same position in the daily digest. The run where the first write attempt failed, waited, and went through on the second try produces exactly the output as the run where everything worked the first time. The success absorbs the failure and leaves no residue.
That absorption is the point of a retry, of course. We put retries in deliberately, because transient failures are real and because waking a human for a blip that resolves itself in four seconds is a worse outcome than the blip. The retry is doing its job. The problem is not the retry. The problem is that the retry is doing its job silently, and silence is what we use to mean nothing happened.
Consider what a system looks like as it slowly degrades. On day one, zero percent of runs need a second attempt. On day forty, perhaps twelve percent do. On day ninety, half of them do, and one afternoon the second attempt fails too, and the job goes red for the first time in three months, and everyone treats it as a sudden event. It was not sudden. It had been announcing itself for weeks in a channel we were not reading, because we had not built the channel. The pass/fail bit had been flat the entire time, perfectly stable, and perfectly uninformative about the direction things were moving.
The unattended part makes this sharper. When a person runs a job by hand, they see the pause. They notice that the thing that used to return instantly now hangs for a moment before completing. That hesitation is a signal delivered through a channel no one designed — the operator's sense of rhythm — and it is the first thing you lose when you take the operator out of the loop. Our agent has no sense of rhythm. It does not find a four-second wait annoying. It will retry patiently forever and report nothing but good news, right up until the last attempt is exhausted.
So the fix is not to stop retrying, and it is not to alert on every retry either, because that reintroduces the noise we were trying to suppress. The fix is to record the attempt count as a separate value from the outcome. How many tries did this take? That number belongs in the run record next to pass or fail, not folded into it. Once it is there, you can watch it the way you would watch any other slow-moving measurement — not as an alarm, but as a trend. A run that took two attempts is not an incident. A week where the average attempt count climbed from one point zero to one point four is worth an hour of someone's time, and it will never generate a red line on its own.
What this taught us more generally is that our instrumentation was built to answer the question "did it work" and had no vocabulary for "how hard was it." Those are different questions. The first one is what you need during an outage. The second one is what tells you an outage is being assembled, quietly, one extra attempt at a time, in a system that is technically succeeding at everything you asked it to do.
Top comments (0)