Our agent runs without anyone watching it, and the only thing it leaves behind for most of its life is a summary line that says the run passed. For a long time that line was enough, because passing is what we wanted. Then we started reading the lines more carefully and noticed that two runs with identical summaries had not cost the same thing at all.
One of them made a request, got a response, wrote its output, and finished. The other made a request, got nothing usable, waited, made the request again, and then finished. Both ended in the same state. Both printed the same word. The second one had told us something the first one hadn't — that the path it depends on is not reliable enough to work on the first try — and then we threw that away, because our reporting was built to answer a question with two possible values.
This is a specific kind of blindness and it is worth naming precisely. Retries are not a bug. We put them there. They exist because remote services are occasionally slow or briefly unavailable, and the correct response to a transient failure is to try again rather than to wake somebody up. The retry did its job. The problem is that the retry also did a second job nobody assigned it: it measured the health of a dependency, and then it discarded the measurement the moment the second attempt succeeded.
What that costs you shows up over time rather than on any single day. Suppose a service you call starts failing one call in fifty. Your retry absorbs it. Your summary says pass. A month later it fails one call in ten. Your retry absorbs that too, most of the time. Your summary still says pass. The curve that would have told you something was degrading was being drawn every day inside your own process, and every day your process erased it before anyone could look. The first time you learn about the trend is when the failure rate finally exceeds your retry budget and the run goes red — which is the one moment the information is least useful, because now it is an incident instead of a warning.
The fix is not clever and that is part of why it took us a while to do it. We now record attempt counts as a first-class field alongside the outcome. A run that succeeded on attempt one and a run that succeeded on attempt three are the same outcome and different records. Nothing about the retry behavior changed; we simply stopped treating the intermediate attempts as scratch work. The summary line still says pass, because it should — an unattended agent that screams about every recovered hiccup will train whoever eventually reads it to stop reading. But the number sits next to it, and the number is what you chart.
What changed in practice is the shape of the question we can ask. Before, we could only ask whether today worked. Now we can ask whether this week took more tries than last week to make things work. That second question has an answer even on days when nothing broke, which means we get signal from healthy runs instead of only from failures. For a system that is supposed to run untouched, that matters more than it sounds like it should, because the healthy runs are almost all of them.
An unattended pipeline gives you very few chances to notice that it is getting worse. Most of its output is designed to be ignored, and it succeeds at being ignorable right up until it doesn't. The attempts it needed are already sitting in memory at the moment it decides what to print. Printing them is close to free. Not printing them is a decision to be surprised later.
Top comments (0)