Every morning our pipeline writes a line that says the work queue had nothing in it. For a long time we read that line the way you read a clean bill of health. Nothing queued, nothing pending, nothing broken. Go back to sleep.
The problem is that the same line gets written in two completely different worlds.
In the first world, the producer step ran, looked at everything it was supposed to look at, and correctly concluded there was no work to do today. Genuine zero. The system is idle because reality is idle.
In the second world, the producer step ran, hit something it could not handle, returned an empty result without raising, and handed that empty result downstream. The consumer opened the queue, found nothing, and logged exactly the same sentence. Zero because the thing that fills the number never got to count.
Those two states are not close to each other. One means the system is working. The other means a stage of the system failed silently and every stage after it is now dutifully processing the absence. But at the level of the log, they are indistinguishable. Same word, same length, same position in the summary. If you are reading the output of an unattended run — and by definition, on an unattended run, the log is all you get — you cannot tell which one happened.
What makes this specific to unattended operation is the absence of the human sanity check. A person watching the pipeline would notice that it has been quiet for three days and that quiet is unusual for a Tuesday. A person has a background model of what the normal volume looks like. The log has no such model. It reports the number it was handed, and the number it was handed is correct in both worlds. The log is not lying. It is answering a narrower question than the one we thought we were asking.
The fix is not to make the log more verbose. We tried that first, and more verbosity just means more lines that are equally true in both worlds. The fix is to make the producer state its own outcome separately from the queue's contents. Not "the queue has zero items" but "the producer ran, examined this many candidates, rejected this many for these reasons, and emitted zero." Now the empty result comes with a provenance. An empty queue with a producer that examined forty candidates and rejected all forty is a healthy zero. An empty queue with a producer that examined zero candidates is a failure wearing a healthy zero's clothes.
This is the same shape as a category of bug we keep rediscovering in different costumes: two distinct system states collapsing into one observable. A run that succeeded on the first try and a run that succeeded on the third try both write "success." A destination that refused a request and a destination that was never reached both write "failed." A queue that is empty and a producer that died quietly both write "empty." In every case the collapse happens because we instrumented the outcome and not the path.
The uncomfortable part is that the collapsed observable is usually the one that looks the most reassuring. Nobody builds an alert on "everything is fine." So the failure mode that disguises itself as fine is the one that survives the longest in production. Ours survived long enough that we only found it by asking, on a whim, why the volume looked lower than we remembered — a question no automated check was positioned to ask, because every automated check was reading the same ambiguous line we were.
If your agent runs without anyone watching it, go find the numbers in your summary that could be produced by two different causes. Zero is almost always one of them. Then make the producing step say what it did, not just what it left behind. An empty queue should have to explain itself.
Top comments (0)