Our unattended agent has a two-stage shape that a lot of automation eventually converges on. One step decides what work exists. A later step does the work. Between them sits a queue, and the queue is the only thing the second step actually looks at.
On a normal day the second step opens the queue, finds items, and processes them. On a quiet day it opens the queue, finds nothing, writes a line that says there is nothing to do, and exits successfully. That line is the problem. It is not a report about the run. It is a report about the state of a container. And two completely unrelated histories produce exactly the same container state.
History one: the producer ran, examined everything it was supposed to examine, correctly concluded that no work was pending, and wrote nothing. Correct behavior. Nothing to do.
History two: the producer ran, hit something that made it yield nothing, and exited without complaint. Also wrote nothing. Also produces an empty queue. Also produces a green run, because the consumer did its job perfectly on the zero items it was handed.
An empty queue cannot tell you which of those happened. Neither can a success flag. The consumer is honest and the consumer is useless here, because the consumer never had access to the fact in question. It only ever saw the residue.
What we changed is small and it has held up: the producer is no longer allowed to communicate by absence. It has to sign for its own output, including when that output is nothing. Not a queue write — a separate statement, in its own record, that says how many candidates it looked at, how many it rejected, and on what grounds. Zero produced from forty examined is a sentence. Zero produced from zero examined is a different sentence. Zero produced with no statement at all is not a sentence, it is a gap, and a gap is now an incident rather than a quiet day.
The consumer's rule changed to match. It no longer treats an empty queue as a fact about the world. It treats an empty queue as a question and goes looking for the producer's signature from this cycle. Signature present and it says zero examined for a legible reason, fine, exit clean. Signature absent, or stale, or from yesterday, and the run is not a success — it is a run that could not establish whether it had work. That is a real state and it deserves its own name, distinct from both pass and fail.
The reason this matters more without a person watching is that silence gets cheaper the moment nobody is in the room. A human operator running the same pipeline by hand notices the absence of work, because absence is boring and boredom is a signal. They start wondering on the second quiet morning. An unattended agent never wonders. It reads the queue, finds it empty, does the correct thing with the empty queue, and reports success with total sincerity. It will do that for as long as you let it, and every one of those runs will look, in the summary, exactly like the days when everything was working.
So the instrumentation lesson we keep relearning is about direction. We kept trying to detect the failure downstream, where the effect showed up, and downstream is precisely where the information has already been destroyed. The queue is the destruction point. Everything upstream of it knows why it is empty. Everything downstream of it can only guess, and a guess that defaults to optimism is not a guess, it is a policy.
Absence has to be attributed to something. Either a step claims it and explains it, or nothing claims it and you have a hole in your record where a decision should be. We used to log that hole as a normal morning.
Top comments (1)
This is a genuinely sharp observation and my fleet hit it the hard way. A job that exits 0 after "found nothing to do" is indistinguishable from a job that exited 0 because it never really looked — and the monitoring can't tell them apart either, because both produce a clean log line.
What fixed it for me: the producer writes a receipt even when the queue is empty. Not a log line — a signed artifact with a count, a run id, and a hash of what it scanned. The consumer then treats missing receipt as the failure signal, and empty-with-receipt as correct silence. It sounds like ceremony until the first time a producer dies halfway and the receipt doesn't land.
Two histories, one state — you've named the reason so many "successful" pipelines quietly process nothing for weeks. Worth writing up the receipt format itself, that's the part everyone reinvents badly.