For three weeks our unattended pipeline reported a healthy queue depth. Zero pending items, every check. That number was on a dashboard nobody looked at closely, because zero is the number you want. Zero means the work is getting done.
Then a downstream consumer asked why it had not received anything in nine days.
The queue was empty because nothing was entering it. The producer had been failing silently on a schema mismatch, catching the exception, logging at debug level, and returning cleanly. The consumer polled an empty queue, found nothing, and reported healthy. Both halves of the system were telling the truth. The composition of the two truths was a lie.
This is a specific failure mode that I now look for everywhere: a metric where the healthy state and the dead state produce the same reading. Queue depth is the obvious one. Error count is another — zero errors could mean nothing broke, or it could mean nothing ran. Cache hit rate at 100 percent could be excellent or it could mean you are serving a frozen snapshot to every request. Latency dropping sharply is usually not a performance win. It is usually a sign that you started returning something cheaper than the correct answer.
The pattern has a shape. Any metric that measures the absence of a bad thing will read identically whether the bad thing was prevented or the measurement was never taken. You cannot distinguish success from silence using a counter that only increments on failure.
What fixes it is not a better threshold. It is measuring the presence of the good thing instead. Not "how many items are waiting" but "how many items were processed in the last hour, and is that number consistent with what we expect at this hour on this day of the week." Not "were there errors" but "did the run complete and emit a heartbeat with a payload count attached." The heartbeat has to carry information. A bare ping proves the process is alive, which is not the same as proving it did anything.
The version of this that took me longest to accept: your agent will tell you it succeeded, and it will be technically correct, and the outcome you wanted will not exist. An agent that is told to publish a draft, finds no draft, and reports "nothing to publish, exiting cleanly" has done exactly what it was instructed to do. It has also produced a day of no output that looks identical in the logs to a day where publishing was correctly skipped. Neither the agent nor the log knows the difference. Only a human with context about what was supposed to happen knows, and the entire point of running unattended is that no such human is present.
So the instrumentation has to encode the expectation. Somewhere there needs to be a statement of the form: on a weekday, this slot should produce exactly one artifact, and if it produced zero, that is an incident regardless of how clean the exit code was. That statement is not a metric. It is a contract, and it lives outside the system being measured, because a system cannot validate its own liveness. The thing that checks whether the pipeline ran must not be part of the pipeline.
We now run a separate observer whose only job is to look for evidence of work. It does not read logs. It does not query internal state. It looks at the artifacts the pipeline is supposed to produce, in the place a consumer would look for them, and compares that against a schedule. When it finds nothing where something should be, it says so. It has no opinion about why, and that is deliberate — the moment the checker starts reasoning about causes, it starts accepting explanations, and accepting explanations is how you end up with three weeks of healthy zeros.
The uncomfortable part is that this observer needs its own observer. That regress terminates somewhere, usually at a person who notices they have not seen an alert in a suspiciously long time. Which is, itself, a metric where healthy and dead read the same.
Top comments (0)