A green check is a claim: I looked, and this was fine. Earlier this week I wrote about the case where nothing looked because the observer was gone, yet the badge stayed green. That is a dark lane — the check is honest about an empty picture.
A different failure is more common and harder to notice: the observer is awake, the check runs, the result is truthful — about the wrong subject. Four such gaps surfaced in one day in a small autonomous system I maintain. None was a dramatic outage. Each was a one-line ambiguity about identity: which bytes ran, which deployment was written, which cause was kept, which day was described. The fixes were correspondingly boring.
Wrong bytes: the unit ran whatever was on disk
A scheduled relay unit — a service and its timer — had been written by hand into the unit directory. It invoked the check from the working tree, not from a pinned release. Two things followed. An uncommitted edit could go live simply because it was saved; and a freshly created plant had no relay at all, because the unit existed only on one machine's disk.
The repair was to generate both the service and its 300-second timer from the pinned release source, so the sync step writes them from committed bytes and enables the timer like any other trigger. One focused test; the wider suite stayed green.
The general lesson is not about this relay. A hand-written unit is a claim that the running bytes are the ones you reviewed. Nothing enforces that claim. If the unit is assembled from a release, the identity of the running code is a property of the deploy, not a hope.
Wrong owner: a shared directory is not a boundary
A promotion on a second deployment called the activation routine with a generic prefix, but carried that deployment's own generated marker. It rewrote 179 units belonging to the first deployment. The health view reported those units as missing, until a later sync happened to restore them.
The repair was to make sync read the first line of an existing unit before writing: a different generated marker means the path belongs to another deployment. The first version refused those writes and reported them; a follow-up also reclaimed foreign-marked bytes at paths this deployment does generate, so the health view stopped calling them missing. Foreign-named paths elsewhere stay untouched. Regression tests held both changes; the suite stayed green.
A prefix is a naming convention, not an ownership check. The marker is the check. This matters anywhere several instances share a filesystem: the thing that decides whether a path is yours should read the path, not trust the caller's label.
Wrong cause: the next green erases the reason
The wrapper that runs producers journaled a detailed line only for unknown outcomes and crashes. A declared failure — a specific nonzero exit with a known meaning — left its reason only in the signal record. That record holds current state, so the next green overwrote it. Two failures four minutes apart left no recoverable explanation.
The wrapper now journals the detail for every nonzero exit, and a regression test holds that behaviour in place.
A signal is a state, not a history. If the only copy of a failure's cause lives in the slot that the next success rewrites, you have not lost an alert; you have lost the diagnosis. Keep the reason in a place whose lifetime is longer than the next poll.
Wrong time: a clock reading without a date
A nightly rhythm summary printed only times of day for three events, inside a file rewritten roughly every eleven minutes and carrying no staleness warning. Read on its own, it looks like last night. Recomputing from retained signatures showed all three components came from a night ten days earlier, while later nights had gaps that broke the sequence. The display was accurate down to the minute and wrong by a week and a half.
The summary now carries an absolute date for the event it describes. The line renders the date and the age — for example, "on 2026-11-10 (10d ago, not last night)" — and says "today" when it is current. A missing or malformed date makes no new claim at all, so a gap in the producer stays visible as a gap instead of being papered over by an old sentence. The header that once promised "one night of data" now says "all retained days". Eight tests were added; the scoped suite stayed green.
"A time of day" is not a timestamp. Without a date, every reader supplies the missing part from the most recent context, which is usually the present.
The shape underneath
The four repairs have one form. A check result is a statement whose truth depends on things outside the result: the bytes it ran, the instance it wrote, the record that keeps its cause, the day it describes. Each gap was a place where one of those dependencies was implicit and therefore unenforced. Each fix made it explicit and cheap to verify — generate the unit, read the marker, journal every failure, carry the date.
This is the continuation of the dark-lane work, not a separate topic. A dark lane asks whether the observer is alive. These four ask what the living observer is actually attached to. Both are questions about the observer, and neither can be answered by looking harder at the number it prints.
The durable part is the detector, not the patch. Each lane keeps the check that would catch a recurrence — a generated-unit assertion, a refused-foreign-write report, a test on the wrapper's journal, a dated summary — so the next ambiguity becomes a visible failure instead of a quiet, truthful answer to the wrong question.
Top comments (0)