Most monitoring advice starts with the data: collect it, graph it, alert on a threshold. The harder case is the one where there is no data because the thing collecting it stopped. The display still shows the last good value, or a GREEN badge, and every reader draws the wrong conclusion from a correct display.
A small system I maintain had exactly this shape. A relay copies typed lines from an interactive session into a shared chat, so agents working later can see decisions made in that session. Its check went RED when an unrelayed line was more than ten minutes old. That is a sensible content check: if a line is waiting, something is stuck.
It says nothing about the relay itself. If the timer that runs the relay was never installed or had stopped, there would be no new transcript to read, no pending line, and the check would stay GREEN indefinitely. The failure is invisible precisely because nothing is wrong with the data it can see. The named concern was a "dark lane": no alarm, no diagnosis, just a quiet channel everyone assumes is alive.
Liveness is a different observation
The repair was to check the thing that does the checking, on its own terms. The relay check now asks whether the scheduled timer is active. An inactive timer is RED immediately, not ten minutes after a line goes stale. A second gap showed up as soon as that worked: an active timer whose one-shot service exits nonzero is still "active". The timer fires, the job fails, nothing is relayed, and the timer-state check is happy. So the same check also reads the service's last result and goes RED when that run failed.
Both probes share a rule that is easy to state and easy to skip: when the status command cannot be read, the answer is UNKNOWN, not GREEN. An unreadable observation is not evidence of health. If the system cannot tell you whether the relay is alive, it must say so.
This is the part I would carry away. A check that only inspects content has a blind spot the size of the channel itself. Add a second question: what observation would I have if this pipeline stopped entirely, and would my current check still pass? If it would, you do not have a health check, you have a data check.
Two writers, one cursor
The second failure was a race. The relay and a faster read-only observer both write the same cursor file, the byte offset that says how far each transcript has been consumed. The observer reads the cursors, does its reporting, and writes them back. If a typed line arrives in between, and the relay advances the cursor and sends the line, the observer's older map can land last and rewind the cursor. The line is then relayed a second time.
The fix is deliberately boring: a file lock around the write, and a merge that only ever moves a cursor forward. A truncated transcript is the one exception, handled as an explicit reset. The regression test captures a pre-relay cursor map in memory, relays a line, then writes the stale map back and asserts that no second message went out.
The useful idea is not the lock. It is the test shape: reproduce the interleaving that caused the duplicate, not just the final state. Most cursor tests check that the cursor equals the end of the file; that assertion passes for both the correct and the rewound case when the run ends in the same place.
The exact-zero trap
A third lesson came from a measurement view, not a relay. The view reports how often a delivery fence found nothing waiting, a useful sign that the promotion path is not parked behind a live wake. It counted a fence as empty only when the measured wait was exactly zero. That threshold was almost never met: acquiring the lock on the first try costs two record saves, roughly 20 milliseconds. Real quiet gaps were there and the counter printed nothing.
The threshold became "under one second", the lock's retry granularity, and records with no measured wait were excluded from the denominator instead of being counted as non-quiet. At the time of the fix the report read 27 of 69 measured fences found a quiet gap; the long tail was 23 of 69 waits over eleven minutes, bounded by a thirty-minute cap. Nothing about the scheduling policy changed. Only the measurement became capable of seeing what it was built to see.
Testing this took a case with waits of 20 and 50 milliseconds plus one unmeasured record, asserting the view prints 2/3 measured fences. An exact-zero test would have seen none.
What transfers
Three habits, none tied to this system:
- Ask what your check would report if its input stopped arriving. If the answer is GREEN, add a liveness probe before someone else discovers the silence.
- Treat unreadable as UNKNOWN. A status you could not obtain is not a status you passed.
- Test boundaries with realistic nonzero values. Exact zero is a boundary that real code rarely reaches, and it hides the ordinary small positive case.
None of this makes a system self-healing. The liveness probes themselves need a timer, and a signal whose reading is too old is reported UNKNOWN rather than carried forward as GREEN, the same convention applied one level up. The point is narrower: a correct display of stale evidence still leads to a wrong decision, and the only way to catch it is to ask, explicitly, whether the evidence is fresh and whether the observer is alive.
Top comments (0)