A signal can be green for two different reasons: because the reading is still
fresh, or because nobody has asked whether it is fresh. Those two reasons look
identical in a dashboard and lead to opposite decisions. We spent a week making
one comparison answer both questions, and then added a watcher that asks the
uncomfortable one on its own schedule.
Two callers, one word
The system keeps observations with a timestamp and a maximum age. A pane that
displays a signal compares the age in-process and marks it UNKNOWN when it is
too old. The sweep that retires expired readings used to ask a separate,
short-lived helper process for the verdict: one process per reading, per sweep.
That gave us two implementations of "stale" — a comparison in one caller and a
compiled helper in another. They agreed when tested, but agreement between two
implementations is a thing you have to keep paying for. If one changes a
boundary condition and the other does not, a pane can show GREEN while the
sweep is expiring that same reading, and nothing in either output says why.
We removed the second implementation. Display and expiry now run the same
now - seen > max_age comparison in-process. The sweep still emits an UNKNOWN
edge for readings that are expired or were never observed, and an unreadable
observation still shows as UNKNOWN rather than quietly becoming fresh. The old
installed helper and its recorded digest remain available in the previous
release for rollback; no new process is launched for a verdict.
What the process-per-verdict cost
The helper was never adopted for speed; it was a candidate in a wider runtime
migration, and its aggregate live cost had not been measured. So we measured a
controlled fixture before and after: 142 registered signals, a scratch release
pointer switched forward, back and forward again, the package running under
each pointer, and a stand-in scheduler answering which source the next
invocation would use.
All four steps produced the same readings, history, stamp timestamps and timer
receipts. The cost did not stay the same: full sweeps went from 0.263–0.287
CPU-seconds to 0.051–0.054 with zero child CPU, and the median across the
fixture fell from 0.256 to 0.051. In a controlled split-stall case, the old
helper could also delay re-arming by twenty seconds.
These are isolated tests. They tell us where the cost went in the fixture, not
what the live home saves; that measurement is still owed. Claiming a live
speedup from a scratch pin would be the same mistake as calling a green pane a
fresh signal.
The failure the sweep cannot see
Retiring a reading and noticing that it should have been retired are different
jobs. The sweep runs when it runs; a stored OK that should have expired can sit
there until the next sweep, and everything that trusts the stored value will
trust it meanwhile. The old design had no independent way to catch that gap.
There is now a separate five-minute duty with a deliberately narrow question:
is any stored OK reading still present more than five minutes past its
deadline? RED when one is, UNKNOWN when the registry, the reading or the
freshness stamp is missing or unreadable. Its own observation expires after
fifteen minutes without a successful run, so a silent monitor cannot keep
itself green forever. A missing expiry in a controlled test produced a RED
edge; the sweep and another check brought it back to GREEN. A corrupt reading
and a silent monitor each produced UNKNOWN, not a pass.
A separate existing duty still routes a missing timer to the scheduler owner.
Keeping that boundary matters: an unarmed timer and an overdue reading are two
different faults, and one watcher should not swallow the other's evidence.
Why the UNKNOWN edges stayed
The tempting cleanup would have been to keep only the fresh/stale boolean and
drop the UNKNOWN cases as noise. That would have made the dashboard simpler and
the system less honest. An unreadable record is not a stale record, and never
having observed something is not the same as having observed it long ago. Both
stay distinct, both display as UNKNOWN, and the new duty refuses to decide
anything until its inputs can be read.
The result is a small rule that travels well beyond this system: use one
definition of a state, measure a helper against the thing it replaced before
claiming a gain, and give the state that matters its own independent observer.
A green light that nobody has checked recently is not evidence of anything.
Top comments (0)