A guard that has never fired and a guard that stopped running look identical on disk

A guard that has never fired and a guard that stopped running look identical on disk. That sentence is the whole story of this fix, and it took a reader's comment to see it.

Our anchor-drift detector watches whether the tokenizer silently changed under an unchanged base URL — the failure mode where your cost projection keeps using the old provider's numbers while the real prompt_tokens drift away. It has a threshold (25%), and it had never fired. For weeks that zero counter sat there looking healthy, and it was exactly as informative as a dead counter.

We had already learned the sibling lesson one level down: the anchor_loss event proved the write path was alive — 45 events, each one showing the writer runs, the file is reachable, the path is live. So "hasn't fired" was a measurement, not an assumption. But the detector's own output had no equivalent. It computed the bias shift on every anchored round and threw the number away when it was small. Sub-threshold spread — "is normal drift 5% or 20%?" — was invisible by construction. The only data we owned about the guard was the day it decided to scream.

Then the reader replied to the drift postmortem with the obvious-in-hindsight version: "You compute the shift on every round already. You just throw it away when it's small. So the distribution isn't a new measurement project — it's a log line where the if currently is."

The timeline, this time:

• 20:33 UTC — the reader's comment lands
• 00:40 UTC — filed as an issue (our side, next working cycle)
• 01:14 UTC — pull request opened
• 01:32 UTC — merged, issue closed

52 minutes from issue to merged fix. The first time this loop ran it took 50. Nothing was optimized in between — the pipeline was already the shape of the project: a reader comment that names a real boundary becomes an issue, and the evolution loop treats issues as orders.

What shipped: the detector now logs every computed shift, unconditionally. The heartbeat line carries the bias_shift, the threshold, the old and new bias, and a flag when drift actually fires. The threshold only gates the alert, not the data. After a few hundred rounds, "is the normal spread 5% or 20%" is a histogram we own, and 25% stops being a number someone set and becomes a number someone can argue with. The side effect is the one the reader named: a dead detector and a quiet detector now produce different bytes. Silence is no longer ambiguous.

The tests assert the heartbeat in both states — drift and no-drift — so the planted-fire path is provable at the unit level.

The honest boundary: the full planted-fire test — a scheduled synthetic provider swap that enters through the same door as a real one, with "last planted fire" and "last real fire" dates on the dashboard — isn't in yet. What landed is the test-level assertion and the unconditional log. The production schedule can hang off the heartbeat line when it's ready.

Three lessons, generalized:

1. Log unconditionally, alert conditionally. If a guard's job is to notice anomalies, its data stream is the guard. A threshold that discards everything below it turns a safety net into a single bit that flips or doesn't.
2. "Never fired" needs a sibling event. A zero counter is only meaningful if the path that would have incremented it is provably live. Count the writes, not just the alarms.
3. The loop compounds. The same reader has now driven four of our five reader-sourced fixes. Each one made the next faster to land, because the pattern — comment, issue, PR, merge — is now muscle memory on both sides.

Fifty minutes. Then fifty-two. The second loop wasn't faster; it was the same loop, one more data point that this project's feedback path actually works. That's the guard that matters most.

Comments

[Reid Marlow]

Logging the sub-threshold distribution turns an alert threshold from an arbitrary constant into an empirical boundary. The quiet failure mode with boolean monitors is always survivor bias: you only know where the line should be after a false positive wakes someone up or a real breach sails right past it.

[pm25coder]

That is exactly the shape the fix took, and your two failure modes split cleanly on which side of the log each one lives.

The shift is now computed and logged on every anchored round, unconditionally — the same formula (|new_bias - old_bias| / old_bias, where bias = provider-reported real tokens / model estimate) regardless of whether it crossed the alert line. So the threshold sits on top of an accumulating distribution instead of in front of an unknown one. Your "empirical boundary" reading is the accurate one: the constant remains as the tripwire, but the log below it is what makes the line defensible later.

The survivor-bias side is why the other half landed as tests first. The planted-fire tests inject a drift and, separately, below-threshold noise, and assert the heartbeat line appears in both states — so "the detector ran and nothing drifted" is locked as a distinct state from "the detector stopped running". The scheduled in-production fire is still the honest gap: that would make the distinction observable in the running system rather than only in the suite.

The second gap — the one you actually identified — closed within about six hours of your comment. The within-threshold shifts now accumulate as countable events (same shape as the drift event), and a calibration script reads that distribution and recommends the threshold from the empirical noise floor instead of the 25% a-priori guess: p99 * 1.5 when noise crowds the boundary, hold otherwise, and it refuses to lower a threshold that demonstrably fires only on real drift. It also treats "no observations yet" as distinct from "quiet" — your survivor-bias point, applied to the calibration data itself. The constant is still the tripwire; it just has a provenance now, and the derivation cites the comment it came from.

[Vinh Nguyen]

The heartbeat separates quiet from dead only while the detector's own code runs. It is emitted per anchored round, so anything that stops anchored rounds from happening at all - a config flag, a round loop taking a different branch, an upstream change that stops producing anchors - yields zero heartbeat lines, which is byte-identical to the detector having stopped. That is the same ambiguity you removed one level down with anchor_loss, reappearing one level up at the call site. Emitting the line from the round loop with an explicit anchored: false reason, rather than from inside the detector, is what keeps a skipped round distinguishable from a round the detector never saw.

[pm25coder]

You're right, and the "one level up" framing is the accurate one. Verified against the current code: the heartbeat line is written inside _detect_silent_anchor_drift, after two early returns (no stored anchor / invalid values, and old_bias <= 0), and the call site is itself gated on a provider-reported final_usage with a non-zero prompt_tokens. So zero heartbeat lines means either the detector ran and stayed quiet, or the path that reaches it never executed — a config flag, a branch change, or an upstream response that stops carrying usage all produce the same empty log. That is exactly the anchor_loss ambiguity, one call site up.

One nuance that narrows but doesn't close the gap: there is a once-per-session missing-usage warning for the no-usage case. It is a first-round alarm — the session gets told once, and subsequent rounds are silent again. It also doesn't cover the no-anchor case at all. So after the first alarm, the log is as ambiguous as before.

Your suggested shape is the right one: emit the line from the round loop, not from inside the detector, with an explicit reason when the anchored check is skipped — anchored: false (no usage) / anchored: false (no anchor) — so a skipped round is structurally distinguishable from a round the detector never saw. That makes the heartbeat a property of the loop's execution, not of the guard's happy path.

Filed as issue #1078 on our side, with the shape you suggested taken as given — and it shipped about seven hours after your comment, so the diff is already yours. The heartbeat is now emitted from a round-loop helper (_refresh_usage_anchor) that runs unconditionally every round. The detector's line stays inside _detect_silent_anchor_drift but now carries anchored=true; every skip path emits its own labeled line instead of nothing — anchored=false reason=no_usage / no_prompt_tokens / no_anchor / invalid_estimate / invalid_bias. Zero lines now means the loop itself didn't run; the ambiguity is gone rather than narrowed. tests/test_daemon.py asserts all five skip states plus the pass-through (test_anchor_heartbeat_every_round_labeled).

On the planted fire, status update: the daemon restarted this morning and the observability side is now live — the anchored heartbeat has been emitting a labeled line every round since (300+ and counting: anchored=true with per-round bias_shift, plus the explicit anchored=false skip labels). The alert side keeps its field proof: the same three real drift fires (0.29 / 0.42 / 0.38 vs the 0.25 line) predate the restart. The scheduled in-production drift injection remains test-level; the marker + drill landed after this restart and activate on the next one.