Last week I wrote about our agents fabricating "done" five times in 17 days and the boring external checks that reduced it. This is the embarrassing sequel: one of those external checks — the guard itself — was dead for about 23 days, and we read its silence as good news.
Nobody fabricated anything this time. That is exactly what makes it worth writing down.
The setup
We run a small operation where AI agents do most of the execution and a human owns the decisions. One of our defense layers is a stop hook: a script that runs at the end of every agent turn and warns about known drift patterns — the agent presenting an option menu instead of deciding, asking "shall I start?" instead of starting, misusing tables, unanswered peer messages going stale, a few dozen more. It is the layer that catches behavioral drift before a human has to.
It had been quiet since around June 18. We noticed the quiet, and — this is the honest part — we interpreted it as discipline improving. Three weeks of zero warnings felt like the rules were finally sticking.
How it actually died
The hook runs under a timeout: 10 seconds, set when the hook was small and fast. Over months, the hook grew — more checks, more files to scan, a board directory that kept accumulating state. By mid-June its real runtime had crept past the limit. When we finally measured it, the hook took 260 seconds. The timeout was still 10.
So every single turn, the harness started the hook, waited 10 seconds, and killed it. No warning output, no error surfaced in the flow we actually read. A killed guard produces the same visible result as a clean pass: nothing.
That is the structural point. In the previous post I called the ugliest failure class success-shaped emptiness — exit code 0 with a zero-byte artifact. This is its mirror on the monitoring side: a dead guard is indistinguishable from a healthy world. Absence of warnings is what "everything is fine" looks like, and it is also what "the instrument is unplugged" looks like. Nothing in between distinguishes them unless you build the distinction.
How it was found
Not by us noticing drift slipping through. A routine harness health check (Claude Code's /checkup) listed the hook as having timed out 15 times. That number was the first loud signal in three weeks — and it came from outside our own defense stack. We then measured the hook standalone, got 260s vs the 10s limit, and the "discipline is improving" story collapsed in about a minute.
Worth sitting with: we preach "re-derive state from the world, don't trust narrative" to our agents, and we had been trusting the narrative zero warnings = good behavior for 23 days without once measuring the instrument that produced the zeros.
The fix — and the two real bugs the fix almost shipped
The speed fix itself was mechanical: switch the hook's shell wiring from login shells to plain ones (startup 3.6s → 0.85s per invocation), batch dozens of per-file process spawns into single passes. Runtime went from 260s to 6–9s. Done, right?
We have a standing rule from the last post: a checker earns trust only after you deliberately try to break it. So the repaired hook went to an independent adversarial QA pass instead of straight to production. That pass found two real P1s in the repair:
- Locale flip. The sped-up scripts inherited the environment's locale instead of pinning it. Under a different locale, date parsing shifted and 5 of the hook's verdicts flipped — same input, different judgment. The original slow hook had masked this by accident. Fix: pin the locale explicitly in all 12 hook scripts, so judgments are deterministic regardless of what shell profile the harness happens to use.
- Zero margin. The fix left the timeout at 10s with a 6–9s runtime — a guard that dies again the moment the board directory gets heavy. Fix: timeout to 30s, roughly 3.7x margin over the heaviest measured state.
Verification was three-way: the implementer re-ran the suite, we re-measured runtime independently (6.3s across three runs), and the QA agent re-ran its adversarial corpus — 30 cases across 3 locale configurations — clean. Verdict logic, warning text, and exit codes unchanged; only performance and determinism moved.
Same day, part two: the guard woke up in a world that had moved on
Hours after the revival, the hook fired its first warnings in three weeks — and two of them were false positives. While it slept, our message-file conventions had drifted: frontmatter written as a dash-list, which the hook's parser predated. The parser read "no reply-needed flag" where a human read "reply not required."
A guard that sleeps through change doesn't resume where it left off; it wakes up wrong. The parser got fixed the same day (one regex), but the general lesson stands: downtime for a guard is not neutral. The world keeps moving, and the guard's model of it silently expires.
What we changed structurally
Four rules we are keeping, in the order we'd install them:
- Monitor the guard, not just with the guard. Guard runtime and timeout/kill counts are now things we look at, not things we assume. The signal that saved us came from a generic harness health check — that layer is now part of the routine, not an accident.
- Timeouts set at install time expire silently. Guards get slower as the system they watch grows. A margin that was 10x at install was 0.04x three weeks ago, and no alarm marks the crossing. Re-measure the instrument on a cadence, or give it enough margin to survive growth.
- Loud degradation. The repaired hook now emits an explicit warning when one of its own internal steps fails, instead of silently skipping it. A guard must be able to say "I am not able to guard" — silence has to mean clean, never broken.
- Repairs get adversarial review, same as new code. The speed fix looked trivial and carried two verdict-affecting bugs. If the guard is worth having, its repair is worth attacking.
The meta-lesson connects back to the fabrication post. We built external checks because agent self-report can't be trusted as evidence. Then we trusted the checks' silence the same way we'd been refusing to trust the agents' prose. Verification infrastructure is subject to its own rules — all the way down, including the layer you just fixed.
We package our working completion-truth checks (bash + PowerShell), the three-state status contract, and a 7-day rollout order as a small kit — linked from my profile. But as with the last post: the fixes above are described completely enough that the post may be all you need.
Top comments (0)