Here is a failure I had not thought about until someone asked me the right question.
You write a PreToolUse hook. It reads the tool call, decides the command is dangerous, and blocks it. You write unit tests for it, because a guard with no tests is a wish. The tests pass. Months go by and the hook never fires, which you read as good news, because nothing bad has happened.
Then one day you find out the harness renamed a field in the payload eighteen weeks ago, your hook has matched nothing since, and every one of your tests is still green. They're green because the test builds the payload itself, in the shape you believed the harness sends. The test and the hook agree with each other perfectly. Neither of them has spoken to the harness in months.
Green tests, dead tripwire. The guard and its tests share a belief, so the tests cannot see the belief go stale.
This is the same shape as a bug I wrote about last time, where a redaction tool verified its own output using the same wrong geometry it had used to draw with. Different domain, identical structure: a check built from the same assumption as the thing it checks is not a check, it's a second copy of your belief.
Why "it hasn't fired" tells you nothing
A guard has two states that look identical from the outside. Working and quiet, because nothing tripped it. Broken and quiet, because it cannot match anything any more.
Nothing in your workflow distinguishes them. You don't get a weekly report of hooks that ran. The absence of refusals feels like safety, and in the broken case it is precisely the opposite: you've been running unguarded while believing you were covered, which is worse than having no hook at all, because you'd have been more careful without one.
Every guard that encodes a fact about an external world has this problem. The payload shape of your agent harness. The name of a tool. Whether CI still runs the check your hook assumes runs. Those facts change without telling you, and the guard goes quiet rather than loud.
Fail closed, but not everywhere
The obvious fix is to fail closed. If this is a tool I'm guarding and the field I need isn't there, that's an anomaly, not an allow. Most hooks, mine included, default to allowing anything they can't parse, which is exactly backwards.
The question I got asked, and did not have a good answer for at first, was whether that means failing closed everywhere, or only for guards on irreversible things.
I now think irreversibility is the wrong axis. The right one is what happens when the hook declines to decide. Three modes, three correct failures:
Block mode: fail closed. The hook is certain from the input alone that the call is wrong. If it can't read the input, refuse. The fallback is a retry or a question, both cheap, and the false block you eat on the day the harness changes is the thing that tells you the shape moved. That one annoying refusal is the monitoring you never set up.
Warn mode: don't fail closed, and say why. A warn hook exists because the rule is a heuristic that will fire on legitimate work, so turning an unparseable payload into a wall contradicts its own design. Emit the warning anyway. And the detail I'd have got wrong: the warning has to say precisely that it couldn't read the input, rather than printing its usual generic text. A warning that can't name the file or the command, but talks as if it can, teaches the model to discount every future warning from that hook.
Auto-allow mode: already fails closed, for free. This is the mode nobody discusses. Hooks that pre-approve read-only commands so they never surface a permission prompt. If one can't parse the input, it simply doesn't allow, and you land on the normal prompt. The fallback is a human, which is exactly where you want to be.
So the rule I'd write now: fail closed wherever declining costs a prompt. Don't, wherever declining costs a block on legitimate work.
The fixture that lies to you
The other half is contract tests, and there's a trap in them.
Recording one real payload from an actual session and replaying it against the hook catches the rename before it costs you anything. Writing that fixture by hand does not, because a hand-written fixture is your belief about the payload shape stored in a second file. You now have two copies of the same assumption and a false sense of coverage. If you can't capture a real one, log the raw input on every invocation and go read it occasionally. Dull, works.
My own setup, honestly: two hooks have real unit tests, one with a table of about thirty adversarial command variants, and both assert the deny path rather than only the happy one. Neither has a recorded fixture. I haven't fixed it yet.
What I still don't have
The thing I'd actually want is a canary. A scheduled job that deliberately issues a known-bad call and asserts it gets refused, so a dead guard announces itself instead of waiting to be discovered. That's a cron and an assertion, it isn't hard, and I haven't built it, which I suspect means most people running a dozen hooks haven't either.
Most of this post came out of the comments on my last one rather than from me. @build996 pushed on which refusals an agent can actually act on, @max_quimby on hooks quietly becoming an unversioned rulebook nobody remembers the reasons for, @syntaxwanderer_26 asked the question that produced the whole fail-closed taxonomy and then supplied the warn-mode detail, and @doriansotpyrc asked how a rule earns promotion from prose into a hook, which I answered as: once means write the sentence, twice means the sentence isn't carrying it.
So, the part I'm stuck on. How would you detect a guard that has been dead for a month? A canary is the only honest answer I've got, and it only covers the failures you thought to simulate. If you've found something better, or you've had a tripwire die quietly and only noticed by accident, I'd like to hear it.
Top comments (18)
Good to see the thread grow into a post. The canary is the piece we never got to in the comments.
On detecting a guard that's been dead for a month, I'd count rather than simulate. A canary proves the guard still refuses the one call you scripted. It can't see the guard quietly losing its grip on real traffic. And logging inside the hook isn't enough either: if the matcher stops matching, the hook is never called and logs nothing, which looks exactly like a quiet week. So compare what the hook says it saw with a record the hook doesn't produce. If the remote shows twelve pushes this week and the push guard saw zero, it's dead, and you found out without guessing what the failure would look like.
It's your redaction bug turned around: the check has to come from a source that doesn't share the guard's belief. The VCS, CI logs and the harness transcript qualify. The hook's own fixtures don't.
Semitexa's
ai:verifyfollows the same rule for its own empty runs: when no check matches the change, the verdict isnothing_to_verify, not a pass, because "nothing ran" and "everything passed" must never look the same.Is there anything in your harness today that would give you that independent count, or would you have to build it?
Counting rather than simulating is better than what I had, and the reason is the part I'd missed: a canary only proves the guard still refuses the call I thought to script.
To your question, it turns out I already have the independent record and hadn't noticed it. The harness writes a JSONL transcript per session, and every tool call lands in it with its name and input, written by the harness rather than by any hook. I counted one of mine just now: 507 Bash calls, 328 edits, 186 writes. Hook refusals show up in there too, so both halves of your comparison come from the one source that doesn't share the guard's belief.
What makes that better than instrumenting forward is that it's retrospective. There are 594 of those transcripts sitting on my disk. So I can ask today whether the pooler guard has seen a matching command in the last month and find out it died in September, instead of adding a counter and waiting a month to learn the same thing. The audit I thought I'd have to wait for is already in the logs.
And agreed on the empty-run verdict. "Nothing ran" and "everything passed" being indistinguishable is the whole post.
And thank you for the comment, this is the most insightful comment I've had on this site.
Thank you, that means a lot from the person who wrote the post. The transcripts are a better answer than mine: written by the harness, retrospective, and already on disk, so the audit costs one script instead of a month of waiting. I'm curious what the count says for your pooler guard. And your thread already changed something on our side: today I filed a task to move our ai:verify gate out of the instructions file and into a Stop hook, with your "fail closed only where declining costs a prompt" rule and version-stamped recorded fixtures written into it.
I ran it. 599 transcripts, 81,200 Bash calls since July, 1,380 of them touching a pooler or RDS host, which is the traffic that guard claims to cover. 170 in the last full week, 82 this week, so the denominator is alive and current.
Then the useful part: I have a denominator and no numerator. The hook refuses things without recording that it refused them, so I can't complete your comparison today. The audit I was pleased with measures the half I already trusted.
One thing the series taught me that I'd have got wrong on my own. Two weeks in October show zero matching commands, which is exactly the shape I'd have read as a dead guard. They're zero because total Bash calls those weeks were also zero, since I wasn't working. So a per-hook count has to be read against total tool traffic, or a holiday and a dead tripwire are the same picture. Same mistake as the post, one level up, in the detector I built to catch it.
Good luck with the Stop hook. I'd be interested to hear whether fail-closed holds up once it's someone else's CI eating the false block. Hope you write a post about it.
The 'two states that look identical from the outside' line is the whole problem — a quiet guard reads as a working guard, and the broken case is worse than no guard because you'd have been more careful without one. Your warn-mode detail is the one I'd steal: a warning that can't name the file or command but talks as if it can teaches the model to discount every future warning. On the canary — have you tried making the scheduled bad call part of the hook's contract test, or does that just move the staleness problem one layer up?
It moves it, and I think I can say where to.
A contract test runs the hook against a payload the test runner hands it. If the harness renames a field, your known-bad call is still in the old shape, so it passes and proves the matcher works against last year's reality. You've tested your code rather than the integration, which is the original failure with a more respectable-looking test wrapped around it.
The canary only escapes that if the bad call goes through the real path: the agent issues it in a real session, the harness builds the payload, the hook refuses it where it normally would. Then the thing under test is the join, which is the part that actually broke.
So contract test for the matching logic, canary for the integration, and they go quiet in different ways. The warn-mode detail you mention I only got right because someone in the earlier thread pushed on it.
That split is the clean answer — contract test proves the matcher, canary proves the join, and neither can cover for the other going quiet. The 'different ways they go quiet' point is the one to watch: when the canary goes quiet, is that itself alarmed, or does it fail the same silent way the original guard did?
One of my gates passed whenever its log file was missing. It enforced the hardest rule in my agent test harness, no live API calls outside the planned captures, by reading the call log. If the file didn't exist it returned True with "absent, so zero calls logged", so a recorder that wrote nothing would have passed. An agent found it while reading the gate code. The fix makes a zero count only when a non-zero next to it shows the recorder ran.
That's the same bug in different clothes, and your fix names the general form better than my post did.
A zero is ambiguous. It means "nothing happened" or "nothing was recorded", and from the inside those are identical. So a zero is only trustworthy next to a non-zero that proves the instrument was alive, which is exactly your check.
I hit it an hour ago counting how often one of my guards should have fired. Two weeks in October showed zero matching commands. Dead guard, obviously. They were zero because I hadn't run the agent at all those weeks, and the only reason I know is a second count sitting beside the first.
Also worth sitting with: your gate was found by an agent reading the code, and mine was found by a GIF I made for a landing page. Neither of those is a process.
The fix that keeps working after the harness changes again: stop building the payload in the test by hand. Record it. Capture the last dozen real PreToolUse calls your harness actually makes — they're small, and nothing sensitive lives in the shape — commit them as golden files, and have the unit test assert the hook matches at least one recent real payload.
Then a harness field rename turns your CI red within a day instead of leaving the tripwire dead for eighteen weeks. The test stops being a mirror of your belief and becomes a diff against the outside world, which is the thing that actually changed.
This composes with the canary idea rather than replacing it: the canary proves the hook still fires at runtime, golden payloads prove the matching logic still describes reality. You want both, because they go quiet in different ways.
Agreed, and "at least one recent real payload" is carrying more weight in that sentence than it looks.
A corpus recorded once and committed becomes the same stale belief with better provenance. Two years of green tests against a dozen payloads from setup day proves nothing about today. So the assertion has to be against payloads newer than some window, which means the recording can't be a one-off, it has to keep running.
Someone in the other thread added the piece that makes it enforceable: stamp each recording with the harness version, and have CI refuse a changed fixture whose stamp didn't move, because re-recording is the reflex that turns a red test green without anyone deciding anything.
And yes on composing rather than replacing. Golden payloads check that the matcher still describes reality, the canary checks the integration still fires. I have neither, so at the moment I'm two good ideas and no implementation.
Same as a smoke detector nobody tests. It stays quiet for months and everyone reads quiet as safe. Pushing the button is the only way to know. We're building that kind of real-world check into ML Systems.
The analogy has a sharp edge that favours your point. The test button checks the siren, not the sensor. You press it, it shrieks, and you've learned the speaker works and nothing whatsoever about whether it can still smell smoke.
Which is the limit of a canary too: it proves the guard still refuses the one call you scripted. The sensor equivalent is counting real traffic against a record the guard doesn't produce.
A harness renamed one field and the tests stayed green because the test built its own payload. A check I'd add is a canary run: on a schedule, feed the hook a payload captured from the real harness rather than one written by hand, and require that a known-dangerous command is actually blocked. I'd also keep a counter of how often the hook matched anything during normal use and alert when it flatlines. That way a quiet hook and a broken hook stop looking identical, and the unit tests no longer have to carry that job alone.
Both halves of that are right, and the thread has turned up a limit on each that's worth knowing before you build them.
On the canary: if the scheduled bad call is handed to the hook by a test runner, it only proves the matcher works on the payload shape you recorded. To exercise the thing that actually breaks, the call has to go through the real harness so the payload gets built the way a real one is.
On the flatline alert: mine would have paged twice in October for a completely healthy hook, because I wasn't working those weeks and every count was zero. It needs a per-hook baseline and a denominator of total tool traffic rather than a global threshold.
@kartik-nvjk and @kami_butt_69b922ade8de012 in this comment section reached those 2 points independently, which probably means they're the real edges rather than my particular mess.
Since the transcripts are already on disk, I'd run the count per hook per week instead of one total. A hook whose matches drop to zero in a single week points straight at the harness update that renamed the field, which matters when the break is eighteen weeks old.
You're right, and I ran it per week within the hour. It dates the break, which a total can't: "it's dead" becomes "it died the week of the 12th", and now there's something specific to go and look at.
One wrinkle I only saw because the data showed it. Two weeks in my series are zero, which is the exact shape of a hook that stopped matching. They're zero because I wasn't working those weeks and total Bash calls were also zero. So the weekly series has to be read as a share of overall tool traffic rather than as a raw count, or a holiday and a dead tripwire look the same. Which is the post's own failure mode turning up one level up, in the detector I built to catch it.