Recall is not enforcement

The rule I have had to learn more than once is four words long, and it is written into 103 files across my agent setup:

Rules are NOT learnings — a non-negotiable rule needs always-loaded text + a deterministic hook, never a memory store (recall ≠ enforcement).

That much is not what I want to argue, because anyone who has watched a memory store fail already agrees. Retrieval is probabilistic and depends on the model deciding the rule is relevant right now. Fine for context, not a control. So you write the rule into always-loaded text, you build a mechanism that can exit non-zero, and you audit the pair: text present, mechanism present, mechanism wired.

That audit passes on rules that are not enforced. Both halves can exist, name each other, be correctly wired, run on every invocation, and the rule still does not hold. That is the claim, and it took a measured failure in my own system to make me believe it.

The two halves, briefly

Neither half substitutes for the other. Always-loaded text is in context every time without anything deciding to fetch it, and it explains the why, which is what lets an agent apply the rule to a case the mechanism never anticipated. A deterministic mechanism (a hook, a lint, a pre-push check, an exit code) holds when the text has not been re-read, or has been read and rationalised away. Text alone is a preference. Mechanism alone produces the worst kind of workaround: the one that satisfies the check and defeats the intent.

My ownership rule states this out loud:

This file is the always-loaded half; the hook is the half that holds when this file has not been re-read. Per CLAUDE.md, recall is not enforcement — which is why the rule is not a memory entry.

The other half is a 192-line hook that exits 2, wired by name in the settings file. Both halves exist and each names the other. That is the model working, and it is the setup, not the point.

A mechanism can implement the shortcut its own rule forbids

I have a quality rubric with an explicit rule: titles are a cheap pre-filter, never the decision. Written down, always loaded, in the right place, in plain language.

There is also a mechanism. It decides whether a topic has already been covered by computing lexical overlap between the topic title and the corpus title, then treating >= 0.8 as consumed.

That is a title comparison used as the sole decision, precisely what the rule forbids, implemented by the code that exists to enforce the rule. Not hypothetically: on one queue it passed three topics carrying a single claim, because their titles barely overlap, while one of the three sat approved and queued.

Nothing was missing. The text was there, the mechanism was there, it ran, and it returned a wrong answer with no error. Every box the standard audit checks was ticked. Until that function escalates to judging claim shape instead of comparing titles, the honest disposition is the one now written next to it: that decision is a human judgement, and the automation is not trusted to make it.

A reasonable person answers that this is a bug, not a category: fix the function and the model is intact. I do not think that survives the next section. Nobody caught it because the audit everyone runs cannot see this class of defect, and it is wrong in the other direction too.

The wiring check fails in both directions

The obvious audit is to grep the settings file for your hook and confirm it is there. It fails both ways, and the census below measures both: thirteen of my hooks are armed and appear nowhere in that file because a parent dispatches them, and three appear in it while being unable to refuse anything today.

So the real check has three questions, in order:

1. Does the always-loaded text exist, and does it say why?
2. Does a mechanism exist, and is it actually reachable from the runtime, dispatch chains included?
3. Has the mechanism ever been observed rejecting the thing it exists to reject?

One and two are the ones people run, and they are exactly the two that cannot tell an enforced rule from a decorated one. Three is the only one that produces evidence. Feed the mechanism the forbidden input and read the exit code. A gate you have never seen refuse is a gate you are assuming.

None of this is my idea, and one authority disagrees about where it belongs

Palantir published it in 2017. Their Alerting and Detection Strategy Framework makes it a mandatory section of every detection they write:

Validation are the steps required to generate a representative true positive event which triggers this alert. This is similar to a unit test and describes how an engineer can cause the ADS to fire.

Google was doing the same thing for alerting rules a year earlier. The SRE Book's Borgmon chapter, chapter 10 and not the testing chapter people usually cite for this, describes synthesising input so a rule can be watched firing, then running those tests in CI before the config ships. And NIST has named the distinction for years. SP 800-53A Rev. 5 defines Examine as "checking, inspecting, reviewing, observing, studying, or analyzing" one or more assessment objects, and Test as "exercising one or more assessment objects under specified conditions to compare actual with expected behavior". They are peers, and a control can be declared effective on Examine alone.

So the idea is old. What is missing is anyone publishing the ratio — of the controls that pass Examine, how many fail Test.

Where I do disagree is with a vendor, about positioning, and the vendor is the one whose runtime I am auditing. Anthropic's own configuration debugging guide tells you to run /hooks to see what is registered, and then:

If the hook appears but doesn't fire, the matcher is the usual cause.

Behavioural proof does appear on that page, further down, via claude --debug. But it appears as debugging — the thing you reach for once you already suspect something is broken. It is never the routine check. That ordering is the whole disagreement: registration is presented as the audit and refusal as the troubleshooting, and my numbers below say it should be the other way round.

The protocol, borrowed rather than invented

Security teams already have a runnable version of this. Red Canary's Atomic Red Team is a library of tests you fire at your own environment, and its getting-started procedure is six steps that reduce to one idea: execute the thing the control exists to catch, then "use the data collected by your security solution to evaluate and improve your detection coverage."

I ran that shape against guard hooks instead of against detections. One substitution matters and it is not cosmetic: Atomic Red Team really executes the attack, and several of my hooks exist to block reads of credentials. Executing the forbidden command to see whether the guard stops it is how you leak the thing you were testing the guard for. So every probe is delivered to the hook on stdin and only the exit code is read. The command never runs.

What I could not borrow is a checklist for this specific case. I looked through the SRE book and workbook, Principles of Chaos, OPA and Conftest, Atomic Red Team, the Palantir framework, NIST 800-53A and 800-115, and the detection-engineering literature. There is no published protocol for auditing the guardrails of an agent harness. That absence is not a gap in the reading. The practice is old and well documented; what I could not find written down anywhere is it pointed at this target. So what follows is someone else's method aimed somewhere new, and the numbers are what came back.

So I ran question three on all ninety

Writing that down obliged me to answer it, which I had never done. I enumerated every hook wired in my harness, resolved the ones that are dispatched as children rather than named directly, then fed each one the input it exists to reject, on stdin, and recorded the exit code.

Ninety hooks. Seventy-seven named in the settings file, thirteen more reachable only through a dispatch parent.

	count	what it means
wired hooks	90	77 named, 13 reachable only via a dispatch parent
advisory by design	59	no refusal primitive anywhere in the file
able to refuse	31	by hand; an automated scan says 34, see below
observed refusing	28	fed the forbidden input, exited 2
returned 0 on what they reject	2	one dead, one dormant by design
not testable	1	reject path needs a fabricated transcript
declared but not armed	3	in the canonical settings, absent from the live one

Fifty-nine of the ninety cannot refuse anything, and that is correct. They inject context, write state, log telemetry. A scan for the four primitives that can actually block returned 31 hits and 59 misses. Calling those 59 "guards" would have been the first error, and it is the one an audit that counts files rather than capabilities makes.

Of the 31 that can block, 28 were observed refusing, at exit 2, with the forbidden input on stdin. One was not testable without fabricating a transcript and making a nondeterministic model call, so its reject path is unmeasured rather than absent.

Two returned 0 on the thing they exist to reject. Those two are the reason the question is worth asking.

The first is a rule-capture hook whose job is to make sure a rule stated in conversation is not lost when the session ends. It filters its capture file by session id. The capture file holds 230 entries and every one of them has a null session id, because the writer stores whatever the event delivered and that event does not deliver one. Feed it a session id that has captures: exit 0. Feed it the null it actually stored: it hits the no-session guard and exits 0. Both branches are dead, the hook is silent by design, and nothing about that is visible from outside. It has been wired, and running, and unable to fire, for as long as those 230 entries have existed.

The second is honest about itself in its own header: it ships dormant pending an allowlist that does not exist yet, and refuses correctly the moment its flag is set.

The finding I was not looking for

The census counted against the canonical settings file. Out of habit I diffed it against the one the runtime actually reads.

The canonical file references 82 hook files. The live one wires 79. Two of the three missing had just been measured blocking at exit 2. They work. They are not armed.

So the harness contains a guard that stops two agents dispatching into the same path, and a gate that validates handoff contracts between agents, both of which refuse correctly when invoked and neither of which is currently invoked at all. Question two, asked carelessly, passes them: the mechanism exists and the file says it is wired. It is the wrong file.

A third guard, which blocks duplicate decision-record numbers, is armed and structurally blind on the repository it lives in: its path matcher is anchored on a directory the harness moved out of, and the path it still matches does not exist on disk.

Three mechanisms, all of them real, none of them able to refuse anything today. I found them by asking one question I had been recommending to other people and had never run on myself.

The instrument, and what it gets wrong

The census above was done by hand the first time. Doing it by hand is how it stays a story, so I turned it into a script that anyone can point at their own hooks directory: it enumerates what is wired, resolves dispatch children, classifies each hook by whether it contains any primitive that can refuse, fires the probes, and diffs the settings file you maintain against the one the runtime reads.

Two of its design choices are the whole point. A hook with no probe is reported UNPROBED and is never counted as passing, because a census that could not test something must report a smaller denominator and a named gap rather than a clean bill. And the declared-versus-armed diff runs every time, unasked, because that is the check that found the two unarmed guards.

It also gets something wrong, and I am leaving the error in. Its automated scan reports 34 hooks capable of blocking where reading them by hand gives 31. The three extra carry an exit-2 inside a top-level catch, which is fail-closed on an internal crash and not a refusal of anything. A regex cannot tell a detection-exit from a crash-exit.

Which is this post's own subject, committed by the tool written for this post: a counting mechanism that cannot distinguish the thing it counts from something that resembles it. I could tune the pattern until the numbers agreed. Then the tool would be right on my repository and quietly wrong on yours, and I would have replaced a visible error with an invisible one. So the count is documented as an upper bound instead.

The script is public at github.com/willianpinho/hook-refusal-census. MIT, no dependencies, read only. Point it at your own hooks directory and it will tell you how much of your guard set it could not answer for.

One thing it will not do is hand you the 28 above. The census that produced that number probed all 31 by hand; the tool ships probes for 12 of them, and reports the other 19 as UNPROBED rather than quietly counting them. A probe is a claim about what a specific guard exists to reject, so it cannot be generic, and writing the missing 19 is work I have not done. The number you get on your own hooks will be smaller than mine for the same reason it should be: it will only count what you actually watched.

Making it generic cost two more findings, and both belong in this post rather than in a changelog.

The first is that the tool gives different answers from different directories. Same probe, same hook: a git-workflow guard returned 0 fired from a folder that was not a repository and 2 fired from inside one, moving the observed-refusing count by a full guard on nothing but where the process happened to be standing. It now prints the directory it fired from, because a measurement that does not report its conditions is not reproducible, and I had been reporting one.

The second is the same failure as the 34-versus-31 count, one layer out. The generic version briefly flagged every settings reference with no matching file in the hooks directory as a missing hook. On the first repository it ran against, that produced five accusations, and all five were ordinary sync scripts living one directory over, invoked from settings exactly as intended. I caught it only because I read the five names instead of the number, which is the same reason the gap above is documented rather than tuned away.

Where this bites outside agent systems

The pattern is not about AI, and outside agent systems it usually shows up as the false positive rather than the missing hook. A lint configured on a path the file is not in. A required status check whose job exits 0 when the test runner never started. A secret scanner whose patterns predate the current token format.

In each case the policy is written, the tooling exists, the check is green, and no one has watched the thing refuse anything. The rule holds while things are calm and stops exactly when they are not, which is when you needed it.

Write the text so people can reason about the rule. Build the mechanism so the rule survives the day nobody is reasoning. Then break the mechanism on purpose, because that is the only one of the three questions that answers with evidence instead of an assumption.