Here is a failure I had not thought about until someone asked me the right question.
You write a PreToolUse hook. It reads the tool call, decides ...
For further actions, you may consider blocking this person and/or reporting abuse
Good to see the thread grow into a post. The canary is the piece we never got to in the comments.
On detecting a guard that's been dead for a month, I'd count rather than simulate. A canary proves the guard still refuses the one call you scripted. It can't see the guard quietly losing its grip on real traffic. And logging inside the hook isn't enough either: if the matcher stops matching, the hook is never called and logs nothing, which looks exactly like a quiet week. So compare what the hook says it saw with a record the hook doesn't produce. If the remote shows twelve pushes this week and the push guard saw zero, it's dead, and you found out without guessing what the failure would look like.
It's your redaction bug turned around: the check has to come from a source that doesn't share the guard's belief. The VCS, CI logs and the harness transcript qualify. The hook's own fixtures don't.
Semitexa's
ai:verifyfollows the same rule for its own empty runs: when no check matches the change, the verdict isnothing_to_verify, not a pass, because "nothing ran" and "everything passed" must never look the same.Is there anything in your harness today that would give you that independent count, or would you have to build it?
Counting rather than simulating is better than what I had, and the reason is the part I'd missed: a canary only proves the guard still refuses the call I thought to script.
To your question, it turns out I already have the independent record and hadn't noticed it. The harness writes a JSONL transcript per session, and every tool call lands in it with its name and input, written by the harness rather than by any hook. I counted one of mine just now: 507 Bash calls, 328 edits, 186 writes. Hook refusals show up in there too, so both halves of your comparison come from the one source that doesn't share the guard's belief.
What makes that better than instrumenting forward is that it's retrospective. There are 594 of those transcripts sitting on my disk. So I can ask today whether the pooler guard has seen a matching command in the last month and find out it died in September, instead of adding a counter and waiting a month to learn the same thing. The audit I thought I'd have to wait for is already in the logs.
And agreed on the empty-run verdict. "Nothing ran" and "everything passed" being indistinguishable is the whole post.
And thank you for the comment, this is the most insightful comment I've had on this site.
Thank you, that means a lot from the person who wrote the post. The transcripts are a better answer than mine: written by the harness, retrospective, and already on disk, so the audit costs one script instead of a month of waiting. I'm curious what the count says for your pooler guard. And your thread already changed something on our side: today I filed a task to move our ai:verify gate out of the instructions file and into a Stop hook, with your "fail closed only where declining costs a prompt" rule and version-stamped recorded fixtures written into it.
I ran it. 599 transcripts, 81,200 Bash calls since July, 1,380 of them touching a pooler or RDS host, which is the traffic that guard claims to cover. 170 in the last full week, 82 this week, so the denominator is alive and current.
Then the useful part: I have a denominator and no numerator. The hook refuses things without recording that it refused them, so I can't complete your comparison today. The audit I was pleased with measures the half I already trusted.
One thing the series taught me that I'd have got wrong on my own. Two weeks in October show zero matching commands, which is exactly the shape I'd have read as a dead guard. They're zero because total Bash calls those weeks were also zero, since I wasn't working. So a per-hook count has to be read against total tool traffic, or a holiday and a dead tripwire are the same picture. Same mistake as the post, one level up, in the detector I built to catch it.
Good luck with the Stop hook. I'd be interested to hear whether fail-closed holds up once it's someone else's CI eating the false block. Hope you write a post about it.
The 'two states that look identical from the outside' line is the whole problem — a quiet guard reads as a working guard, and the broken case is worse than no guard because you'd have been more careful without one. Your warn-mode detail is the one I'd steal: a warning that can't name the file or command but talks as if it can teaches the model to discount every future warning. On the canary — have you tried making the scheduled bad call part of the hook's contract test, or does that just move the staleness problem one layer up?
It moves it, and I think I can say where to.
A contract test runs the hook against a payload the test runner hands it. If the harness renames a field, your known-bad call is still in the old shape, so it passes and proves the matcher works against last year's reality. You've tested your code rather than the integration, which is the original failure with a more respectable-looking test wrapped around it.
The canary only escapes that if the bad call goes through the real path: the agent issues it in a real session, the harness builds the payload, the hook refuses it where it normally would. Then the thing under test is the join, which is the part that actually broke.
So contract test for the matching logic, canary for the integration, and they go quiet in different ways. The warn-mode detail you mention I only got right because someone in the earlier thread pushed on it.
That split is the clean answer — contract test proves the matcher, canary proves the join, and neither can cover for the other going quiet. The 'different ways they go quiet' point is the one to watch: when the canary goes quiet, is that itself alarmed, or does it fail the same silent way the original guard did?
"A check built from the same assumption as the thing it checks is a second copy of your belief" is the line I will be repeating. The test and the hook agreed with each other perfectly and neither had spoken to the harness in months.
What I liked most is how the thread moved from simulating to counting. A canary proves the guard still refuses the one call you scripted. Counting real matches against the harness transcripts tells you whether it still sees live traffic, and running that per week, as Kartik suggested, dates the break instead of just reporting it.
Jackson's gate that passed when its log file was missing is the same bug in different clothes. A zero only means something next to a non-zero that proves the instrument was alive.
At Ascendra Ventures we have started writing "how do we know this guard is still working" into the handover for client automation, because the people who inherit it rarely know to ask. Now that you have a denominator, what would you accept as a healthy numerator for the pooler guard?
One of my gates passed whenever its log file was missing. It enforced the hardest rule in my agent test harness, no live API calls outside the planned captures, by reading the call log. If the file didn't exist it returned True with "absent, so zero calls logged", so a recorder that wrote nothing would have passed. An agent found it while reading the gate code. The fix makes a zero count only when a non-zero next to it shows the recorder ran.
That's the same bug in different clothes, and your fix names the general form better than my post did.
A zero is ambiguous. It means "nothing happened" or "nothing was recorded", and from the inside those are identical. So a zero is only trustworthy next to a non-zero that proves the instrument was alive, which is exactly your check.
I hit it an hour ago counting how often one of my guards should have fired. Two weeks in October showed zero matching commands. Dead guard, obviously. They were zero because I hadn't run the agent at all those weeks, and the only reason I know is a second count sitting beside the first.
Also worth sitting with: your gate was found by an agent reading the code, and mine was found by a GIF I made for a landing page. Neither of those is a process.
The fix that keeps working after the harness changes again: stop building the payload in the test by hand. Record it. Capture the last dozen real PreToolUse calls your harness actually makes — they're small, and nothing sensitive lives in the shape — commit them as golden files, and have the unit test assert the hook matches at least one recent real payload.
Then a harness field rename turns your CI red within a day instead of leaving the tripwire dead for eighteen weeks. The test stops being a mirror of your belief and becomes a diff against the outside world, which is the thing that actually changed.
This composes with the canary idea rather than replacing it: the canary proves the hook still fires at runtime, golden payloads prove the matching logic still describes reality. You want both, because they go quiet in different ways.
Agreed, and "at least one recent real payload" is carrying more weight in that sentence than it looks.
A corpus recorded once and committed becomes the same stale belief with better provenance. Two years of green tests against a dozen payloads from setup day proves nothing about today. So the assertion has to be against payloads newer than some window, which means the recording can't be a one-off, it has to keep running.
Someone in the other thread added the piece that makes it enforceable: stamp each recording with the harness version, and have CI refuse a changed fixture whose stamp didn't move, because re-recording is the reflex that turns a red test green without anyone deciding anything.
And yes on composing rather than replacing. Golden payloads check that the matcher still describes reality, the canary checks the integration still fires. I have neither, so at the moment I'm two good ideas and no implementation.
Same as a smoke detector nobody tests. It stays quiet for months and everyone reads quiet as safe. Pushing the button is the only way to know. We're building that kind of real-world check into ML Systems.
The analogy has a sharp edge that favours your point. The test button checks the siren, not the sensor. You press it, it shrieks, and you've learned the speaker works and nothing whatsoever about whether it can still smell smoke.
Which is the limit of a canary too: it proves the guard still refuses the one call you scripted. The sensor equivalent is counting real traffic against a record the guard doesn't produce.
A harness renamed one field and the tests stayed green because the test built its own payload. A check I'd add is a canary run: on a schedule, feed the hook a payload captured from the real harness rather than one written by hand, and require that a known-dangerous command is actually blocked. I'd also keep a counter of how often the hook matched anything during normal use and alert when it flatlines. That way a quiet hook and a broken hook stop looking identical, and the unit tests no longer have to carry that job alone.
Both halves of that are right, and the thread has turned up a limit on each that's worth knowing before you build them.
On the canary: if the scheduled bad call is handed to the hook by a test runner, it only proves the matcher works on the payload shape you recorded. To exercise the thing that actually breaks, the call has to go through the real harness so the payload gets built the way a real one is.
On the flatline alert: mine would have paged twice in October for a completely healthy hook, because I wasn't working those weeks and every count was zero. It needs a per-hook baseline and a denominator of total tool traffic rather than a global threshold.
@kartik-nvjk and @kami_butt_69b922ade8de012 in this comment section reached those 2 points independently, which probably means they're the real edges rather than my particular mess.
Since the transcripts are already on disk, I'd run the count per hook per week instead of one total. A hook whose matches drop to zero in a single week points straight at the harness update that renamed the field, which matters when the break is eighteen weeks old.
You're right, and I ran it per week within the hour. It dates the break, which a total can't: "it's dead" becomes "it died the week of the 12th", and now there's something specific to go and look at.
One wrinkle I only saw because the data showed it. Two weeks in my series are zero, which is the exact shape of a hook that stopped matching. They're zero because I wasn't working those weeks and total Bash calls were also zero. So the weekly series has to be read as a share of overall tool traffic rather than as a raw count, or a holiday and a dead tripwire look the same. Which is the post's own failure mode turning up one level up, in the detector I built to catch it.