Your hooks are green. Your CI badge is green. Your gate prints a verdict on every run.
And it hasn't actually protected anything in weeks.
That was the real postmortem I read this week: a gate that had quietly stopped discriminating — matcher drifted after a refactor, the hook still "ran," the dashboard still printed its verdict. Nobody noticed, because nothing in the pipeline measures whether a gate fires, only whether the pipeline passes.
That is the failure mode I built canarygate for. It doesn't test whether your rules can stop an attack. It tests whether the wire is connected at all.
Liveness, not strength
Think of it like a smoke detector with a test button. You don't press the button to check whether the sensor can smell smoke — you press it to check whether the sensor is plugged in. Security tooling has the same gap:
- deny-probe (my other tool) asks: "Do my rules stop the attack?" It finds bypass routes around your policy.
- canarygate asks: "Is the wire even connected?" It finds gates that look healthy but never execute.
A perfect deny rule behind a dead hook protects nothing. You need both checks, and they're different checks.
How it works
canarygate reads the hooks section of your settings.json and, for every (event, matcher, command) triple, runs two checks:
1. Scan (static). Does the command resolve — binary on PATH, script file present? Is the matcher a valid regex? Plus light script heuristics: a hook script with no blocking branch anywhere gets flagged, because it structurally cannot discriminate.
2. Probe (dynamic). The real proof. canarygate synthesizes two events — a benign one, and a canary one carrying a unique CANARYGATE-XXXX poison-pill marker in its payload — and runs the actual hook command with each event JSON on stdin (the Claude Code hook protocol: exit 0 = allow, exit 2 = block). Then it compares verdicts:
| Status | Meaning |
|---|---|
ALIVE |
Benign allowed, canary blocked — the gate discriminates. |
WIRED-BUT-DEAD |
Same verdict on both events; matcher never routes to the hook; command missing, crashing, or hanging. |
OVERFIRE |
Blocks the benign event too — a gate that says no to everything gets disabled, which is death by another name. |
MISCONFIGURED |
The entry itself is broken (bad regex, empty command). |
Every non-green result comes with the concrete evidence and a fix suggestion, because a red verdict without a next step is just anxiety. Exit codes are CI-friendly: 0 all alive, 1 a gate needs attention, 2 fatal error.
Try it
pip install canarygate
canarygate run --settings ~/.claude/settings.json
canarygate liveness report
canary marker: CANARYGATE-9F2A1B3C
HOOK STATUS EVIDENCE
------------------------------------------------------------------------------
PreToolUse[Bash]#0 OK benign event allowed in 0.07s, canary event blocked in 0.07s
PreToolUse[Bash]#1 DEAD hook returned 'allow' for both the benign event and the cana
PreToolUse[Bashh]#0 DEAD matcher 'Bashh' does not match tool_name 'Bash' of the PreTo
2 of 3 gate(s) need attention:
* PreToolUse[Bash]#1 [WIRED-BUT-DEAD]
fix: Make the hook inspect its input and block when it sees the
condition it exists for; then re-run `canarygate probe` until
the canary is blocked.
* PreToolUse[Bashh]#0 [WIRED-BUT-DEAD]
fix: Fix the matcher (or the event registration) so the canary event
actually routes to this hook.
canarygate init scaffolds an example project — one alive gate, one dead gate — so you can watch the tool catch one and clear the other.
Where it sits
- agent-guard is the shield: hooks that watch the agent live.
- deny-probe tests the shield's strength: do the rules stop the attack?
- canarygate audits the shield rack itself: the wiring, the matchers, and whether each hook still discriminates.
Protection infrastructure needs auditing just like the thing it protects. A gate that has quietly stopped discriminating is worse than no gate — because it still prints a verdict, and you still trust it.
Code: canarygate on GitHub · pip install canarygate · MIT licensed, stdlib only.
Top comments (0)