I almost deleted 29 rules from my agent's governance system.
Their hit rates sat near zero. Some hadn't fired across dozens of sessions. By every metric I had, they were dead weight — rules that consumed context budget without ever intercepting a mistake.
One script stopped me. It asked a different question than my dashboard did: not "did the rule fire?" but "can the rule fire?"
Twenty-nine detectors were broken. The rules weren't useless. They were running blind — looking for the wrong command, a deprecated tool name, a signal that had changed format months ago.
That near-miss is about signal quality, not about which rules I eventually demoted. Separately, I restructured my Core layer from 15 rules to 9 — seven demoted, one promoted, six slots freed. The hard part wasn't the restructuring. It was figuring out which retirement signals I could trust.
The Obvious Approach (That Doesn't Work)
If you want to retire rules automatically, the obvious signal is hit rate - how often each rule fires. Low hit rate = not useful = retire it. Sort by hit count, cut the bottom N. Done.
I almost built that. Then two readers pointed out why it breaks.
Mike (jugeni) left a comment that reframed the whole problem:
"A decayed count on a broken detector isn't evidence the rule is unused, it's evidence the detector never had a chance to fire."
And dipankar_sarkar added:
"The cost of the two errors isn't symmetric."
These two comments describe three traps that make hit rate misleading. I missed all three when I was planning a simple decay curve.
Three Traps That Make Hit Rate Misleading
Trap 1: The Broken Detector
Your rule says "before committing, check that tests pass." The detection logic looks for a test execution step. If the agent ran tests, the rule fires. If it didn't, the rule stays silent.
Now imagine the detection logic is broken - it looks for the wrong command, or the command format changed, or the tool name was updated. The detector runs, finds nothing, and reports: zero violations. Your dashboard shows a green checkmark. The rule looks like it's never firing.
But the detector is blind. The rule has never had a chance to do its job.
When I ran an automated check on my rules, 29 had broken detectors. Twenty-nine rules that looked inactive but were actually running with their eyes closed.
I learned this the hard way while writing this article. My source notes cited "5 broken detectors" from a changelog entry dated two weeks ago. When I ran the live script today, it returned 29. The detection coverage had expanded; the documentation hadn't. I'd been trusting a stale snapshot - exactly the trap I'm describing.
This is the first trap: zero hits can mean "the rule is unused" or "the detector is broken." Hit rate alone can't tell the difference.
Trap 2: The High-Consequence Operation
Some rules guard things you can undo. "Use consistent variable naming." If you retire that rule and naming gets inconsistent, you fix it with a rename. Costly, but recoverable.
Other rules guard things that are hard to reverse. "Don't delete production data." "Don't send a real email to customers." "Don't spend money on infrastructure." If you retire one of those because it "rarely fires" - and then it fires exactly once - you may not get a rollback.
The asymmetry is the trap. A rule that fires in 1 out of 100 sessions looks like a retirement candidate by hit rate. But if that rule guards a high-consequence operation, the cost of being wrong isn't "inconvenience." It's damage you can't easily fix.
dipankar_sarkar's formulation was sharper than mine:
"The cost of the two errors isn't symmetric."
His proposal: treat reversibility as a binary classification. Unknown or high-consequence operations get rules that never auto-retire. Frequency-based decisions only apply to advisory and style rules.
I'll be honest: this classification is partially manual right now. I use tool class as a conservative first-pass proxy - operations that write, spend, send, or mutate are treated as high-risk by default. This does not prove they are literally irreversible. A write may be transactional; an external request may be idempotent. Tool class is a risk proxy, not a proof. And the ideal - deriving reversibility by replaying tasks with the rule absent and watching for unrecoverable effects - is not yet implemented.
Trap 3: The One I Haven't Solved
The first two traps are observed in my data. The third is a design risk I have not yet validated in production.
A rule says "don't store card tokens in the database." It was written because you were using Stripe, and storing tokens would pull you into PCI compliance scope. The rule fires whenever the agent tries to write card data.
Two years later, you switch to a different payment processor. The rule still says "don't store card tokens." But the reason - PCI scope - may no longer apply in the same way. Or it may apply differently.
The rule's trigger key is an entity ("card_token column"), not a constraint ("card data in our DB pulls us into PCI scope"). When the constraint changes but the trigger doesn't, the rule either fires when it shouldn't or stays silent when it should fire.
This is the third trap: hit rate measures whether the trigger fired. It says nothing about whether the constraint is still valid. A rule can have a perfect hit rate and be protecting the wrong thing.
I won't claim I've solved this. Keying rules on constraints rather than entities is a design principle I'm working toward. For now, I can only flag rules whose triggers reference specific entities - and acknowledge I don't have a systematic solution yet.
The Pipeline (and What's Missing)
Once you see the three traps, the solution structure follows naturally. You can't just decay by hit rate. You need to filter out each trap before making retirement decisions.
Here's what I built - and what I didn't.
Filter 1: Check Detector Health
Before trusting any hit-rate signal, I check whether the detector is actually working. An automated script classifies every rule into one of three states:
- detector_healthy: the detector fires and produces verdicts
- detector_broken: the detector runs but can't produce a verdict - it's looking for the wrong thing, or the command format changed
- rule_genuinely_unused: the detector is healthy, but the rule never fires because the situation never arises
Only rules in the last category are safe candidates for retirement. Detector_broken rules are excluded - not because they're useless, but because their hit-rate signal is untrustworthy.
Out of 229 rules checked (the system has 289 total, but archived rules and rules without detector metadata are outside this check): 48 healthy, 29 broken, 152 genuinely unused.
Note: "healthy" means the detector can execute and produce internally consistent verdicts. It does not prove semantic coverage - a detector that only checks one command path can be "healthy" while missing other scenarios.
Filter 2: Exclude High-Consequence Rules
For rules that pass Filter 1 (detector is working, rule genuinely fires rarely), I check what kind of operation the rule guards.
Rules that guard high-consequence operations (write, spend, send, mutate - as a conservative proxy) are excluded from automatic retirement. They stay regardless of hit rate.
This is a binary gate, not a weighted score. dipankar_sarkar's argument was that weighting (e.g., "importance = 0.8") just moves the risk from the rule to the weight. A binary tag narrows the set of rules eligible for automation; it does not make the classification correct.
The classification is partially manual. Tool class covers most cases, but edge cases require human review - and human judgment rots the same way rules do.
What's Not Built: Frequency Decay
After Filter 1 and Filter 2, you're left with rules that have healthy detectors and guard reversible operations. For those, hit rate becomes a less obviously misleading signal.
What I have now is a binary decision: based on verdict distribution and linkage filtering, a rule is either a demotion candidate or it isn't. There's no continuous decay curve.
What dipankar_sarkar proposed - multiply hit count by ~0.9 per session, demote below threshold - is not implemented. What I have is a guarded check, not a decay algorithm. The missing piece is the frequency decay itself.
I want to be clear about this distinction: the filters are built and running. The decay mechanism is planned. The Core restructuring I'll describe next used the filters' output, not a decay algorithm.
What Actually Happened: Two Separate Events
I need to separate two things that happened in the same week, because the article initially conflated them.
Event 1: The pipeline archived 17 Task rules. The lifecycle check identified 28 degradation candidates. A linkage filter removed 11 (rules in conflict groups, safety-critical rules, known-traps tracking), leaving 17. Those 17 Task rules were archived - moved to a retired tier where they're still available but don't load by default. An audit log recorded each action. A rollback test passed.
Event 2: I restructured Core from 15 to 9. This was a separate, semi-manual decision. Seven Core rules were demoted to Task. One Task rule was promoted to Core. The pipeline provided candidate signals, but the final Core restructuring was a human judgment call informed by - not determined by - the pipeline output.
The demoted Core rules were low-frequency: OS flow-specific rules whose enforcement had been structured into protocol documents, a rule with no recorded misses in the observed sample, and platform-specific rules that native tool calling now handles. The promoted rule had 40 observed sessions with no recorded misses across multiple domains.
I'm not claiming the pipeline caused the Core restructuring. I'm saying the pipeline filtered out unreliable signals, and I then made restructuring decisions with better information than I would have had with raw hit counts.
Four Gaps I Haven't Closed
It's a flow constraint, not a permission-isolated gate. The scripts run when I invoke them. The agent could theoretically modify the scripts. This is not a system where the agent cannot bypass the check. It's a system where I run checks the agent doesn't run on itself.
The pipeline is heavy. Multiple scripts, audit logs, rollback mechanisms. I'm aware this might be over-engineered for most setups. If you have 20 rules, you don't need this. If you have 200, you might.
Frequency decay is not built. The missing piece is a sliding-window or exponential decay algorithm. What I have is a binary decision based on verdict distribution. It's better than manual squinting, in my experience - but I haven't measured that claim against a defined comparison.
Reversibility classification is partially manual. Tool class is a coarse proxy. Edge cases need human review. The ideal - replay-based derivation - is not implemented.
Start With One Check
You don't need a pipeline to benefit from this. The highest-value action is the one most people skip: check whether your detectors are actually working.
For each rule in your CLAUDE.md, AGENTS.md, or .cursorrules that has zero or near-zero observed hits, ask:
- What would trigger this rule? Name the specific signal - a command, a file pattern, a behavior.
- Is that signal still detectable? If the rule looks for a tool name that was renamed, a command that was deprecated, or a pattern that changed format, the detector is broken.
- If the detector is broken, what happens? The rule looks inactive. Your dashboard says "zero violations." But the rule was supposed to protect you - and it can't.
In my current check, 29 of 229 rules were classified as detector-broken. I cannot generalize that rate to other rule systems. But the exercise itself - asking "is my detector actually working?" - took me about 30 minutes for a 20-rule subset. Your mileage will vary.
If you want to go further, classify your rules by what they guard:
- Does this rule protect something easily reversible (naming, formatting, style)?
- Or something hard to reverse (data deletion, external calls, money, state mutation)?
High-consequence rules should never be auto-retired, regardless of hit rate. That single distinction prevents the most dangerous retirement mistake I can think of - though I haven't measured how often it actually happens.
The Bigger Picture
The ratchet effect — rules only get added, almost never retired — was the diagnosis. This article is what happened next.
Early reader feedback shaped this design more than my own analysis did. I built two filters (detector health and reversibility) that make hit rate less misleading, then a binary decision that still needs human judgment. A decay algorithm is still missing — when it's built, it will only apply to rules that pass both filters.
The next question - which I'll write about next - is: after you retire a rule, how do you know you were right? Retiring a rule is a claim that it's no longer needed. But "no observed violations since retirement" is not the same as "the rule was unnecessary." The absence of evidence can point to a blind detector — with safety still undetermined.
That's the next trap. I'll tell you what I find.
Production Log
This section is generated from the system's own production logs during the writing of this article.
Skills Triggered
| Skill | Trigger Count | Purpose |
|---|---|---|
| Session initialization | 1 | Context loading, memory injection, rule activation |
| Skill_UniversalSearch | 3 | Source discovery for claims |
Correction Chain
| # | What Happened | Source | Classification |
|---|---|---|---|
| 1 | Initial positioning cited "5 detector_broken rules" from historical changelog | Fact source data was stale (v5.1 integration, 2026-07-15) | Corrected after running detector_quality_check.py: current count is 29. Rule: always snapshot from live script output, not from historical changelog entries. |
| 2 | Draft v1 conflated Core restructuring (7 demoted) with pipeline execution (17 archived) as a single causal chain | Reviewer (Patrick) flagged a causal-attribution error | Separated into two independent events in draft v2. Pipeline provides signals; Core restructuring is a human judgment call. |
| 3 | Draft v1 presented frequency decay as an implemented Phase 3 | Reviewer (Patrick + Dr. Chen) identified mechanism status mismatch with fact source (source_pack) | Reframed as "What's Not Built" - binary decision exists, decay algorithm is planned. |
| 4 | Draft v2 lede still compressed the 29-detector near-miss and Core 15→9 into one timeline; Bigger Picture duplicated the decay sentence; section title said three gaps but listed four | Round-2 multi-persona review (P1-1..P1-4) | Decoupled lede; merged Bigger Picture; renamed to Four Gaps; Production Log English-only. |
What I Learned
The biggest correction during writing wasn't a factual error - it was a stale data point that became a structural one. The source notes cited "5 detector_broken rules" because that was the number when the detector quality check was first integrated. Running the script today showed 29. The detection coverage expanded, but the documentation hadn't caught up.
Then the reviewers found a bigger problem: I had strung together three things that happened in the same week - the pipeline filtering, the Task archiving, and the Core restructuring - as if they were a single causal chain. They weren't. The pipeline filtered signals. I made restructuring decisions. Conflating them made the pipeline sound more powerful than it is.
Round 2 caught the residual version of the same failure mode: even after the body separated the events, the opening still invited readers to read them as one story. Fixing the middle is not enough if the lede re-merges the timeline.
Author note: This article is co-written with my AI agent. I handle real experience, judgment, and final sign-off; the agent handles architecture, drafting, and fact sourcing. The system this article describes also produced this article - including the corrections logged above.
Part of the Agent Influence series. Previous: Why Adding More Rules Makes Your Agent Dumber
Top comments (0)