A reader asked me a question I couldn't fully answer: "Do you retire rules, or does interception count keep them alive?" The honest answer was: mostly, I don't. And that's a problem.
This article is co-authored with my AI agent. I handle real experience, judgment, and final sign-off; the agent handles architecture, drafting, and fact sourcing. The system this article describes also produced this article - including catching its own false claims during the writing process. The behind-the-scenes log at the end shows what happened.
After I published my last article about building an error notebook for my AI agent, a reader named dipankar_sarkar left a comment:
"Do you retire rules, or does interception count keep them alive?"
I stared at that question for a while. Because the answer was uncomfortable.
I had 268 rules. (When I published my last article, it was 266 - the system keeps learning.) I'd been adding them for two months - every time the agent made a mistake, I'd extract a rule and file it away. But retire them? Almost never. The file only grew. And I'd started noticing something: the agent was following fewer rules, not more.
This article is about why that happens, what it costs you, and what to do about it.
The Ratchet Problem
Your CLAUDE.md is a ratchet wrench. It only tightens, never loosens.
Every time your agent makes a mistake, you add a rule. "Don't do X." "Always check Y." "When Z happens, execute W." The file grows. You never remove anything - because how do you know which rule is safe to delete?
I've been there. My rule file - called AGENTS.md, the equivalent of CLAUDE.md in opencode - grew to 3,000+ characters. My own system history diagnosed the root cause in six words:
"Rules only added, never removed. No exit mechanism."
After slimming it from 170 lines to 106, compliance improved in my workflow. This is an N=1 observation, not a benchmark - but the direction was clear: less noise, more signal.
Why More Rules = Worse Behavior
This sounds counterintuitive. More rules should mean more guidance, right? In my experience, three things break:
1. Attention dilution. LLMs have finite context windows. Every rule in the context competes for the model's attention budget. When you have 10 rules, each gets a meaningful share. When you have 200, each gets noise-level weight. The signal-to-noise ratio drops. Anthropic's memory best practices recommend keeping CLAUDE.md specific and structured, and reviewing it regularly - not making it exhaustive.
2. Trigger conflicts. Rules can contradict each other. "Always be concise" vs "Always explain your reasoning." "Use the fastest approach" vs "Always write tests first." You don't notice when you write them one at a time. But the agent has to follow all of them simultaneously - and when they conflict, it picks one. Usually the wrong one.
3. Injection noise. Rules that never fire still consume context budget. A rule about database migration patterns sits in your file, doing nothing, while the agent is trying to write a CSS fix. It's dead weight - but the agent still reads it, processes it, and allocates attention to it.
The result: your rule file becomes a graveyard of good intentions. The agent skims it, picks up a few rules, ignores the rest. You think you have 200 rules working for you. You actually have maybe 15.
The Data: Most Rules Fire Rarely
I track every rule's effectiveness - how many sessions each rule actually fired in. After 175 sessions and 157 tracked rules, here's what the data looks like:
| Sessions where rule fired | Rules | Percentage |
|---|---|---|
| 11+ sessions | 8 | 5% |
| 4-10 sessions | 30 | 19% |
| 1-3 sessions | 119 | 76% |
76% of my tracked rules fired in 3 or fewer sessions out of 175. Only 8 rules (5%) fired in more than 10 sessions. My top rule - "don't skip the documentation before starting work" - fired in 23 sessions, about 13% of the time.
A note on what "fired" means: my system distinguishes between objectively verified interceptions (confirmed via tool-call logs or script output) and self-reported ones (the agent says "per rule X, I'm executing Y"). The session count above includes both. The self-reported numbers are a useful signal, but they're not independently verified - consistent with the theme of my previous article: declaration ≠ fact.
The long tail is where retirement candidates live. These aren't bad rules. They're rules that worked once or twice, and then the context shifted - the model got updated, the project changed, the mistake stopped recurring. But the rule is still there, taking up space.
Out of 268 total rules, I've archived 165. That's 62% - retired, not deleted, but out of the active set. Only 14 rules are always loaded (Core). Another 89 load on demand when the task type matches (Task). The rest must earn their way into the context when a task makes them relevant.
How I Govern Rules
Four mechanisms keep the system from drowning in its own rules:
1. Three-layer architecture.
Core: 14 rules (always loaded - the non-negotiables)
Task: 89 rules (loaded by task type - investment, coding, writing)
Archive: 165 rules (retired, searchable on demand)
Think of it like your phone. The home screen (Core) has your 14 most-used apps. The app library (Task) has apps organized by category. The cloud (Archive) has apps you rarely open but might need someday. You don't delete apps - you move them off the home screen.
2. Core ≤ 15 hard cap. When Core hits 16, the lowest-performing rule gets demoted. I use a rough heuristic - interception rate weighted by importance - to decide which goes. Safety-critical rules (like "backup before delete") are exempt. This isn't an optimized formula; it's a judgment call that works. Past 15, I see compliance drop.
3. Hit/miss tracking. Every rule's effectiveness is measured. Rules that fire rarely are retirement candidates. The tracking itself is a mix of objective signals (script logs, tool-call hooks) and self-reported signals (agent acknowledgments). Neither is perfect - but together, they surface the long tail.
4. Cleanup cadence. I borrow a principle from project governance: every 3 feature iterations, 1 cleanup sprint. The same rhythm applies to rules - every few rounds of adding rules, I do a retirement pass. This isn't a documented rule-level mechanism. It's a practice that works.
The Demo: Scan Your Rule File for Dead Weight
You don't need my full tracking system to start finding retirement candidates. Here are two ways to audit your own rule file right now.
Option A: The "Last Hit" Test (Manual)
Open your CLAUDE.md. For each rule, ask yourself:
"When was the last time this rule actually prevented a mistake?"
If you can't answer, or the answer is "months ago," it's a retirement candidate. But use judgment: safety-critical rules (like "backup before delete"), rare-but-high-impact rules, and seasonal rules should stay even with low hit rates. Archive low-risk rules with no recent hits. Archive ≠ delete - you can always bring a rule back.
Find 3 retirement candidates today. That's it. You'll be surprised how many rules are dead weight.
Option B: Rule-File Hygiene Scanner (Script)
If you want a quick automated check, here's a script that scans any markdown rules file and flags rules worth reviewing. It does not measure effectiveness - it can't, because it doesn't have hit data. What it does is flag rules that are structurally weak: no trigger condition, too vague to be actionable, or near-duplicates of another rule.
#!/usr/bin/env python3
"""Rule-file hygiene scanner - flags structurally weak rules."""
import re, sys
def scan(path):
with open(path) as f: lines = f.readlines()
rules = [(i+1, re.sub(r'^[-*]\s+|^\d+\.\s+', '', l.strip()))
for i, l in enumerate(lines)
if re.match(r'^[-*]\s+', l.strip()) or re.match(r'^\d+\.\s+', l.strip())]
candidates, seen = [], {}
for ln, text in rules:
r = []
if not re.search(r'\b(when|if|must|always|never|before)\b', text, re.I):
r.append('NO_TRIGGER')
if len(text) < 15: r.append('TOO_VAGUE')
k = text.lower()[:30]
if k in seen: r.append(f'DUP@L{seen[k]}')
else: seen[k] = ln
if r: candidates.append((ln, text[:60], r))
total = len(rules)
d = 'LOW' if total <= 10 else 'MEDIUM' if total <= 20 else 'HIGH'
print(f'{total} rules found. Density: {d}')
if candidates:
print(f'{len(candidates)} rules worth reviewing:')
for ln, t, r in candidates: print(f' L{ln}: {t}... -> {", ".join(r)}')
else: print('No structural issues found.')
if __name__ == '__main__':
scan(sys.argv[1] if len(sys.argv) > 1 else 'CLAUDE.md')
Save it as scan_rules.py and run it:
python scan_rules.py CLAUDE.md
Output:
26 rules found. Density: HIGH
8 rules worth reviewing:
L3: Use TypeScript... -> NO_TRIGGER
L7: Be concise... -> TOO_VAGUE
L14: Always write tests... -> DUP@L11
Three flags: NO_TRIGGER (no "when/if/must" condition), TOO_VAGUE (under 15 characters), DUP@L (first 30 characters overlap another rule). Each is a review candidate - not an automatic retirement.
Think of it as a lint check for your rule file. It catches structural problems before you even start tracking effectiveness.
What Doesn't Work (Yet)
I need to be honest about what this system doesn't do. A reader on my last article also asked: "What happens with a genuinely new failure mode that has never tripped a rule before? Do you track novel slip-throughs vs known recurrences?"
The answer: no, I don't. Here's what's missing:
Retirement is manual. I review low-hit rules and decide. There's no automated degradation - no "rule auto-retires after 30 consecutive misses." I'm evaluating this, but it's not implemented.
No conflict detection. I can't detect when two rules contradict each other. If "be concise" and "explain thoroughly" are both in Core, the agent just picks one. I don't even know which one wins. This is a known gap in evaluation.
No novel vs known tracking. When a mistake happens, I can't tell if it's a known error recurring (rule failed to intercept) or a genuinely new failure mode (no rule exists). A reader pointed this out, and it's the most fundamental limitation - the system is reactive, not predictive.
No risk-aware retirement. The "last hit" test is crude. Safety-critical rules, rare-but-high-impact rules, and seasonal rules shouldn't be retired just because they haven't fired recently. The system doesn't currently distinguish between "low hit rate because the rule is useless" and "low hit rate because the failure is rare but catastrophic." I make this judgment manually; it should be systematized.
This is still a 1-user system. 175 sessions, one workflow, one agent platform. The patterns I see might not generalize. The hit distribution might look different for a team or a different model.
What You Can Do
1. Run the "last hit" test. Open your rule file. For each rule, ask when it last prevented a mistake. If you can't answer and the rule isn't safety-critical, archive it. Find 3 today.
2. Run the hygiene scanner. python scan_rules.py CLAUDE.md - 30 seconds, see your structural dead weight.
3. Track hits for 30 days. You don't need my full system. Just add a note next to each rule: "[hit 2026-07-10]" when it fires. After 30 days, the long tail will reveal itself. Then retire the bottom 20%.
The harder problem in agent governance isn't memory. It's forgetting. Your agent doesn't need more rules. It needs a metabolism - a way to break down and eliminate what no longer works.
Previous article: I Built an Error Notebook for My AI Agent - how to build the error notebook and intercept mistakes before they happen.
Next article: Giving LLM a body - why rules, memory, and verification are just organs of a larger control system.
Behind the Scenes: How This Article Was Built (click to expand)
> The system this article describes also produced this article. Here's what actually happened - including false claims caught during review.
What the system did during production
| Stage | What happened | Why it matters |
|---|---|---|
| Source pack | Every claim was verified against raw data before writing began | Prevented false claims from reaching the draft |
| Data analysis | L2_Hits.csv was analyzed programmatically | Initially used the wrong column (self_reported instead of total) |
| Fact-checking | The "3:1 cleanup ratio" was searched in docs - found it's project-level, not rule-level | Prevented overclaiming a practice as a documented mechanism |
| External review | A reviewer caught 4 P0 issues that internal review missed | Including the wrong data column, demo overclaim, and metric naming |
Mistakes caught during production
| What the agent assumed | How it was caught | What happened |
|---|---|---|
Used self_reported_hit_count (67) but called it "hits" |
External review: CSV has hit_count=1516, objective_hit_count=1449, self_reported_hit_count=67
|
Switched to sessions_hit as primary metric; added caveat about self-reported vs objective |
| "Only 14 rules are active" (implying Task rules are inactive) | External review: Task rules load on demand, they're not inactive | Changed to "14 rules are always loaded"; Task described as "load on demand" |
| Demo "automates hit/miss tracking" | External review: script only does text heuristic scanning, doesn't read hit data | Reframed as "hygiene scanner"; explicitly states "it does not measure effectiveness" |
| Used "266" for series consistency while citing current session data | External review: mixing old total with current data is inconsistent | Changed to 268 (current actual); noted evolution from 266 in opening |
| "3:1 cleanup ratio for rules" (documented mechanism) | Source pack verification: 3:1 is project-level cadence, not rule-level | Reframed as practice, not documented mechanism |
interception_rate × importance presented as a real mechanism |
External review: no source in docs for this formula | Labeled as "rough heuristic" and "judgment call" |
| Anthropic guidance paraphrased as "short and specific" | External review: not the actual language used | Replaced with accurate description + link to source |
What the system learned
Three things emerged from writing this article:
- Verify data columns before naming metrics. "Hit count" means different things in different schemas. Name the exact column you're using.
- Don't conflate "always loaded" with "active." On-demand loading is still active - just selective.
- Demo scope must match demo claims. A text scanner is not a hit-rate analyzer. Say what it does, not what you wish it did.
These lessons are now in the error notebook. The system's own article production process is its own dogfood.
Top comments (21)
Rule retirement is the part most agent setups ignore. Adding rules feels responsible, but stale rules become background noise and eventually conflict with the work. A rule audit should probably track last used, failure prevented, conflict risk, and whether the rule belongs in prose or in a deterministic tool check.
Three of those axes the system already tracks. Last-used: per-rule hit count split into script-verified and self-reported. Conflict risk: three-layer check (conflict groups, domain+action overlap, text similarity). Failure-prevented: subtler - 30 of 31 high-violation rules turned out to be detector problems, not rule problems, so "the rule fired" and "the rule caught a real failure" aren't the same thing.
The fourth axis - prose vs deterministic tool check - is the one that reframes the audit. Most rules live in prose (soft constraint, agent can comply or not). A subset are enforced by scripts that fire regardless. The audit question I hadn't asked: which prose rules should have been promoted to checks? A rule guarding something irreversible shouldn't stay in prose if a script can enforce it. The audit should flag rules whose enforcement path is weaker than their failure cost warrants.
That distinction between detector problems and rule problems is huge. A rule firing is only a signal; the maintenance question is whether it prevented a real failure or just exposed noisy instrumentation.
Glad the question landed. The framing that unlocked this for me: your always-loaded rules are a cache, and you are never evicting. Interception count is a hit counter, so treat it like one. Two additions to the retire question. First, decay, not raw count. Weight interceptions by recency. Lifetime count keeps a rule alive forever off one old spike, but a sliding window tells you what is load-bearing now. Second, firing is not enough to earn the always-loaded tier. If the model would derive the behavior from a one-line reminder, demote it to a note injected lazily when the triggering context shows up. Reserve the permanent slots for genuinely counterintuitive constraints the model will not reconstruct on its own. The 3000-char AGENTS.md is really a cache-eviction problem wearing a documentation costume.
The cache framing is sharper than my "ratchet" metaphor. A ratchet describes the symptom (only tightens). A cache with no eviction policy describes the mechanism - and points directly at the fix.
On decay: this is exactly the direction I'm evaluating. Lifetime count is a broken metric for the reason you describe - one spike keeps a rule alive forever. A sliding window (or exponential decay) would surface what's actually load-bearing. It's not implemented yet. Currently I do this manually: review low-hit rules every few weeks and decide. It works, but it doesn't scale, and it's biased by my memory of recent sessions.
On "firing is not enough to earn the always-loaded tier" - this is the gap I didn't articulate in the article. You're drawing a distinction between:
Since the cache framing landed: the eviction policy that maps cleanest onto your interception counts is LFU-with-aging, not plain LRU. A rule that fired 40 times in month one but zero in the last three weeks is exactly what you want to retire, and pure LRU would keep it alive on that stale history. So decay the count: each session multiply every rule's interception count by ~0.9, and when one drops below a threshold, demote it from always-loaded to on-demand (description-gated) rather than deleting it. That gives you a reversible eviction. The rule leaves the resident set but is still fetchable if its pattern recurs, and you stop having to guess which ones are safe to remove.
LFU-with-aging is the right name for what I was circling around without the vocabulary.
Current state: I have a lifecycle check that's a coarser version of this. It looks at whether a rule's "obligations" (applicable sessions) produced any verdict recently. If zero applicable sessions → demotion candidate. But it's binary (applicable/not), not decayed. Your 0.9-per-session approach would catch the exact case I miss: a rule that was applicable 40 times in month one and zero since still shows "historically active" in my system.
The "demote, don't delete" principle is already structural in my Archive tier — archived rules stay fetchable. What's missing is the automatic trigger. Right now demotion is semi-manual: the lifecycle check flags candidates, I review, I confirm. Your decay approach makes it automatic and reversible, which is strictly better.
One design question I'm wrestling with: the threshold. Some rules fire infrequently but guard irreversible actions (e.g., "verify before destructive file operation"). Pure frequency decay would retire them. I'm thinking about weighting the decay rate by a rule's importance tag — slower decay for rules that prevent irreversible damage. But I haven't worked out whether that introduces its own failure mode.
This thread has sharpened my thinking more than the last few solo iterations. The 0.9 decay is going into the next implementation.
The importance-weighted decay is the right instinct, and the failure mode you're sensing is real: you've moved the risk onto the importance tag, and now a mis-set tag on a 'verify before destructive op' rule gets it decayed out, with asymmetric cost, retiring a safety guard is irreversible, retiring a style rule is nothing. So I'd stop treating it as one decay curve with a weight. Split by what the rule guards, not how often it fires. Frequency-decay is the right model for advisory and style rules, where a stale rule just adds noise. For rules whose guarded action is irreversible, frequency is the wrong axis entirely, blast radius is a property of the action, not of usage, so those shouldn't decay on frequency at all, only on explicit supersession. Then the tag you actually need is binary and fail-safe: unknown or irreversible then never auto-retire. Same reason you keep archived rules fetchable, the default on the dangerous class has to be the safe one, because the cost of the two errors isn't symmetric.
"Split by what the rule guards, not how often it fires" - cleaner than the weighted decay I proposed. The weight moves the risk; a binary tag eliminates it.
"Unknown or irreversible -> never auto-retire" - the default should be the safe one. This makes it structural, not a judgment call.
We have half of this: demoted rules go to Archive, not deleted. What's missing is the irreversibility tag itself - our metadata tracks triggers and check methods, but not "what happens if this rule is silently absent when it should fire."
The tag is the hard part, because if you hand-label it, it rots the same way the rules did.
What worked: derive it instead of declaring it. Take the archived rule, replay the task set with it absent, and watch only for unrecoverable effects, a write that lands, a delete, an external call, a state change with no inverse. If absence ever crosses into "cannot be undone," it is irreversible, full stop, regardless of how rarely it fires.
Frequency and reversibility are orthogonal, and your Archive-not-delete default is exactly what makes the replay safe to run. The label stays honest because the harness re-derives it as the code moves under it.
"Derive, don't declare" — that's the upgrade I was missing. A hand-labeled irreversibility tag has the same ratchet problem as the rules themselves.
The replay approach is elegant because Archive-not-delete already makes absence safe to test. Our gap: we don't capture task sets in replayable format yet. Each session is unique, so "replay with rule absent" needs either recorded traces or a canonical test suite.
"Frequency and reversibility are orthogonal" is the cleanest map of the design space I've seen. We were treating them as related. They're not. The binary tag doesn't need a human — it needs a replay harness that detects irreversible effects. That's a build, not a classification exercise.
The gap you named (no replayable task sets) bites less than it looks, because the two axes cost different amounts to measure.
Reversibility is derivable statically. It is a property of the effect the rule guards, not the task: does the gated action write to disk, spend money, send a request, mutate prod. You read that off the tool or syscall class the rule names. No replay needed.
Frequency is the axis that needs evidence. And here is the payoff of splitting them. The reversible-guarding rules are exactly the ones whose absence is safe to test in live traffic. Shadow-remove them in production, count outcomes-changed, move on. You never build a replay harness for those.
Only the irreversible-guarding minority needs the recorded-trace suite, because you cannot safely A/B their absence live. So the taxonomy does not just map the design space. It tells you which small subset of rules is worth the expensive harness. The rest prune with a counter and a shadow flag.
This collapses the problem. I was treating the missing replay harness as a blocker for the whole approach - you've split it so the expensive path only applies to the minority.
Reversibility readable off the tool class (write, spend, send, mutate) without running anything. The system already demotes rather than deletes - a rule leaves the resident set but stays fetchable - so shadow-removing a reversible rule and watching whether the violations it would have caught start appearing is the natural extension. The replay harness is only needed for the irreversible minority.
"The taxonomy tells you which small subset is worth the expensive harness" - that's the scope reduction I was missing.
"Rules only added, never removed. No exit mechanism" is the whole post in six words and it generalizes past CLAUDE.md. I've hit the same shape building a memory-authority gate: notes accumulate authority just by existing, and nothing in the system asks whether a note still deserves to override what's already known. Your archive tier (165 of 268, 62%) is basically a TTL you're enforcing by hand. The part I'd push on: your hit/miss tracking mixes objectively verified interceptions with self-reported ones, and you flag that honestly, but self-reported "per rule X, I'm executing Y" is exactly the kind of claim that looks like evidence and isn't, declaration passing as fact. Have you run the hygiene scanner's flags against which rules turned out to be false positives in your hit data? That'd tell you whether structural weakness (no trigger, vague, duplicate) actually predicts low firing rate, or if they're independent problems.
"Rules only added, never removed. No exit mechanism" - that's the six-word version I should have led with.
The memory-authority gate parallel is real. "Notes accumulate authority just by existing" - that's exactly the dynamic. A rule enters my always-loaded tier and nothing asks whether it still deserves the slot. My archive tier is a manual TTL, as you say. Currently building automatic demotion via decayed interception counts (LFU-with-aging, see thread with Dipankar above), not deployed yet.
On the self-reported data push: you're pointing at the structural weakness, and I now have data that proves your point.
Since writing this article, I rebuilt the evidence system with independent verification sources (git diff audits, session transcripts). Three evidence tiers: deterministic (git audit) > independent (exit codes, transcripts) > agent claim (self-reported markers).
The data is stark. Self-reported markers account for 68% of all evidence collected. But they caught zero real violations. All 182 actual violations (95 "failed execution" + 87 "contradicted evidence") were detected by independent or deterministic sources - exit codes and git diffs, not the agent's own claims. The self-reported tier generates pass verdicts; the independent tier catches failures. "Declaration passing as fact" is exactly what was happening, and the numbers confirm it.
On the hygiene scanner correlation: I haven't run that exact analysis, but I did something adjacent. I reviewed 31 rules that showed high violation rates. 30 of 31 turned out to have structural detection problems - partial compliance signals, mixed patterns - not rule quality issues. The rules weren't broken; the detection was. After fixing detection, only 5 rules showed real violations (all caught by exit_code verification). This suggests structural weakness in detection predicts false violation signals more than low firing rate - but that's from adjacent data, not the direct correlation you're asking for. Still on the to-do list.
30 of 31 being detection problems, not rule quality, is the finding I'd lead the whole post with if I had it. It says something sharper than "rules need pruning": most bad rules aren't bad ideas, they're good ideas wearing broken detectors, and that changes what LFU-with-aging is actually measuring.
Decayed interception count can't tell those apart. A rule with a genuinely broken detector looks identical to a rule nobody needs, both show near-zero firings, both drift toward demotion under the same signal. If detection quality dominates the way your 31-rule sample suggests, demoting on decay before fixing detection risks quietly archiving rules that are still load-bearing but invisible to their own trigger.
Worth running detector-repair as a pass that happens before the aging gate gets trusted, not alongside it: fix the 30, then let decay counts run on rules whose detector is actually working, and see whether the demotion signal looks different. Right now a decayed count on a broken detector isn't evidence the rule is unused, it's evidence the detector never had a chance to fire, and those two conclusions point at opposite fixes.
A decayed count on a broken detector isn't evidence the rule is unused, it's evidence the detector never had a chance to fire" - exactly right. I was treating detector repair and decay as parallel tracks. They're sequential.
The 30/31 finding came from a manual review: after fixing detection logic, only 5 of 31 showed genuine failures (all exit-code verified). The other 26 were false positives - load-bearing rules invisible to their own triggers.
The gap: we don't have an automated detector-quality check. Today the only way to find a broken detector is tracing false violation signals back to source. That needs automating before any decay signal is trustworthy.
26 of 31 being false positives on rules that were actually load-bearing is the number that should worry you more than 5 real failures would, a real failure just means the rule fired and got ignored, a false positive means the rule was silently protecting you and every dashboard you had said otherwise.
The automation gap you named, tracing false-violation signals back to source is currently the only way to find a broken detector, is the same shape as the mechanizability problem showing up elsewhere in this cluster this week: a manual review that works is still a manual review, and it doesn't scale past the one pass you already ran. What would a detector-quality check actually look at mechanically, agreement between the detector's fire rate and an independent signal (execution outcome, exit code, git diff), or something structural in the detector's own pattern (regex fragility, missing negative-case coverage)? The second is cheaper to build and catches a narrower class of failure, the first is closer to ground truth but needs the same independent-outcome data you already had lying around for the 5 real ones.
The framing inversion lands: 26 false positives on load-bearing rules is worse than 5 real failures. A false positive means the guardrail was working and every signal told us to remove it. That's the failure mode that would have made the system less safe by trying to improve it.
On detector-quality: we're starting with option (b) — structural analysis (regex fragility, missing negative cases) — because it's cheaper and runs without execution. But you're right that it catches a narrower class. The full version is option (a): fire rate vs execution outcome. We already have that independent-outcome pipeline for the 5 real violations (exit codes, git diffs). Generalizing it to every rule is the build that makes detector-quality automatic. Your two-option framing is now the design spec.
Structural analysis first makes sense as a cheap prefilter, and it'll catch a real chunk given the 30/31 finding already skewed toward pattern-level breaks. Worth naming what it can't catch though: a detector can pass structural review clean, no fragile regex, real negative cases, well-formed, and still misfire against ground truth, because structural soundness checks that the pattern is well-built, not that it's checking the right thing. A rule can be a syntactically perfect detector for the wrong signal.
So (b) narrows what needs (a), it doesn't replace it. The rules that pass structural review aren't cleared, they're just no longer the cheap case. If the eventual build treats structural-pass as "done" rather than "promoted to the harder queue," the fire-rate-vs-outcome pipeline never gets applied to the rules that most need it, the ones that look fine on inspection and are still quietly wrong. Given you already have the independent-outcome pipeline running for the 5 real violations, the cheapest next step might be running it against a handful of structurally-clean rules just to see if the false-positive rate there is actually zero or just unmeasured.
"(b) narrows, doesn't replace" - that's the distinction I was missing. Structural-pass means "promoted to the harder queue," not done. On your suggestion: that's the cheapest test we can run. The independent-outcome pipeline already works for the 5 real violations. Running it against a handful of structurally-clean rules tells us whether that false-positive rate is zero or just unmeasured. That's the next step.