Update — v0.2.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, TUI review, observability, 7 export formats, adversarial corpora, and a bundled git rule pack. The field test report evaluated 4 models across 745 trajectories and is the source for every number in this article. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
My model produces triggers like "when git push fails with non-fast-forward" — specific, concrete, names the tool and the error. 94.4% of triggers are specific or moderate. Only 5.6% are generic. The specificity scorer works. The extraction prompt works. The model is doing its job.
And 6 of 10 golden scenarios still fail — because the trigger matches the right failure AND matches clean successes it would break. The trigger is specific about the failure. It is broad about when it fires.
Specific does not mean safe.
The trigger specificity numbers
The v0.2.0 field test measured trigger specificity across all 4 models. The learnings & fixes log §1.3 has the breakdown:
| Model | Specific | Moderate | Generic | Generic % |
|---|---|---|---|---|
| Local Llama 3.2B | 253/337 (75.1%) | 65/337 (19.3%) | 19/337 | 5.6% |
| Local Qwen 4B | 30/30 (100%) | 0/30 (0%) | 0/30 | 0% |
Both models are well under the 10% generic target. The triggers name concrete tools and error conditions — "when git push fails with non-fast-forward", "pip install fails with version conflict", "docker build fails with package not found", "terraform plan fails with state lock error", "pytest fails with AssertionError", "kubectl apply fails with NotFound for CRD". These are not vague platitudes. They are specific failure descriptions.
The extraction layer is working. The model produces high-quality triggers. And that is exactly what makes the remaining failures so hard to fix.
The golden corpus: 2 pass, 2 fail, 6 inconclusive
After the matcher fixes (Fix 1-4) eliminated all matcher_gap inconclusives, the golden corpus settled at 2P / 2F / 6I on both local models. The 6 inconclusives are not matcher gaps — the matcher finds matches for all of them. They are ambiguous_evidence: the trigger matches the reference failure, but it also matches clean successes in the reference corpus. The replay engine can't decide whether the trigger is safe to promote.
Here is the golden re-run data after Fix 1-4:
| Model | Pass | Fail | Inconclusive | Inconclusive breakdown |
|---|---|---|---|---|
| Llama 3.2B | 2 | 2 | 6 | 0 broad_trigger / 0 matcher_gap / 12 ambiguous_evidence |
| Qwen 4B | 2 | 2 | 6 | 0 broad_trigger / 0 matcher_gap / 12 ambiguous_evidence |
Zero matcher_gap on both models — Fix 1-4 confirmed. All 6 inconclusives are ambiguous_evidence. The triggers are specific enough to match the failure, but they also match clean successes. The replay engine says "maybe" — the trigger prevents real failures, but it also breaks real successes. Not safe to promote. Not safe to reject. Inconclusive.
What "specific but broad" looks like
Take G-001: the trigger is "when git push fails with non-fast-forward". This is a specific trigger — it names the tool (git), the action (push), and the error (non-fast-forward). The specificity scorer classifies it as "specific".
But the trigger also matches clean git push trajectories in the reference corpus. A trajectory where the agent runs git push origin feature and it succeeds — the trigger's tokens ("git", "push") appear in that trajectory. The matcher scores it above threshold. The simulator classifies it as broken — the trigger would fire on a clean success and interfere with it.
The trigger is specific about which failure it targets. It is broad about which trajectories it fires on. Those are two different properties, and the benchmark measures both — but the specificity scorer only measures the first.
This is the distinction the v0.2.0 broad-trigger penalty was built to capture. The scorer now distinguishes three cases, documented in learnings-fixes.md §6.2:
| Condition | Verdict | Meaning |
|---|---|---|
broken > prevented |
Fail | Dangerously broad — breaks more than it prevents |
broken > 0 but ≤ prevented |
Inconclusive | Broad but fixable — breaks some, prevents more |
broken == 0 |
Pass (if precision ≥ 0.8) | Safe — prevents failures without breaking successes |
The 6 golden inconclusives fall into the middle bucket: broken > 0 but ≤ prevented. The triggers prevent real failures, but they also break some successes. Not dangerously broad (that would be a fail), but not safe either (that would be a pass). They are broad but fixable.
The broad-trigger penalty in action
The broad-trigger penalty fires across the v0.2.0 sweep. The field test report documents the attribution:
| Corpus | Model | broad_trigger attributions | ambiguous_evidence |
|---|---|---|---|
| golden | Llama 3.2B | 0 | 12 (6 scenarios × 2 models) |
| nearmiss | Llama 3.2B | 3 | 25 |
| failures/positive | Llama 3.2B | 7 | 56 |
The penalty correctly downgrades "matches but breaks successes" from fail to inconclusive. Without it, these would be hard fails — the matcher finds the match, the simulator counts broken successes, and the verdict is "fail" because broken > 0. With the penalty, the verdict is "inconclusive" because broken ≤ prevented — the trigger prevents more failures than it breaks successes. That is a more honest verdict. The trigger is not dangerous. It is not safe. It is in between.
The linter also got a new check: check_broadness() flags triggers with score_specificity == "generic" as broad. This catches the easy case — triggers like "when a command fails" that are both generic and broad. But the harder case — triggers that are specific about the failure but broad about when they fire — is not caught by the linter. The linter checks specificity. The scorer checks breadth. They measure different things.
Why specificity and breadth are different properties
Specificity measures: does the trigger name a concrete failure? "when git push fails with non-fast-forward" is specific. "when a command fails" is generic.
Breadth measures: does the trigger fire on trajectories it shouldn't? A trigger that fires on clean git pushes (because "git" and "push" appear in them) is broad — even if it is specific about the failure it targets.
A trigger can be specific and broad. "when git push fails with non-fast-forward" is specific (names the error) and broad (matches clean git pushes). A trigger can be generic and narrow. "when step_1 fails" is generic (names a step number, not a failure) and narrow (only matches trajectories with step_1 — which is all of them, so actually broad, but in a degenerate way). A trigger can be specific and narrow. "when git push to origin on branch feature fails with non-fast-forward because the remote contains commits you don't have" — this is specific AND narrow. It would only fire on the exact failure. But the model doesn't produce triggers this narrow, and if it did, they would be too narrow to be useful — they would only match one specific scenario.
The sweet spot is specific and narrow enough to be safe but broad enough to be useful. CauterRule's triggers are specific but too broad. The fix is not to make them more specific — they are already specific. The fix is to make them narrower in what they match — which means either narrowing the trigger itself (harder prompt engineering) or improving the matcher's ability to distinguish "matches the failure" from "matches any trajectory with the same tool name" (semantic matching, planned for v0.3.0).
What I learned from this
Specificity is necessary but not sufficient. The model produces specific triggers — 94.4% specific or moderate. The specificity scorer works. The extraction prompt works. And 6 of 10 golden scenarios are still inconclusive because the triggers match too broadly. Specificity measures whether the trigger names a concrete failure. It does not measure whether the trigger fires on the right trajectories and stays silent on the wrong ones. That is breadth, and breadth is what the replay engine measures.
The two properties need separate metrics. The specificity scorer checks "does the trigger name a concrete failure?" The broad-trigger penalty checks "does the trigger break more successes than it prevents failures?" These are different questions. A trigger can score well on one and poorly on the other. The v0.2.0 field test produced triggers that score well on specificity and poorly on breadth — specific but too broad. The fix is not to conflate the two metrics. The fix is to measure both and let the replay engine arbitrate.
The broad-trigger penalty is a correct verdict on an insufficient fix. The penalty correctly classifies "matches but breaks successes" as inconclusive. That is the right verdict — the trigger is not dangerous (fail) and not safe (pass). But it is not a fix. It is a more honest diagnosis. The actual fix — making the trigger narrower in what it matches — requires either semantic matching (the matcher understands that "non-fast-forward" and "authentication error" are different failure classes even though both are "git push fails with X") or reference corpus expansion (more trajectories so the replay engine has more data to distinguish "matches the failure" from "matches the tool"). Both are planned for v0.3.0.
The remaining golden gap is a matching problem, not an extraction problem. The model is extracting the right triggers. The specificity is excellent. The problem is that the matcher can't tell the difference between "the trigger matches the failure" and "the trigger matches any trajectory involving the same tool." That is a semantic matching gap, not a prompt engineering gap. Investing in prompt tuning to make the triggers narrower would make them less useful. Investing in semantic matching would let the replay engine distinguish "fires on the right failure" from "fires on any git push." The fix is downstream of extraction, not upstream.
Open questions
Would a semantic matcher (comparing failure_class between trigger and reference) convert the 6 golden inconclusives to passes? The inconclusives are ambiguous_evidence — the trigger matches the failure and some successes. A semantic matcher that can tell "non-fast-forward" from "clean push" would reduce the broken count, potentially to zero. If broken == 0, the verdict becomes pass. But how much semantic matching is needed — embeddings? A failure-class taxonomy? A simple keyword comparison?
The broad-trigger penalty produces inconclusive, not fail. Is inconclusive the right verdict for "broad but fixable"? The alternative is to fail these triggers — they are too broad to promote. But that would penalize triggers that prevent more failures than they break successes. The penalty's choice — inconclusive for broken ≤ prevented, fail for broken > prevented — is a judgment call. Would a stricter threshold (inconclusive only if broken ≤ prevented * 0.5) produce better promotion decisions?
6 of 10 golden scenarios are ambiguous_evidence. The other 4 are 2 pass, 2 fail. The 2 fails have broken > prevented — dangerously broad. The 2 passes have broken == 0. Is there a middle ground — a trigger that is specific, matches the failure, and breaks zero successes? Or is broken == 0 only achievable when the trigger is so narrow that it only matches the exact reference failure? If so, the golden pass rate ceiling is set by how narrow the model can make triggers without making them useless.
The check_broadness() linter check flags score_specificity == "generic" triggers as broad. But the golden inconclusives are specific triggers that are broad. Should the linter also check breadth — not just "is the trigger generic?" but "does the trigger match too many clean successes?" That would require the linter to run the replay engine, which it currently doesn't. Is that a reasonable thing for a linter to do, or should breadth stay in the replay engine's domain?
The cloud models post-Fix 8 have golden 50% (5P/1F/4I). The 4 inconclusives are the same ambiguous_evidence pattern — specific but broad. Would Fix 8's recovery exclusion help the local models the same way it helped cloud? Local golden is 20% (2P/2F/6I) pre-Fix 8. If Fix 8 moves local golden the same 30 points it moved cloud, local would be at 50% too. But local triggers are broader. Will the recovery exclusion help as much, or will the broader triggers still match too many clean successes?
The broader lesson
Specificity is not safety. A trigger can name the exact failure — "when git push fails with non-fast-forward" — and still fire on every clean git push in your reference corpus. The specificity scorer measures whether the trigger describes a concrete failure. It does not measure whether the trigger fires on the right trajectories and stays silent on the wrong ones.
CauterRule's v0.2.0 field test produced triggers that are 94.4% specific or moderate. The extraction layer is working. And 6 of 10 golden scenarios are inconclusive because the triggers match too broadly. The problem is not extraction. The problem is matching — the replay engine can't distinguish "fires on the failure" from "fires on any trajectory with the same tool."
The fix is not to make the triggers more specific. They are already specific. The fix is to make the matcher smarter — to understand that "non-fast-forward" and "authentication error" are different failure classes even when both are "git push fails with X". That is semantic matching, and it is the v0.3.0 priority.
If you are building a system that extracts rules from failures, measure specificity and breadth separately. Specificity tells you whether the rule describes a real failure. Breadth tells you whether the rule fires only when it should. A rule that is specific but broad is not safe. A rule that is specific and narrow is the goal. And the gap between the two is a matching problem, not an extraction problem.
CauterRule v0.2.0 is released. The full trigger specificity analysis, broad-trigger penalty breakdown, and golden corpus results are in the field test report and the learnings & fixes document. The repo is public. Install with
pip install cauterule. Changelog · Release notes
Top comments (1)
The distinction between specificity and breadth is the exact mechanical vs. semantic boundary. The extraction layer produces a semantically correct rule ("git push fails"), but the replay engine evaluates it mechanically (keyword matching). The matcher sees "git push" and fires, regardless of the exit state, because it lacks a semantic boundary.
Your v0.3.0 plan for semantic matching is the right fix. Without it, the matcher is structurally blind—it cannot distinguish a failure trajectory from a success trajectory if they use the same tool. The rule isn't too broad; the verification layer is just too mechanical. Adding a state/exit-code check or a semantic class comparison is what turns a keyword match into a safe invariant.