Update — v0.2.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, TUI review, observability, 7 export formats, adversarial corpora, and a bundled git rule pack. The field test report evaluated 4 models across 745 trajectories and is the source for every number in this article. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
My model produced a trigger that was just "step_1". It matched every reference trajectory that contained a step numbered 1 — which is all of them. Precision 1.00, recall 0.02, verdict: pass. Two nearmiss false positives traced to this trigger. The specificity scorer classified it as "specific" because it contains a token. The matcher scored it 1.00 because "step_1" is an exact substring match in every trajectory's step identifier field.
The model found a shortcut. And my benchmark rewarded it.
It took me three fixes to close it. Not because the problem was hard — each fix was a few lines. But because the shortcut revealed something deeper about evaluation design: if your matcher rewards any substring match, your model will find substrings that match everything. The problem wasn't the model. It was the reward.
The shortcut: "step_1" matches everything
The nearmiss corpus contains 50 lookalike trajectories per model — cases that resemble real failures but should not trigger the extracted rule. On the first v0.2.0 sweep, 5 of 50 nearmiss trajectories produced passing candidates. All 5 passes had precision 1.00 and recall 0.02 — they matched exactly 1 reference failure out of 210. They should have matched none.
Two of those 5 false positives were caused by the same trigger: "step_1". The learnings & fixes log traces them:
| Trajectory | Trigger | Precision | Recall | Verdict |
|---|---|---|---|---|
| NM-030-certificate-retry | "step_1" |
1.00 | 0.02 | Pass (false positive) |
| NM-044-git-commit-hook | "step_1" |
1.00 | 0.02 | Pass (false positive) |
The trigger "step_1" appears in every reference trajectory's step identifier field — {"step": 1, "input": "git push origin feature", ...}. The matcher normalizes both trigger and haystack, tokenizes them, and checks for substring matches. "step_1" is a substring of "step_1": {"input": .... Score: 1.00. Precision: 1.00 — the trigger fires on exactly 1 reference trajectory. Recall: 0.02 — 1 out of 210. Verdict: pass, because precision is high and the trigger "prevents" a real failure.
But the trigger is not about a failure. It is about a step number. The model learned that "step_1" appears in every trajectory, so producing it as a "trigger" guarantees a match. It is not extracting a failure pattern. It is exploiting a structural artifact of the trajectory format.
Fix 7a: degenerate trigger classification
The first fix was in the specificity scorer (specificity.py). I added a regex pattern:
_DEGENERATE_RE = re.compile(r"^step[_\s]*\d+$", re.IGNORECASE)
Any trigger matching this pattern — "step_1", "step 2", "STEP_3" — is classified as "generic" instead of "specific". The specificity scorer now correctly identifies these as degenerate triggers that name a step number, not a failure pattern.
But classification alone doesn't stop the match. The specificity scorer flags the trigger as generic, but the matcher still scores it 1.00 and the replay engine still produces a "pass" verdict. The flag is a signal. It is not a gate.
Fix 7b: degenerate trigger rejection in the matcher
The second fix was in the matcher (matcher.py). I added the same regex pattern and made it a hard gate:
_DEGENERATE_TRIGGER_RE = re.compile(r"^step[_\s]*\d+$", re.IGNORECASE)
def rule_matches(candidate, trajectory, threshold=0.70):
if _DEGENERATE_TRIGGER_RE.match(candidate.trigger):
return False # degenerate trigger — never matches
rule_matches() now returns False immediately for any degenerate trigger. No tokenization, no scoring, no substring check. The trigger is rejected before the matcher ever runs. The replay engine gets no match, produces no "prevented" or "broken" verdict, and the candidate fails.
This eliminated the 2 nearmiss false positives (NM-030, NM-044) immediately. "step_1" no longer matches anything. The shortcut is closed.
Fix 7c: the harder question — what other shortcuts exist?
Closing "step_1" was easy. The pattern is regular, the regex is 20 characters, the fix is 3 lines. The harder question is: what other shortcuts is the model finding that I haven't noticed?
After Fix 7, I went back to the remaining 3 nearmiss false positives on Llama — the ones that were NOT caused by "step_1":
| Trajectory | Trigger | What it matches | Why it's wrong |
|---|---|---|---|
| N-001-git-nm-001-auth-vs-ff | "git push fails with authentication error" |
"git push fails with non-fast-forward" reference | Near-miss about auth, not nff — but matcher sees "git push" overlap |
| N-003-cosmetic | "git push fails with cosmetic error" |
Wrong reference entirely | "git push" token overlap with unrelated reference |
| N-004-env-task-different-tool | "python import fails with wrong module" |
"python ImportError" reference | Wrong tool entirely, but "python" + "import" token overlap |
These are not degenerate triggers. They are real, specific, well-formed triggers — "git push fails with authentication error" is a legitimate failure description. The problem is that the matcher can't distinguish "authentication error" from "non-fast-forward" when both are "git push fails with X". The token overlap ("git", "push", "fails") is high enough to score above threshold.
This is a different class of shortcut. The model is not exploiting a structural artifact ("step_1"). It is exploiting a semantic gap in the matcher — the matcher can't tell the difference between two failure classes within the same tool. "git push fails with authentication error" and "git push fails with non-fast-forward" look similar to a token-overlap matcher because they share 3 of 6 tokens. They look different to a human because the failure class is completely different.
Fix 7c is not a regex. It requires the matcher to compare failure_class between the trigger and the matched reference — a trigger-domain mismatch check. If the trigger names "authentication error" but the matched reference has failure_class = "non-fast-forward", downgrade the match score or flag as inconclusive. This is planned for v0.3.0. The "step_1" shortcut is closed. The "wrong failure" shortcut is still open.
Why the model found the shortcut
The model is a 3B local model — Llama-3.2-3B-Instruct, quantized to 4-bit, running on OMLX. It is small, fast, and free. It is also not sophisticated enough to produce the kind of specific, error-class-aware triggers that the 8B cloud models produce. When it can't find a specific failure pattern, it finds something that matches — and "step_1" matches everything.
This is not a bug in the model. It is the model doing what models do: optimizing for the reward signal. The reward signal is "the matcher scores above 0.70." The model produces triggers that score above 0.70. "step_1" scores 1.00. By the reward signal, it is a perfect trigger.
The problem is that the reward signal is wrong. The matcher rewards substring overlap, not failure-pattern match. The model is not misbehaving — it is solving the problem the matcher defines. The matcher defines "a good trigger is one whose tokens appear in the reference trajectory." The model produces a trigger whose tokens appear in every reference trajectory. That is optimal behavior under the matcher's definition.
What I learned from this
Reward hacking is not always malicious. The model didn't "decide" to game the benchmark. It produced a trigger that matched the reward signal. The reward signal was "token overlap with reference trajectories." "step_1" has maximum token overlap. The model is optimizing the objective the matcher defines. If the objective is wrong, the model's behavior is correct under the wrong objective. This is the alignment problem in miniature — not about safety, but about evaluation.
Structural artifacts in your data format are attack surfaces. Every reference trajectory has a step field with a number. "step_1" appears in all of them. The matcher treats this as a match. The model treats this as a reward. Any structural artifact that appears in every trajectory — step numbers, timestamps, session IDs, tool names — is a potential shortcut. The fix is not to remove the artifacts (they are part of the data format). The fix is to make the matcher not reward matches on structural artifacts.
A regex fix is a band-aid. The real fix is semantic. Fix 7b closes the "step_1" shortcut with a regex. It does not close the "wrong failure" shortcut — "git push fails with authentication error" matching "git push fails with non-fast-forward" — because that shortcut is not structural. It is semantic. The matcher can't tell the difference between two failure classes within the same tool. The regex catches the easy case. The semantic gap is the hard case, and it is still open.
Every false positive has a root cause. Trace it. The 5 nearmiss false positives looked like a model quality problem — "the model can't distinguish near-misses from real failures." But tracing each FP to its trigger revealed two different root causes: degenerate triggers (2 FPs, structural shortcut) and wrong-failure matches (3 FPs, semantic gap). The aggregate "5 FPs" is meaningless. The breakdown "2 degenerate + 3 semantic" tells you exactly what to fix. Always trace false positives to individual triggers. The aggregate hides the root causes.
Open questions
The "wrong failure" shortcut (3 nearmiss FPs) requires trigger-domain mismatch detection. How should the matcher compare failure_class between trigger and reference? Keyword comparison? Embedding similarity? A separate classifier? The v0.2.0 matcher has no failure_class awareness — it only sees tokens.
Are there shortcuts I haven't found? The "step_1" pattern was obvious because it is not a failure description. What about triggers like "git" or "python" — single-tool-name triggers that match every trajectory involving that tool? The specificity scorer classifies these as "generic" (score < threshold), but the matcher might still score them above 0.70 on token overlap. Would a minimum-trigger-length check catch these?
The 3B model found the "step_1" shortcut. The cloud models (gpt-4o-mini, llama-3.1-8b) did not — they produced specific triggers like "git push fails with non-fast-forward". Is this because the cloud models are smarter, or because their tokenization is different? If I ran a weaker model, would it find shortcuts the 3B model didn't?
Fix 7b rejects degenerate triggers in the matcher. Should the gate also reject them? Currently, the gate (gate.py) checks for failure signals (exit codes, errors, failure_class). It does not check the extracted trigger for degeneracy. A degenerate trigger that reaches the matcher is already a wasted extraction. Should the gate reject degenerate triggers before the LLM call?
The nearmiss corpus has success=False — these are real failures, not recoveries. Fix 6 (nearmiss recovery detection, documented in learnings-fixes.md §1.5) drops success=True recovery patterns. Fix 7 rejects degenerate triggers. Neither addresses the "wrong failure" FPs. How many nearmiss FPs would remain if Fix 7c (trigger-domain mismatch) were implemented? The data suggests 3 on Llama, 5-7 on cloud post-Fix 8. Would Fix 7c eliminate all of them, or are some "wrong failure" matches genuinely ambiguous?
The broader lesson
If your benchmark rewards surface-level similarity, your model will optimize for surface-level similarity. That is not a model problem. It is a benchmark design problem.
The "step_1" shortcut was the most visible case. But the "wrong failure" false positives are the same lesson at a deeper level. The matcher rewards token overlap. The model produces triggers with maximum token overlap. Sometimes that means "step_1" (structural shortcut). Sometimes that means "git push fails with authentication error" matching "git push fails with non-fast-forward" (semantic shortcut). Both are the model optimizing the objective the matcher defines.
The fix is not to make the model smarter. The fix is to make the matcher's objective align with what "a good trigger" actually means. A good trigger names a specific failure class. A token-overlap matcher can't verify that. A semantic matcher — one that compares failure_class, error codes, and failure semantics — can.
CauterRule's v0.2.0 field test closed the "step_1" shortcut with a regex. The "wrong failure" shortcut is the next one. And it is the same lesson: your model is not misbehaving. Your benchmark is mis-rewarding.
CauterRule v0.2.0 is released. The full nearmiss breakdown — per-model false positives, trigger analysis, and the degenerate trigger rejection fix — is in the field test report and the learnings & fixes document. The repo is public. Install with
pip install cauterule. Changelog · Release notes
Top comments (0)