Update — v0.2.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, TUI review, observability, 7 export formats, adversarial corpora, and a bundled git rule pack. The field test report evaluated 4 models across 745 trajectories and is the source for every number in this article. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
I spent a week fixing the replay matcher. Four fixes — a precision formula bug, 50+ distinctive phrases, 10 new alias entries, a raised threshold floor. 359 validation tests, all green. The golden pass rate moved from 10% to 20%. One week, four fixes, ten percentage points.
Then I found a 6-line fix in the simulator. It checked 6 keywords in failure_class. Golden jumped from 20% to 50% on both cloud models. One fix, six lines, thirty percentage points.
I was optimizing the wrong layer.
The golden corpus was stuck at 20%
The golden corpus contains 10 canonical failure scenarios — git push non-fast-forward, pip version conflict, docker build package not found, kubectl CRD missing, terraform state lock, pytest assertion, deploy timeout. Each has a known expected rule. The model extracts a trigger, the replay engine checks whether that trigger matches the reference trajectory. A pass means the trigger matches the right failure. The release gate target is ≥70%.
Before the matcher fixes, golden was at 10% — 1 pass, 8 inconclusive, 1 fail. The 8 inconclusives were all matcher_gap: the trigger was specific, but the matcher couldn't bridge it to the reference. "When git push fails with non-fast-forward" should match a trajectory whose error says "! [rejected] non-fast-forward" — but the matcher's token-F1 score was 0.50, below the 0.70 threshold.
The v0.1.0 field test report named this as the #1 engineering target: "Replay and matcher calibration is now the highest-value engineering target." So I fixed the matcher.
The matcher week: Fix 1 through Fix 4
Fix 1 was a precision formula bug. The formula was precision = weighted_hit / len(haystack_tokens) — it divided by the size of the reference trajectory, not the size of the trigger. A 6-token trigger matching 5 of them against a 17-token haystack gave precision 5/17 = 0.29, even though 83% of the trigger was covered. I changed it to divide by the trigger size. On G-001 (git push non-fast-forward), the score went from 0.50 to 0.61. Not enough to pass, but the right direction.
Fix 2 added distinctive phrase fallback. "non-fast-forward", "ModuleNotFoundError", "OOMKilled" — 50+ error codes and phrases. If a distinctive phrase appears in the raw trigger and its normalized form appears in the haystack, floor the score at 0.70. On G-001, this pushed the score from 0.61 to 0.70. It passed.
Fix 3 expanded the alias map from 11 to 22 entries — "version conflict" → "dependency resolver conflict", "package not found" → "unable to find", "state lock" → "error acquiring". Fix 4 raised the alias phrase floor from 0.65 to 0.70, so that a verbatim alias match in the haystack counts as a pass.
Here is the score progression for G-001 across all four fixes, traced in the learnings & fixes log:
| Fix | What changed | G-001 score |
|---|---|---|
| Before all fixes | — | 0.50 |
| Fix 1 (precision formula) |
/ haystack → / trigger
|
0.61 |
| Fix 2 (distinctive phrases) | "non-fast-forward" floors at 0.70 | 0.70 |
| Fix 3 (expanded aliases) | Covered G-005, G-010 | 0.70 |
| Fix 4 (raised floor 0.65→0.70) | Alias phrase hits now pass | 0.70 |
Combined impact: golden inconclusive dropped from 80% to 0%. Every trigger now produces a meaningful verdict. The matcher finds matches for all 10 golden scenarios. 359 validation tests, 0 failures. The matcher was fixed.
But the golden pass rate only moved from 10% to 20%. 2 passes, 8 fails, 0 inconclusive. The 8 "fail" verdicts meant the triggers matched the reference failure — but they also matched clean successes in the reference corpus. The triggers were too broad.
The matcher was fixed. The pass rate was not.
The 6-line simulator fix
The problem was not in the matcher. It was in the simulator — the component that classifies what happens when a trigger matches a reference trajectory.
The simulator takes a candidate trigger and runs it against every reference trajectory. For each trajectory, it asks: does this trigger fire? If it fires on a trajectory that was a failure, that's prevented — the trigger would have caught a real failure. If it fires on a trajectory that was a success, that's broken — the trigger would have interfered with a correct outcome. The classification logic was simple: if trajectory.success == True and the trigger matches, classify as broken. If trajectory.success == False and the trigger matches, classify as prevented.
But some success=True trajectories are not clean successes. They are recoveries — the agent failed, then self-resolved. A trajectory with failure_class = "temp error, retry succeeded" and success = True is a near-miss, not a clean success. The agent encountered a failure, recovered from it, and the final outcome was success. But the failure was real — it just didn't stick.
The simulator was counting these as broken. A trigger that fires on a recovery trajectory is not breaking a success — it is firing on a near-miss that happened to recover. Penalizing the trigger for matching a recovery is wrong. The fix, documented in learnings-fixes.md §4.8, is 6 lines:
RECOVERY_KEYWORDS = {"temp", "near", "retry", "recover", "intermittent", "flaky"}
if trajectory.success and any(kw in trajectory.failure_class for kw in RECOVERY_KEYWORDS):
classification = "near_miss" # not "broken"
The numbers
I re-ran the cloud sweep on golden, failures/positive, and nearmiss after Fix 8. The full results are in the v0.2.0 field test report §2:
| Corpus | Model | Pre-Fix 8 | Post-Fix 8 | Delta |
|---|---|---|---|---|
| golden | gpt-4o-mini | 2P / 2F / 6I (20%) | 5P / 1F / 4I (50%) | +3 passes |
| golden | llama-3.1-8b | 2P / 2F / 6I (20%) | 5P / 1F / 4I (50%) | +3 passes |
| failures/positive | gpt-4o-mini | 15P / 6F / 29I (30%) | 22P / 5F / 23I (44%) | +7 passes |
| failures/positive | llama-3.1-8b | 15P / 7F / 28I (30%) | 27P / 5F / 18I (54%) | +12 passes |
| nearmiss | gpt-4o-mini | 2 FPs (96% precision) | 5 FPs (90% precision) | +3 FPs |
| nearmiss | llama-3.1-8b | 5 FPs (90% precision) | 7 FPs (86% precision) | +2 FPs |
Golden jumped 20% → 50% on both cloud models. Failures/positive jumped 30% → 44-54% — llama-3.1-8b now meets the ≥50% release threshold for the first time. Nearmiss false positives rose slightly (2→5 on gpt-4o-mini, 5→7 on llama-3.1-8b) — an acceptable tradeoff for the golden and failures gains, but a tradeoff, not a free win.
Fix 8 moved the golden pass rate more than Fix 1-4 combined. One fix, six lines, thirty percentage points. The matcher week gave me ten. The simulator fix gave me thirty.
Why the simulator was the right layer
The matcher fixes were necessary. Without them, 80% of golden scenarios were inconclusive — the matcher couldn't find matches. Fix 1-4 eliminated the inconclusive bucket. Every trigger now gets a decisive verdict. That felt like progress, and it was — but only half the progress I needed.
A decisive verdict is not the same as a correct verdict. The matcher fixes converted inconclusive to fail. The triggers now match — but they match too broadly, so the replay engine says "fail" instead of "inconclusive." The problem shifted from "the matcher can't decide" to "the matcher decides wrong."
The wrong decision was in the simulator's classification. Recovery trajectories with success=True were counted as broken — the trigger broke a success. But they weren't successes. They were recoveries. The trigger didn't break anything — it fired on a near-miss that self-resolved. Fix 8 reclassifies these as near_miss, so the trigger is no longer penalized for matching them.
This is why Fix 8 outperformed Fix 1-4. The matcher fixes addressed the matching layer — can the trigger be found in the reference? The simulator fix addressed the classification layer — when the trigger is found, what does that mean? The matching layer was necessary but not sufficient. The classification layer was where the actual pass rate lived.
Why I missed it for a week
I missed it because the matcher fixes were working. Inconclusive dropped from 80% to 0%. Every trigger got a verdict. The validation suites passed — 359 tests, 0 failures. The field test report's #1 engineering target was "replay and matcher calibration," and I was fixing the matcher. Every signal I had said I was working on the right layer.
The signal I didn't have was the attribution of the remaining 8 "fail" verdicts. The report said the triggers were "too broad" — matching successes they would break. I accepted that framing. The triggers are specific (94.4% specific or moderate per the field test report §A.9), but they match too broadly. The fix must be to make them narrower — prompt tuning, trigger specificity checks, broad-trigger penalty.
But the triggers weren't too broad. The reference corpus was misclassified. The "successes" the triggers were breaking weren't successes — they were recoveries. The triggers were matching correctly. The simulator was classifying incorrectly. The broad-trigger penalty I built — broken > prevented → fail — was working correctly on bad data.
I spent a week optimizing the matcher because the report said the matcher was the problem. The report was right about v0.1.0 — the matcher's 80% inconclusive rate was the #1 issue. But after Fix 1-4 eliminated the inconclusives, the #1 issue shifted to the simulator's classification logic. The report didn't update because I hadn't re-run the field test yet. I was working from a stale diagnosis.
What I learned from this
The biggest fix is often not in the layer the report names. The v0.1.0 report named the matcher as the #1 target. It was right — for v0.1.0. But after fixing the matcher, the #1 target shifted to the simulator. The report didn't update because the field test hadn't been re-run. I was optimizing against a stale diagnosis. Re-run the field test after every fix. Let the new data tell you where the next bottleneck is.
A decisive verdict is not the same as a correct verdict. Fix 1-4 eliminated inconclusives — every trigger now gets a pass or fail. That felt like progress. But the 8 "fail" verdicts were wrong — the triggers were being penalized for matching recoveries, not successes. Converting inconclusive to fail is only progress if the fail verdicts are correct.
Classification bugs masquerade as model quality problems. Golden pass rate was 20% across all four models — local and cloud. That looked like a model capability ceiling. Every explanation pointed to the model or the corpus. None pointed to the simulator's 6-line classification logic. Classification bugs are invisible because they affect all models uniformly — if every model gets the same wrong verdict, it looks like a system property, not a bug.
One fix can outperform four. Fix 1-4 gave me 10 percentage points. Fix 8 gave me 30. The matcher fixes were necessary foundations — without them, Fix 8 would have had no verdicts to improve. But the lever that moved the pass rate was not in the matcher. It was in the simulator. Do not over-invest in one layer just because the report named it first.
Open questions
The remaining 4 golden inconclusives on cloud (50% vs 70% target) are trigger-breadth issues — triggers match the reference but also match clean successes. Would reference corpus expansion (currently 230 trajectories, per learnings-fixes.md §7.1) to 330-430 give the simulator enough data to distinguish "this trigger matches real failures" from "this trigger matches too broadly"? Or is the issue that the triggers themselves need to be narrower?
The nearmiss false positives rose slightly after Fix 8 (gpt-4o-mini 2→5, llama-3.1-8b 5→7). These are "wrong failure" scenarios — "git push fails with authentication error" matching "git push fails with non-fast-forward". Fix 8 doesn't address them because they have success=False. Would a failure_class mismatch check in the matcher catch these without hurting the golden gains?
Fix 8 has not been re-run on local OMLX models yet. The fix is model-independent — it changes simulator classification, not extraction. But local models produce broader triggers. Will the recovery exclusion help them as much as it helped cloud, or will the broader triggers still match too many clean successes?
The 6 recovery keywords ("temp", "near", "retry", "recover", "intermittent", "flaky") were chosen from the annotated corpus vocabulary. Are there recovery patterns in the corpus that these keywords miss? A trajectory with failure_class = "transient network blip" would not match any of the 6. Would a semantic classifier (embeddings on failure_class) be more robust than keyword matching?
If I had re-run the field test after Fix 1-4 instead of continuing to Fix 5-7, would the data have pointed me to the simulator sooner? The 8 "fail" verdicts with broken > prevented were the clue — the triggers were matching recoveries, not successes. But I didn't look at which specific trajectories were being counted as "broken." I accepted the "too broad" framing and moved to the broad-trigger penalty.
The broader lesson
When a bottleneck persists across every model — local and cloud, 3B and 8B — the problem is almost never in the model. It is in the evaluation layer that classifies the model's output. A model-independent failure is a classification failure.
CauterRule's golden pass rate was stuck at 20% for a week because I was fixing the matcher when the simulator was the problem. The matcher was the named bottleneck in the v0.1.0 report, and it was right — for v0.1.0. But after the matcher was fixed, the bottleneck shifted. I didn't notice because I didn't re-run the field test. I kept optimizing the named layer instead of letting the data re-diagnose.
The 6-line fix that moved golden 20% → 50% was not a clever insight. It was a classification bug that affected all models uniformly. It looked like a model ceiling because every model hit the same 20%. It was a 6-line bug in the simulator that counted recoveries as successes.
If a bottleneck persists across every model you test, stop tuning the model. Stop tuning the matcher. Look at the layer that classifies the output — the layer that decides what a "pass" and a "fail" mean. That is where model-independent failures live. And that is where the biggest fixes hide.
CauterRule v0.2.0 is released. The full fix-by-fix breakdown — Fix 1 through Fix 8, with before/after scores on every golden scenario — is in the field test report and the learnings & fixes document. The repo is public. Install with
pip install cauterule. Changelog · Release notes
Top comments (0)