Update — v0.3.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.0 field test report evaluated 2 cloud models across 40 corpora and 4,768 trajectory-runs and is the source for every number below. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
For two field tests, one number refused to move: golden recall sat at 0.087. The reports both named the same culprit — "replay and matcher calibration is now the highest-value engineering target." So we did the obvious thing and worked on the matcher.
The matcher was not the problem. The denominator was.
The symptom: a good rule scored almost zero
Replay grades a candidate trigger against reference trajectories. Recall is the fraction of failures the trigger catches:
precision = prevented / (prevented + broken)
recall = prevented / total_failures
A v0.2.0 candidate that prevented 3 real failures — a genuinely useful rule — was scored 3 / ~200 = 0.015. No threshold can admit 0.015 without also admitting noise. The rule wasn't weak. The denominator was the entire corpus.
Ask the question the metric was actually answering: "does this git rule also prevent docker, python, terraform, and browser failures?" Of course it doesn't. It was never supposed to.
The assumption we made (and where it broke)
We built replay on one implicit assumption:
Assumption: every failure in the reference pool is a fair test case for every candidate rule.
That assumption is wrong whenever candidates are domain-specific — which, for a real rule engine, is always. The consequence is subtle and dangerous: the metric doesn't fail loudly. It just makes good rules look worthless, uniformly, across every model. And a metric that makes everything look equally bad is a metric that hides where the actual problem is.
We also assumed recall was a model measurement. It is a ratio, and we were controlling the denominator as if it were irrelevant.
The fix: scope the reference pool to the source domain (§708)
v0.3.0 filters reference_trajs to the source trajectory's domain before replay:
# scope the comparison set to the candidate's own domain
references = [t for t in reference_pool if t.domain == candidate.source_domain]
prevented = sum(1 for t in references if rule_matches(candidate, t) and not t.success)
recall = prevented / len([t for t in references if not t.success])
The pool did not shrink — it grew from 230 to 444 references. What changed is the slice each candidate is judged against:
| Candidate domain | Scoped references | Global pool (old) |
|---|---|---|
| git | 19 | ~200 |
| python | 30 | ~200 |
| docker | 30 | ~200 |
A rule that prevents 3 git failures is now scored 3 / ~27 ≈ 0.11, not 3 / 200 = 0.015.
We made one more change alongside it, and it deserves to be stated plainly: the pass threshold dropped from 0.8 to 0.5. That is not a matcher fix. It is the acknowledgement that once scores are computed honestly, 0.8 was calibrated to an inflated scale. Report both or you are hiding the trick.
The data
Golden and failures/positive recall, both cloud models, pre- and post-fix:
| Corpus | Model | Recall pre | Recall post | Delta |
|---|---|---|---|---|
| golden | gpt-4o-mini | 0.068 | 0.170 | 2.5× |
| golden | llama-3.1-8b | 0.104 | 0.228 | 2.2× |
| failures/positive | gpt-4o-mini | 0.068 | 0.182 | 2.7× |
| failures/positive | llama-3.1-8b | 0.104 | 0.277 | 2.7× |
No model change. No prompt change. No change to the core matcher scoring. Golden pass rate moved to 40% (gpt-4o-mini) and 50% (llama-3.1-8b).
What worked
- Recall became diagnosable. Against a global pool, everything looked equally hopeless. Against a scoped pool, the recall signal separates a specific rule from a vague one.
- The fix is model-independent. It held across both models — the signature of an evaluation bug, not a capability ceiling.
- It exposed the real bottleneck. Once the denominator was honest, the remaining failures pointed at the matcher's inability to read paraphrases — a concrete, addressable problem.
What didn't work
- Recall is still far below target. 0.170–0.228 against a 0.70-ish ambition. Doubling a small number leaves a small number.
- The paraphrase gap is untouched. Semantic matching runs at a 0.2 blend weight and cannot bridge "non-fast-forward" against "Updates were rejected because the remote contains work that you do not have locally." Same event, different tokens.
- Scoping could flatter reference-rich domains. python and docker have ~30 references; git has 19. We have not measured whether thin domains are systematically penalized.
- Changing two things at once blurs attribution. Scoping and the threshold drop landed together; the 2–3× is scoping's, but the pass-rate movement is both.
Questions we still can't answer
- Would a 0.4–0.5 semantic weight close golden, or does the token-F1 term still dominate the blend?
- Is domain the right scope, or should it be failure-class? We chose domain because the label already exists on every trajectory — a convenience assumption.
- At what reference count per domain does scoping stop helping? We have no curve, only two field tests.
- If a candidate has no domain label, do we fall back to the global pool, and does that reintroduce the bug?
What I learned
Evaluation-set composition is a product decision, not a detail. "Recall against all failures" sounds rigorous and is actually a different question than the one you care about.
Uniformly bad numbers across every model usually mean an evaluation bug. When gpt-4o-mini and llama-3.1-8b agree that everything is hopeless, suspect the denominator before the model.
Scoping is not cherry-picking — if it is principled and pre-registered. We scope by a trajectory property fixed before scoring. We are not removing failures a candidate missed; we are removing failures it was never meant to catch. If that boundary moves after you see the results, you are just tuning the metric until it flatters you.
Recall is a floor. Every reference you add lowers recall without changing the rule. Report it as "at least this many," never as "this many."
The broader lesson
If you score a specific output against a set of examples, ask what the denominator means before you optimize the scorer.
CauterRule spent two releases blaming a matcher for a number the reference pool was manufacturing. The fix was not a better algorithm. It was recognizing that "recall" should mean recall within the problem this rule is about. Change the comparison slice and the same model — the same everything — looks 2–3× better and, more importantly, becomes diagnosable.
References
- CauterRule v0.3.0 release notes
- v0.3.0 field test report
- Per-model cloud comparison
- Threshold calibration (§708 / threshold changes)
- PRD — success metrics
- Corpus format spec (domain labels)
- User guide · Changelog
CauterRule v0.3.0 is released. The full recall data and the 40-corpus × 2-model matrix are in the field test report. The repo is public. Install with
pip install cauterule. Changelog · Release notes
Top comments (0)