Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote.
pip install cauterulegives you the full CLI, MCP server, 7 export formats, and a bundled git rule pack. The field test report evaluated 4 models across 394 trajectories and is the source for every number in this article. Release notes · Changelog
CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.
Every benchmark produces three buckets: pass, fail, and inconclusive. Most people only read the first two. In CauterRule's field test, the third bucket was bigger than both of them combined — and it was the one that mattered most.
The majority of our results were "can't tell"
Across all four models and the full corpus, the field test produced 1,538 candidates. Here is how they split:
| Verdict | Count | % of total |
|---|---|---|
| Pass | 365 | 23.7% |
| Fail | 359 | 23.3% |
| Inconclusive | 814 | 52.9% |
More than half of every result the benchmark produced was inconclusive. The replay engine could not decide whether the rule was good or bad. It did not pass the candidate. It did not reject it. It returned "maybe."
That is not a fringe bucket. It is the majority of the output. And it is the bucket that most benchmark reports skip entirely, because it is uncomfortable to talk about. A pass is a win. A fail is a loss. An inconclusive is an embarrassment — it means your evaluation tool could not do its job.
But if you ignore the inconclusive bucket, you are ignoring more than half of your data. And you are probably drawing the wrong conclusions from the half you kept.
The inconclusive rate varied by model — but not the way you'd expect
| Model | Candidates | Inconclusive | Inconclusive % |
|---|---|---|---|
| Local Llama 3.2B | 379 | 189 | 49.9% |
| Local Qwen 4B | 373 | 209 | 56.0% |
| Cloud GPT-4o-mini | 394 | 248 | 62.9% |
| Cloud Llama 3.1 8B | 392 | 168 | 42.9% |
The strongest cloud model — meta-llama/llama-3.1-8b-instruct — had the lowest inconclusive rate at 42.9%. That makes sense: better extraction produces more specific triggers, which are easier for the replay engine to verify.
But gpt-4o-mini — a paid cloud model — had the highest inconclusive rate at 62.9%. Higher than both local models. This is the kind of finding that only surfaces when you stop reading pass counts and start reading the full distribution. On raw pass count, GPT-4o-mini looked comparable to the locals (77 vs 72 vs 93). On inconclusive rate, it was the most indecisive model in the field test.
That does not mean GPT-4o-mini is a bad model. It means the model produced candidates that the replay engine could not evaluate. The model's output was not worse — it was less scorable. Those are different failures, and they point to different fixes.
What "inconclusive" actually means
The replay engine takes a candidate rule and runs it against reference trajectories. It checks whether the rule's trigger fires on the right cases (recall) and stays silent on the wrong ones (precision). If both precision and recall are high enough, the verdict is pass. If the rule clearly fires on wrong cases or misses all right cases, the verdict is fail.
If neither condition is clearly met, the verdict is inconclusive.
The matcher that decides this is not a sophisticated judge. It uses substring and token-overlap heuristics. It checks whether words in the trigger appear in the trajectory. If the overlap is in a gray zone — not high enough to pass, not low enough to fail — the matcher returns inconclusive.
This is not a secret. The field test report names it explicitly as the #1 engineering target:
Replay and matcher calibration is now the highest-value engineering target.
The report also says:
The matcher and simulator are still heuristic and can over- or under-fire on substring/token overlap.
That means the inconclusive bucket is contaminated. It contains at least two different populations:
Rules that are too vague to verify — the model extracted a generic trigger like "when a command fails" instead of "when git push fails with non-fast-forward." The matcher cannot tell whether this fires on the right cases because the trigger is too broad to evaluate. This is a model problem.
Rules that are specific and reasonable but the matcher is too weak to confirm — the model extracted a good rule, but the substring/token-overlap heuristic cannot recognize that the trigger matches the reference trajectory. This is an engine problem.
The field test data cannot cleanly separate these two populations. That is itself a finding: the benchmark is not instrumented well enough to tell you why a result is inconclusive. And without that, you cannot tell whether the inconclusive bucket is mostly model weakness or mostly engine weakness.
The curated vs raw split reveals where inconclusives concentrate
The sharpest insight comes from splitting the inconclusive rate by corpus type.
| Model | Curated inconclusive % | Raw inconclusive % |
|---|---|---|
| Local Llama 3.2B | 33.0% | 55.2% |
| Local Qwen 4B | 34.0% | 63.4% |
| Cloud GPT-4o-mini | 38.3% | 70.7% |
| Cloud Llama 3.1 8B | 14.9% | 51.7% |
On curated corpora, the inconclusive rates are manageable — especially for the strongest cloud model (14.9%). On raw corpora, every model jumps above 50%. GPT-4o-mini hits 70.7%.
Two raw corpora drove most of the inconclusive volume:
| Raw corpus | Llama 3.2B inc % | Qwen 4B inc % | GPT-4o-mini inc % | Llama 8B inc % |
|---|---|---|---|---|
raw/ci |
45.0% | 81.8% | 90.0% | 54.5% |
raw/sibling-repos |
90.0% | 90.0% | 70.0% | 100.0% |
raw/ci produced 110 trajectories per model. For GPT-4o-mini, 99 of those 110 were inconclusive — 90%. For Qwen 4B, 90 of 110 — 82%. The replay engine was essentially paralyzed on this corpus.
raw/sibling-repos was worse: 10 trajectories per model, and for the strongest cloud model, all 10 were inconclusive. 100%. The engine could not produce a single decisive verdict.
The report's interpretation is direct:
Raw corpora, especially
raw/ci, produce many inconclusive outcomes because the current replay setup is a weak fit for that breadth.
raw/sibling-reposremains mostly inconclusive for both cloud models, meaning harness-transfer quality is still not strong.
That is an important distinction. The inconclusive problem is not evenly distributed. It is concentrated in raw corpora where the replay engine's heuristics are a poor fit for the data shape. On curated corpora — where the trajectories are clean, the failure signatures are clear, and the expected rules are known — the engine performs much better.
Why better models did not solve this
If inconclusives were mostly a model problem, stronger models should reduce the inconclusive rate. They did, but only partially.
| Model | Inconclusive % | Pass % | Fail % |
|---|---|---|---|
| Local Llama 3.2B | 49.9% | 19.0% | 31.1% |
| Local Qwen 4B | 56.0% | 24.9% | 19.0% |
| Cloud GPT-4o-mini | 62.9% | 19.5% | 17.5% |
| Cloud Llama 3.1 8B | 42.9% | 31.4% | 25.8% |
The strongest model reduced inconclusives from ~50-63% to ~43%. That is a real improvement, but it left the inconclusive bucket as the largest category. Even the best model in the field test could not get the replay engine to a decisive verdict on the majority of its output.
And the cloud model with the best parse stability — gpt-4o-mini — had the worst inconclusive rate. Perfect formatting did not help the matcher decide. That confirms what the report says: better structured output does not automatically translate to good replay outcomes. The bottleneck is not extraction. It is judgment.
This is the same conclusion the field test report reaches from a different angle. The cloud runs "proved that better structured output does not automatically translate to good safety behavior or high replay pass rates." The inconclusive data is the concrete evidence for that claim.
What the inconclusive bucket costs you
The inconclusive bucket is not neutral. It has real costs that compound if you ignore it.
It makes your pass rate look lower than it is. If 53% of your results are inconclusive, your pass rate is measured against a denominator that includes a large chunk of "we don't know." The true pass rate — the rate at which the model produces good rules — is somewhere between the reported pass rate and pass-plus-inconclusive. For Llama 8B, that range is 31.4%–74.2%. That is a massive uncertainty band, and it means the pass count is a floor, not a ceiling.
It makes your fail rate look lower than it is. The same logic applies to fails. Some inconclusives are rules that should have been rejected but the matcher could not prove it. Those are hidden failures — safety risks that do not show up in the fail count because the engine was too weak to catch them.
It blocks promotion. In a production system, inconclusive rules are not promoted. They sit in a queue waiting for human review or better evaluation. If 53% of your output is inconclusive, your promotion pipeline is choked. You are producing candidates faster than you can adjudicate them.
It hides the real product problem. When you cannot tell whether a rule is good or bad, you cannot tell whether your product is good or bad. The inconclusive bucket is a measurement gap, and measurement gaps are where product risks hide. The report is explicit: "the benchmark does not yet turn that processing into consistently actionable quality signals."
What a better judge would look like
The current replay engine is a string matcher. It checks whether words in the trigger appear in the trajectory. That is a reasonable starting point, but it is not sufficient for the problem CauterRule is trying to solve.
A better judge would need to do three things the current engine cannot:
Semantic matching. Instead of substring overlap, the matcher needs to understand whether a trigger like "when a non-fast-forward push is rejected" fires on a trajectory where the error message says "Updates were rejected because the remote contains work that you do not have locally." Those are the same event described differently. A string matcher misses that. A semantic matcher — even a lightweight embedding-based one — would catch it.
Corpus-aware calibration. The same matcher threshold does not work for all corpora. Curated corpora with clean failure signatures can tolerate stricter matching. Raw corpora with messy, multi-step trajectories need looser matching or a different matching strategy entirely. The report says the replay setup is "a weak fit for that breadth." The fix is not one better matcher. It is a matcher that adapts to the corpus type.
Inconclusive attribution. When the matcher returns inconclusive, it should say why. Is the trigger too broad? Is the trajectory too messy? Is the matcher's vocabulary too narrow? Without attribution, every inconclusive looks the same, and you cannot prioritize fixes. The current benchmark lumps all inconclusives together, which is why the report says "the benchmark still mixes several failure modes together."
What I learned from this
The inconclusive bucket is the most important column in the table. Pass and fail tell you what the system decided. Inconclusive tells you what the system could not decide. In a product where the core value proposition is "we promote only safe rules," the inability to decide is a bigger risk than a wrong decision. A wrong decision can be caught and reverted. An undecidable result sits in limbo forever.
Your benchmark is only as good as its judge. CauterRule's extraction layer improved dramatically after the parser fixes. The model comparison produced clean, parseable candidates across 1,538 rows. But the replay engine — the thing that decides whether those candidates are good — is still a heuristic string matcher. That means the benchmark's quality ceiling is set by the judge, not by the models. Investing in better models while the judge is weak is optimizing the wrong layer.
Raw breadth without evaluation depth is not evidence. The raw corpora produced 1,175 candidates across all models. 745 of those were inconclusive — 63%. The product can process raw corpora, but it cannot evaluate them. The report says this directly: "broad coverage that produces mostly inconclusives is not yet the strongest evidence of product trustworthiness." If you are benchmarking a learning system, do not confuse processing coverage with evaluation coverage.
The inconclusive rate is a product health metric, not just a benchmark metric. If your inconclusive rate is 53%, your product has a measurement problem that blocks every downstream decision — promotion, safety, trust. Tracking the inconclusive rate over time tells you whether the product is getting better at judgment, not just at extraction. It should be a first-class metric in any learning system's dashboard.
Open questions
- Would a semantic matcher (even a lightweight embedding-based one) materially reduce the inconclusive rate on raw corpora, or is the problem deeper than matching strategy? The current data cannot answer this because we only tested one matcher.
- The curated inconclusive rate for Llama 8B was 14.9%. Is that the realistic floor for the current matcher, or can calibration push it lower? If 15% is the floor, that means even on clean data, 1 in 7 rules cannot be evaluated.
- GPT-4o-mini had the highest inconclusive rate (62.9%) despite perfect parse stability. Is that because the model produces more nuanced triggers that the string matcher cannot handle, or because it produces vaguer triggers? The data does not distinguish.
- The
raw/sibling-reposcorpus produced 100% inconclusives for the strongest cloud model. Is that a matcher problem, a corpus problem, or a harness-transfer problem? The report says "harness-transfer quality is still not strong," which suggests the corpus itself may be the issue, not the engine. - If we split the inconclusive bucket by root cause (model vagueness vs engine weakness), what proportion would fall into each? That split would completely change the product roadmap. If 70% of inconclusives are engine weakness, the fix is replay calibration. If 70% are model vagueness, the fix is prompt engineering and trigger specificity.
The broader lesson
If you are building any system that evaluates AI output — a replay engine, a guardrail checker, a quality scorer, a promotion gate — track the inconclusive rate as seriously as you track pass and fail. The inconclusive bucket is where measurement gaps live. And measurement gaps are where product risks hide.
A benchmark that returns 53% "can't tell" is not measuring quality. It is measuring uncertainty. And uncertainty is the most expensive thing in a learning system, because it blocks every downstream decision without telling you why.
The next milestone for CauterRule is not more extraction breadth or better models. It is a judge that can decide.
CauterRule v0.1.0 is released. The full model-by-model breakdown — including pass/inconclusive/fail distributions for all 4 models across curated and raw corpora — is in the field test report. The repo is public. Install with
pip install cauterule. Changelog · Release notes
Top comments (1)
The inconclusive bucket is a useful result, not noise. I’d log judge disagreement by prompt family and keep a small curated set around, because a stable abstention signal is easier to operate than pretending every edge case has a winner.