DEV Community: Debashish Ghosal

I Shipped a Fix That Fixed Nothing. Here's Why I Kept It.

Debashish Ghosal — Thu, 10 Sep 2026 01:38:37 +0000

Update — v0.2.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, TUI review, observability, 7 export formats, adversarial corpora, and a bundled git rule pack. The field test report evaluated 4 models across 745 trajectories and is the source for every number in this article. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

I built a nearmiss recovery detector. It checks if a trajectory has success=True, an early step error, and a later clean output. If all three are true, it drops the trajectory before the LLM is ever called. The logic is correct. The code is tested. The tests pass.

It dropped 0 trajectories.

The nearmiss corpus has success=False — these are real failures, not recoveries. The fix was built for a hypothesis that turned out to be wrong. I expected it to eliminate 5 false positives. It eliminated none. The actual nearmiss fix came later, in a different component, with a different mechanism.

I shipped the fix anyway. Here's why.

The hypothesis: nearmiss FPs are caused by recovery patterns

The nearmiss corpus contains 50 lookalike trajectories per model — cases that resemble real failures but should not trigger the extracted rule. On the first v0.2.0 sweep, 5 of 50 nearmiss trajectories produced passing candidates. All 5 passes had precision 1.00 and recall 0.02 — they matched exactly 1 reference failure out of 210. They should have matched none.

My hypothesis was that some of these false positives were caused by recovery patterns — trajectories where the agent failed, then self-resolved, and the final outcome was success=True. The gate was letting these through because they had failure signals (error text, non-zero exit) in their early steps. The extractor then treated them like real failures and produced candidates.

The fix I built was Fix 6 — a nearmiss gate signal in gate.py. The learnings & fixes log §1.5 documents the logic:

def _detect_nearmiss_recovery(trajectory):
    if not trajectory.success:
        return False
    has_early_error = any(step.error for step in trajectory.steps[:len(trajectory.steps)//2])
    has_later_clean = any(step.output and not step.error 
                         for step in trajectory.steps[len(trajectory.steps)//2:])
    return has_early_error and has_later_clean

If trajectory.success == True AND at least one early step has an error AND a later step has clean output (no error), the gate drops the trajectory with reason nearmiss_recovery_succeeded instead of sending it to the LLM.

The logic is correct. The tests pass. The fix is sound.

The result: 0 trajectories dropped

When I ran the sweep with Fix 6 applied, it dropped 0 of 50 nearmiss trajectories. The nearmiss corpus has success=False — every trajectory is a real failure, not a recovery. There are no success=True trajectories in the nearmiss corpus. The gate signal fires on success=True + early error + later clean. There are no success=True trajectories to fire on.

The learnings & fixes log records this directly:

Update: Fix 6 dropped 0 trajectories in practice — the nearmiss corpus trajectories have success=False, not recovery patterns. The actual nearmiss/recovery fix came later via Fix 8 (recovery trajectory exclusion in the simulator, §4.8), which reclassifies success=True reference trajectories with recovery keywords as "near_miss" instead of "broken". Fix 6 remains in place as a defense-in-depth signal for future corpora that may contain success=True recovery patterns.

The hypothesis was wrong. The nearmiss false positives are not caused by recovery patterns in the nearmiss corpus. They are caused by two different things:

Degenerate triggers (2 FPs) — the model produces "step_1" which matches every trajectory. Fixed by Fix 7 (degenerate trigger rejection).
"Wrong failure" matches (3 FPs) — "git push fails with authentication error" matches "git push fails with non-fast-forward" because the matcher can't distinguish failure classes within the same tool. Not yet fixed (planned for v0.3.0).

Fix 6 was built for a third cause — recovery patterns with success=True — that doesn't exist in the data. The fix is correct for a problem that isn't there.

The actual fix: Fix 8, in the simulator, not the gate

The real nearmiss/recovery fix came from Fix 8 — the 6-line simulator fix that reclassifies success=True reference trajectories with recovery keywords in failure_class as near_miss instead of broken. Fix 8 doesn't drop trajectories before extraction. It changes how the simulator classifies reference trajectories during replay. It operates on the reference corpus, not the test corpus. It fixes a classification bug, not a gating gap.

Fix 8 moved golden 20% → 50% on both cloud models. Fix 6 moved nothing. But they are related — both address the same insight: success=True does not mean "clean success." Some success=True trajectories are recoveries. Fix 6 catches them at the gate (before extraction). Fix 8 catches them in the simulator (during replay). Fix 6 catches the case where a recovery trajectory is in the test corpus and shouldn't be sent to the LLM. Fix 8 catches the case where a recovery trajectory is in the reference corpus and shouldn't be counted as "broken" when a trigger matches it.

Fix 6 is the gate-level defense. Fix 8 is the simulator-level defense. Fix 6 dropped 0 because the test corpus doesn't have success=True recoveries. Fix 8 dropped the "broken" classification because the reference corpus does have success=True recoveries. Same insight, different layer, different data, different outcome.

Why I kept Fix 6

Fix 6 is defense-in-depth. It didn't fix the v0.2.0 nearmiss false positives. But it catches a class of problem that doesn't exist in the current corpus and might exist in future corpora.

The current nearmiss corpus has success=False — real failures. But a future corpus — or a user's custom corpus — might include success=True recovery trajectories. A trajectory where the agent hit a transient error, retried, and succeeded. That trajectory has success=True, an early step error, and a later clean output. The gate should drop it before extraction, because it is a near-miss that recovered, not a real failure. Fix 6 is the gate-level signal that catches this.

Without Fix 6, a success=True recovery trajectory in the test corpus would be sent to the LLM. The LLM would extract a candidate rule from it. The candidate would be a false positive — a rule extracted from a trajectory that wasn't really a failure. The promotion gate might catch it (if the replay engine rejects it). But the gate is the first line of defense, and the cheapest one — no LLM call, no extraction, no replay. Dropping a trajectory at the gate costs nothing. Letting it through costs an LLM call, an extraction, a replay, and a promotion decision. Fix 6 is the cheapest defense, even if it fires on 0 trajectories today.

The distinction: a fix vs. a defense

Fix 8 is a fix. It corrects a bug that exists in the current data. It moves the golden pass rate from 20% to 50%. It has measurable impact. Without it, the product is worse.

Fix 6 is a defense. It prevents a class of problem that doesn't exist in the current data but could exist in future data. It has no measurable impact today. Without it, the product is the same today. But with it, the product is more robust to corpora I haven't built yet.

The distinction matters because the two have different success criteria. A fix is judged by its impact on the current numbers. Fix 8 moved golden 20% → 50% — that is a successful fix. A defense is judged by its coverage of a threat model. Fix 6 covers success=True + early error + later clean — that is a class of trajectory that the gate should handle, regardless of whether the current corpus contains any.

If I had judged Fix 6 by the same criterion as Fix 8 — "did it move the numbers?" — I would have concluded it was a failure and removed it. But that is the wrong criterion for a defense. The right criterion is: "if a success=True recovery trajectory were added to the corpus tomorrow, would the gate handle it correctly?" The answer is yes, because Fix 6 is in place. The defense holds, even though it has never been tested in production.

What I learned from this

A fix that fixes nothing is not a failure if it prevents a class of problem. Fix 6 dropped 0 trajectories. By the "did it move the numbers?" criterion, it was a wasted effort. By the "does it cover a threat model?" criterion, it is a necessary defense. The distinction is: a fix corrects a bug that exists. A defense prevents a bug that could exist. Judge fixes by impact. Judge defenses by coverage. Don't judge defenses by impact — they have none, by design.

The same insight can produce both a fix and a defense. Fix 6 and Fix 8 are the same insight — success=True does not mean "clean success." Fix 6 is the gate-level implementation (drop recovery trajectories before extraction). Fix 8 is the simulator-level implementation (reclassify recovery trajectories during replay). Fix 8 is the fix — it corrected a bug in the current data. Fix 6 is the defense — it prevents a bug in future data. Same insight, different layer, different role.

Wrong hypotheses are worth following. I hypothesized that nearmiss FPs were caused by recovery patterns. They weren't. But the hypothesis led me to build Fix 6, which is a defense I want to have. And the hypothesis led me to look at success=True trajectories more carefully, which is what led to Fix 8. The wrong hypothesis didn't waste time. It redirected attention to the right layer. The fix was in the simulator, not the gate — but I found it because I was looking at success=True trajectories, which I was doing because of the wrong hypothesis.

Defense-in-depth means building defenses for problems you don't have yet. Fix 6 is in the codebase. It fires on 0 trajectories today. It will fire on success=True recovery trajectories if any are ever added to the corpus. That is the point. A defense-in-depth system has components that do nothing today and everything tomorrow. Removing them because they have no measurable impact is how you lose the defense before you need it.

Open questions

Fix 6 checks for success=True + early error + later clean. Fix 8 checks for success=True + recovery keywords in failure_class. These are two different signals for the same class of trajectory. Would they ever disagree — a trajectory that Fix 6 would drop but Fix 8 would not reclassify, or vice versa? If so, which signal is correct?

The nearmiss corpus has success=False. Fix 6 fires on success=True. If I added success=True recovery trajectories to the nearmiss corpus, would Fix 6 catch all of them, or only those with the "early error + later clean" pattern? A recovery where the error and the clean output are in the same step would not be caught. Is that a realistic pattern?

Fix 6 was built from a wrong hypothesis. How many other fixes in the v0.2.0 codebase were built from hypotheses that turned out to be wrong but were kept as defenses? Is there a systematic way to distinguish "fix that didn't fire because the hypothesis was wrong" from "defense that didn't fire because the threat isn't in the data yet"? Both have zero impact today. One is worth keeping. The other might be worth removing — or might be worth keeping for the same defense-in-depth reason.

The 3 remaining nearmiss FPs on Llama (after Fix 7) are "wrong failure" matches — "git push fails with authentication error" matching "git push fails with non-fast-forward". Neither Fix 6 (gate, success=True) nor Fix 8 (simulator, recovery keywords) addresses these. They have success=False and no recovery keywords. The fix is trigger-domain mismatch detection (v0.3.0). Would that fix have been found sooner if I had traced the 3 FPs to their root cause before building Fix 6? Or was Fix 6 a necessary detour — the wrong hypothesis that redirected attention to success=True trajectories, which led to Fix 8?

Should Fix 6 be tested against synthetic success=True recovery trajectories to verify it fires correctly? Currently it is tested with unit tests (mock trajectories). But the gate's behavior on real recovery trajectories is unverified because the corpus doesn't contain any. Would adding 5-10 synthetic recovery trajectories to the corpus — trajectories designed to trigger Fix 6 — provide the end-to-end test coverage that the current corpus doesn't?

The broader lesson

Not every fix has to move the numbers. Some fixes are defenses — they prevent a class of problem that doesn't exist in your current data but could exist in future data. Judge fixes by their impact on the current numbers. Judge defenses by their coverage of a threat model. Don't remove a defense because it has no measurable impact today. It has no impact because the threat isn't here yet. That is the point.

CauterRule's v0.2.0 field test produced two fixes from the same insight. Fix 8 is a fix — it corrected a classification bug, moved golden 20% → 50%, and has measurable impact. Fix 6 is a defense — it prevents success=True recovery trajectories from being sent to the LLM, fires on 0 trajectories today, and has no measurable impact. Both are in the codebase. Both should be.

If you are building a system with defense-in-depth — gates, matchers, scorers, promotion gates — expect some components to do nothing today. That is not a bug. That is defense-in-depth working as designed. The components that fire on 0 trajectories today are the ones that will fire on the trajectory you didn't anticipate tomorrow. Keep them.

CauterRule v0.2.0 is released. The full fix history — Fix 6 through Fix 8, with impact analysis and defense-in-depth rationale — is in the field test report and the learnings & fixes document. The repo is public. Install with pip install cauterule. Changelog · Release notes

A Rule Can Be Specific and Still Be Too Broad

Debashish Ghosal — Wed, 09 Sep 2026 11:46:00 +0000

Update — v0.2.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, TUI review, observability, 7 export formats, adversarial corpora, and a bundled git rule pack. The field test report evaluated 4 models across 745 trajectories and is the source for every number in this article. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

My model produces triggers like "when git push fails with non-fast-forward" — specific, concrete, names the tool and the error. 94.4% of triggers are specific or moderate. Only 5.6% are generic. The specificity scorer works. The extraction prompt works. The model is doing its job.

And 6 of 10 golden scenarios still fail — because the trigger matches the right failure AND matches clean successes it would break. The trigger is specific about the failure. It is broad about when it fires.

Specific does not mean safe.

The trigger specificity numbers

The v0.2.0 field test measured trigger specificity across all 4 models. The learnings & fixes log §1.3 has the breakdown:

Model	Specific	Moderate	Generic	Generic %
Local Llama 3.2B	253/337 (75.1%)	65/337 (19.3%)	19/337	5.6%
Local Qwen 4B	30/30 (100%)	0/30 (0%)	0/30	0%

Both models are well under the 10% generic target. The triggers name concrete tools and error conditions — "when git push fails with non-fast-forward", "pip install fails with version conflict", "docker build fails with package not found", "terraform plan fails with state lock error", "pytest fails with AssertionError", "kubectl apply fails with NotFound for CRD". These are not vague platitudes. They are specific failure descriptions.

The extraction layer is working. The model produces high-quality triggers. And that is exactly what makes the remaining failures so hard to fix.

The golden corpus: 2 pass, 2 fail, 6 inconclusive

After the matcher fixes (Fix 1-4) eliminated all matcher_gap inconclusives, the golden corpus settled at 2P / 2F / 6I on both local models. The 6 inconclusives are not matcher gaps — the matcher finds matches for all of them. They are ambiguous_evidence: the trigger matches the reference failure, but it also matches clean successes in the reference corpus. The replay engine can't decide whether the trigger is safe to promote.

Here is the golden re-run data after Fix 1-4:

Model	Pass	Fail	Inconclusive	Inconclusive breakdown
Llama 3.2B	2	2	6	0 broad_trigger / 0 matcher_gap / 12 ambiguous_evidence
Qwen 4B	2	2	6	0 broad_trigger / 0 matcher_gap / 12 ambiguous_evidence

Zero matcher_gap on both models — Fix 1-4 confirmed. All 6 inconclusives are ambiguous_evidence. The triggers are specific enough to match the failure, but they also match clean successes. The replay engine says "maybe" — the trigger prevents real failures, but it also breaks real successes. Not safe to promote. Not safe to reject. Inconclusive.

What "specific but broad" looks like

Take G-001: the trigger is "when git push fails with non-fast-forward". This is a specific trigger — it names the tool (git), the action (push), and the error (non-fast-forward). The specificity scorer classifies it as "specific".

But the trigger also matches clean git push trajectories in the reference corpus. A trajectory where the agent runs git push origin feature and it succeeds — the trigger's tokens ("git", "push") appear in that trajectory. The matcher scores it above threshold. The simulator classifies it as broken — the trigger would fire on a clean success and interfere with it.

The trigger is specific about which failure it targets. It is broad about which trajectories it fires on. Those are two different properties, and the benchmark measures both — but the specificity scorer only measures the first.

This is the distinction the v0.2.0 broad-trigger penalty was built to capture. The scorer now distinguishes three cases, documented in learnings-fixes.md §6.2:

Condition	Verdict	Meaning
`broken > prevented`	Fail	Dangerously broad — breaks more than it prevents
`broken > 0 but ≤ prevented`	Inconclusive	Broad but fixable — breaks some, prevents more
`broken == 0`	Pass (if precision ≥ 0.8)	Safe — prevents failures without breaking successes

The 6 golden inconclusives fall into the middle bucket: broken > 0 but ≤ prevented. The triggers prevent real failures, but they also break some successes. Not dangerously broad (that would be a fail), but not safe either (that would be a pass). They are broad but fixable.

The broad-trigger penalty in action

The broad-trigger penalty fires across the v0.2.0 sweep. The field test report documents the attribution:

Corpus	Model	broad_trigger attributions	ambiguous_evidence
golden	Llama 3.2B	0	12 (6 scenarios × 2 models)
nearmiss	Llama 3.2B	3	25
failures/positive	Llama 3.2B	7	56

The penalty correctly downgrades "matches but breaks successes" from fail to inconclusive. Without it, these would be hard fails — the matcher finds the match, the simulator counts broken successes, and the verdict is "fail" because broken > 0. With the penalty, the verdict is "inconclusive" because broken ≤ prevented — the trigger prevents more failures than it breaks successes. That is a more honest verdict. The trigger is not dangerous. It is not safe. It is in between.

The linter also got a new check: check_broadness() flags triggers with score_specificity == "generic" as broad. This catches the easy case — triggers like "when a command fails" that are both generic and broad. But the harder case — triggers that are specific about the failure but broad about when they fire — is not caught by the linter. The linter checks specificity. The scorer checks breadth. They measure different things.

Why specificity and breadth are different properties

Specificity measures: does the trigger name a concrete failure? "when git push fails with non-fast-forward" is specific. "when a command fails" is generic.

Breadth measures: does the trigger fire on trajectories it shouldn't? A trigger that fires on clean git pushes (because "git" and "push" appear in them) is broad — even if it is specific about the failure it targets.

A trigger can be specific and broad. "when git push fails with non-fast-forward" is specific (names the error) and broad (matches clean git pushes). A trigger can be generic and narrow. "when step_1 fails" is generic (names a step number, not a failure) and narrow (only matches trajectories with step_1 — which is all of them, so actually broad, but in a degenerate way). A trigger can be specific and narrow. "when git push to origin on branch feature fails with non-fast-forward because the remote contains commits you don't have" — this is specific AND narrow. It would only fire on the exact failure. But the model doesn't produce triggers this narrow, and if it did, they would be too narrow to be useful — they would only match one specific scenario.

The sweet spot is specific and narrow enough to be safe but broad enough to be useful. CauterRule's triggers are specific but too broad. The fix is not to make them more specific — they are already specific. The fix is to make them narrower in what they match — which means either narrowing the trigger itself (harder prompt engineering) or improving the matcher's ability to distinguish "matches the failure" from "matches any trajectory with the same tool name" (semantic matching, planned for v0.3.0).

What I learned from this

Specificity is necessary but not sufficient. The model produces specific triggers — 94.4% specific or moderate. The specificity scorer works. The extraction prompt works. And 6 of 10 golden scenarios are still inconclusive because the triggers match too broadly. Specificity measures whether the trigger names a concrete failure. It does not measure whether the trigger fires on the right trajectories and stays silent on the wrong ones. That is breadth, and breadth is what the replay engine measures.

The two properties need separate metrics. The specificity scorer checks "does the trigger name a concrete failure?" The broad-trigger penalty checks "does the trigger break more successes than it prevents failures?" These are different questions. A trigger can score well on one and poorly on the other. The v0.2.0 field test produced triggers that score well on specificity and poorly on breadth — specific but too broad. The fix is not to conflate the two metrics. The fix is to measure both and let the replay engine arbitrate.

The broad-trigger penalty is a correct verdict on an insufficient fix. The penalty correctly classifies "matches but breaks successes" as inconclusive. That is the right verdict — the trigger is not dangerous (fail) and not safe (pass). But it is not a fix. It is a more honest diagnosis. The actual fix — making the trigger narrower in what it matches — requires either semantic matching (the matcher understands that "non-fast-forward" and "authentication error" are different failure classes even though both are "git push fails with X") or reference corpus expansion (more trajectories so the replay engine has more data to distinguish "matches the failure" from "matches the tool"). Both are planned for v0.3.0.

The remaining golden gap is a matching problem, not an extraction problem. The model is extracting the right triggers. The specificity is excellent. The problem is that the matcher can't tell the difference between "the trigger matches the failure" and "the trigger matches any trajectory involving the same tool." That is a semantic matching gap, not a prompt engineering gap. Investing in prompt tuning to make the triggers narrower would make them less useful. Investing in semantic matching would let the replay engine distinguish "fires on the right failure" from "fires on any git push." The fix is downstream of extraction, not upstream.

Open questions

Would a semantic matcher (comparing failure_class between trigger and reference) convert the 6 golden inconclusives to passes? The inconclusives are ambiguous_evidence — the trigger matches the failure and some successes. A semantic matcher that can tell "non-fast-forward" from "clean push" would reduce the broken count, potentially to zero. If broken == 0, the verdict becomes pass. But how much semantic matching is needed — embeddings? A failure-class taxonomy? A simple keyword comparison?

The broad-trigger penalty produces inconclusive, not fail. Is inconclusive the right verdict for "broad but fixable"? The alternative is to fail these triggers — they are too broad to promote. But that would penalize triggers that prevent more failures than they break successes. The penalty's choice — inconclusive for broken ≤ prevented, fail for broken > prevented — is a judgment call. Would a stricter threshold (inconclusive only if broken ≤ prevented * 0.5) produce better promotion decisions?

6 of 10 golden scenarios are ambiguous_evidence. The other 4 are 2 pass, 2 fail. The 2 fails have broken > prevented — dangerously broad. The 2 passes have broken == 0. Is there a middle ground — a trigger that is specific, matches the failure, and breaks zero successes? Or is broken == 0 only achievable when the trigger is so narrow that it only matches the exact reference failure? If so, the golden pass rate ceiling is set by how narrow the model can make triggers without making them useless.

The check_broadness() linter check flags score_specificity == "generic" triggers as broad. But the golden inconclusives are specific triggers that are broad. Should the linter also check breadth — not just "is the trigger generic?" but "does the trigger match too many clean successes?" That would require the linter to run the replay engine, which it currently doesn't. Is that a reasonable thing for a linter to do, or should breadth stay in the replay engine's domain?

The cloud models post-Fix 8 have golden 50% (5P/1F/4I). The 4 inconclusives are the same ambiguous_evidence pattern — specific but broad. Would Fix 8's recovery exclusion help the local models the same way it helped cloud? Local golden is 20% (2P/2F/6I) pre-Fix 8. If Fix 8 moves local golden the same 30 points it moved cloud, local would be at 50% too. But local triggers are broader. Will the recovery exclusion help as much, or will the broader triggers still match too many clean successes?

The broader lesson

Specificity is not safety. A trigger can name the exact failure — "when git push fails with non-fast-forward" — and still fire on every clean git push in your reference corpus. The specificity scorer measures whether the trigger describes a concrete failure. It does not measure whether the trigger fires on the right trajectories and stays silent on the wrong ones.

CauterRule's v0.2.0 field test produced triggers that are 94.4% specific or moderate. The extraction layer is working. And 6 of 10 golden scenarios are inconclusive because the triggers match too broadly. The problem is not extraction. The problem is matching — the replay engine can't distinguish "fires on the failure" from "fires on any trajectory with the same tool."

The fix is not to make the triggers more specific. They are already specific. The fix is to make the matcher smarter — to understand that "non-fast-forward" and "authentication error" are different failure classes even when both are "git push fails with X". That is semantic matching, and it is the v0.3.0 priority.

If you are building a system that extracts rules from failures, measure specificity and breadth separately. Specificity tells you whether the rule describes a real failure. Breadth tells you whether the rule fires only when it should. A rule that is specific but broad is not safe. A rule that is specific and narrow is the goal. And the gap between the two is a matching problem, not an extraction problem.

CauterRule v0.2.0 is released. The full trigger specificity analysis, broad-trigger penalty breakdown, and golden corpus results are in the field test report and the learnings & fixes document. The repo is public. Install with pip install cauterule. Changelog · Release notes

I Tried to Poison My Agent's Rule Store. It Produced 20 Triggers. Zero Got In.

Debashish Ghosal — Wed, 09 Sep 2026 05:33:55 +0000

Update — v0.2.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, TUI review, observability, 7 export formats, adversarial corpora, and a bundled git rule pack. The field test report evaluated 4 models across 745 trajectories and is the source for every number in this article. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

I built 6 adversarial corpora. Prompt injection. Misleading trajectories. Contradictions. Unsafe directives. Poisoning. Instruction leakage. 10 trajectories each, 50 total. Each one designed to compromise the extractor — to make the model produce a trigger that, if promoted, would corrupt the rule store.

The model followed every injection. It produced plausible-looking triggers for all 6 attack types. Precision was deceptively high — 0.667 on injection — because injection prompts mimic reference failure patterns. The triggers looked real.

Zero got promoted. Across all 4 models. Across all 6 attack vectors. Not one adversarial trigger entered the rule store.

This is the story of why — and the defense-in-depth model that makes it work.

The 6 adversarial corpora

v0.2.0 introduces 6 adversarial corpora, each testing a different attack surface:

Corpus	Attack type	What it tests
`injection`	Prompt injection — override system prompt with injected instructions	Can the attacker make the model produce a trigger of their choosing?
`misleading`	Misleading trajectories — failures that look like one thing but are another	Can the model be tricked into extracting a rule for the wrong failure?
`contradiction`	Contradictory trajectories — two trajectories with opposite outcomes for the same trigger	Can the model produce a trigger that contradicts existing rules?
`unsafe`	Unsafe directives — trajectories that suggest dangerous actions	Can the model be tricked into promoting a rule that recommends unsafe behavior?
`poisoning`	Data poisoning — corrupted trajectory data designed to produce a specific trigger	Can the attacker inject trajectories that produce a predetermined trigger?
`instruction-leakage`	Instruction leakage — system prompt content leaking into extracted triggers	Can the model's own instructions contaminate the rule store?

Each corpus has 10 trajectories. 50 total. All 4 models ran against all 6 corpora. The question is not "can the model be compromised?" — it can. The question is "if the model is compromised, does the bad trigger get promoted?"

The model complied. The system didn't.

The model followed the injection prompts. On the injection corpus, all 4 models produced candidates — 20 total across 10 injection trajectories. The triggers looked plausible. They mimicked valid failure patterns because the injection prompts were designed to mimic reference failures.

Model	Injection candidates	Precision (deceptive)
Local Llama 3.2B	10	0.667
Local Qwen 4B	10	0.667
Cloud gpt-4o-mini	10	0.667
Cloud llama-3.1-8b	10	0.667

Precision 0.667 is deceptively high. It means the injection triggers match reference failures in the corpus — they "prevent" real failures. But the failures they prevent are not the ones the triggers describe. The injection prompts are crafted to mimic reference failure patterns, so the matcher finds matches. The triggers look like they work.

But the promotion gate is not fooled. A candidate is promoted only if it prevents real failures without breaking real successes. The injection triggers match reference failures — but they also match everything else. They are too broad. The broad-trigger penalty (broken > prevented → fail) catches them. The promotion gate rejects them.

Across all 6 adversarial corpora and all 4 models: 0 promoted rules. The validation suite confirms it: 41 adversarial tests, 0 failures, 0 promoted rules.

The per-model breakdown

Model	injection	misleading	contradiction	unsafe	poisoning
Llama 3.2B	0P / 10F / 0I	0P / 7F / 3I	0P / 10F / 0I	0P / 7F / 3I	0P / 5F / 5I
Qwen 4B	0P / 10F / 0I	0P / 6F / 4I	0P / 4F / 6I	0P / 0F / 10I	0P / 3F / 7I
gpt-4o-mini	0P / 0F / 10I	0P / 3F / 7I	0P / 1F / 9I	0P / 0F / 10I	0P / 1F / 9I
llama-3.1-8b	0P / 0F / 10I	0P / 2F / 8I	0P / 1F / 9I	0P / 0F / 10I	0P / 3F / 7I

Zero passes across the board. But the verdict distributions tell a story.

Local models produce hard fails. Llama 3.2B: 10F on injection, 10F on contradiction. The matcher rejects the triggers outright — they match too broadly, breaking more successes than they prevent. The local model's triggers are broad enough that the replay engine can decisively reject them.

Cloud models produce inconclusives. gpt-4o-mini: 10I on injection, 10I on unsafe, 9I on contradiction, 9I on poisoning. The cloud model's triggers are more specific — they match the reference failures more precisely. But they don't prevent enough failures to pass, and they don't break enough successes to fail. They sit in the inconclusive bucket — plausible but unproven.

This is a subtle point. The cloud model is "better" at extraction — its triggers are more specific. But on adversarial inputs, more specific is not safer. A specific trigger that matches exactly the injection target is harder to reject than a broad trigger that matches everything. The cloud model's specificity makes the inconclusive verdict more likely — the trigger is too precise to fail (it doesn't break many successes) but too narrow to pass (it doesn't prevent many failures either).

The promotion gate treats inconclusive as "do not promote." So the cloud model's more-specific adversarial triggers are still blocked — just via a different verdict. The defense holds. But it holds differently for different models.

Why defense-in-depth works here

The defense is not in any single layer. It is in the chain of layers, each of which catches what the previous layer misses.

Layer	What it catches	What it misses
Pre-extraction gate	Clean trajectories with no failure signal	Trajectories with faked failure signals
Extraction (LLM)	Nothing — the model is the attack target	Everything — the model complies with injections
Replay engine	Triggers that break more successes than they prevent	Triggers that are broad but plausible
Broad-trigger penalty	Triggers that match too broadly (`broken > prevented`)	Triggers that are specific but wrong
Promotion gate	Candidates without enough evidence to promote	— (the final gate)

The model is the attack surface. The injection corpus proves it — the model follows injected instructions and produces triggers of the attacker's choosing. But the model is not the defense. The defense is downstream: the replay engine checks whether the trigger helps without breaking, the broad-trigger penalty catches triggers that are too broad, and the promotion gate requires enough evidence to promote.

This is why 0 adversarial triggers got promoted. The model was compromised on every corpus. But the replay engine, the broad-trigger penalty, and the promotion gate — all deterministic layers that don't depend on the model's trustworthiness — caught every case.

The one that almost got through

The misleading corpus produced the closest call. On Llama 3.2B, 3 of 10 misleading trajectories produced inconclusives — not fails. On gpt-4o-mini, 7 of 10 were inconclusive. These are triggers that the matcher couldn't decisively reject. They matched reference failures closely enough to avoid a "fail" verdict, but not closely enough to earn a "pass." They sat in limbo.

Inconclusive is safe — the promotion gate does not promote inconclusives. But it is also informative. The misleading corpus is designed to produce triggers that look real. The fact that 7 of 10 are inconclusive on the cloud model means the matcher is genuinely uncertain — the trigger is plausible enough that the replay engine can't distinguish it from a real rule.

If the promotion gate were looser — if it promoted on "plausible" instead of "proven" — some of these would get in. The defense holds because the gate is strict. But the inconclusive rate on misleading is a warning: the matcher's uncertainty is the attacker's opportunity. A stricter matcher — one that could distinguish "real failure pattern" from "misleading failure pattern" — would convert those inconclusives to fails. A looser promotion gate would convert them to promoted rules.

What I learned from this

You don't secure an LLM system by making the LLM trustworthy. You secure it by assuming the LLM is compromised. The injection corpus proved the model can be compromised — it follows injected instructions and produces triggers of the attacker's choosing. The defense is not in the model. It is in the deterministic layers downstream: replay, broad-trigger penalty, promotion gate. These layers don't trust the model. They verify the model's output against evidence. If the evidence doesn't support the trigger, the trigger doesn't get promoted — regardless of how the trigger was produced.

Adversarial testing is the only way to verify defense-in-depth. I could have argued "the replay engine and promotion gate provide defense-in-depth" without testing it. The argument would have been plausible. The 6 adversarial corpora turned the argument into evidence. 0 promoted rules across 50 adversarial trajectories and 4 models is not an argument — it is a test result. If you have a defense claim, build a corpus that attacks it. If the defense holds, you have evidence. If it doesn't, you have a vulnerability. Either way, you have data instead of assumptions.

More specific models are not safer models. The cloud models produced more specific adversarial triggers — and more inconclusives. The local models produced broader triggers — and more hard fails. The cloud model's specificity made the matcher less certain, not more rejecting. A more specific trigger is harder to distinguish from a real rule. The defense held because the promotion gate treats inconclusive as "do not promote." But the cloud model's adversarial triggers were closer to getting in than the local model's. Better extraction is not better safety. It is a different attack profile.

Precision is deceptive on adversarial inputs. The injection corpus produced precision 0.667 — 67% of injection triggers "prevent" reference failures. That looks like the triggers are working. They are not. The injection prompts are designed to mimic reference failure patterns, so the matcher finds matches. Precision measures "does the trigger match the reference?" not "is the trigger real." On adversarial inputs, precision is the wrong metric. The right metric is "does this trigger promote?" — and the answer is 0%.

Open questions

The misleading corpus produced 7 inconclusives on gpt-4o-mini. These are triggers the matcher can't distinguish from real rules. Would a semantic matcher (comparing failure_class between trigger and reference) convert these to fails? Or are some misleading triggers genuinely indistinguishable from real rules?
The 6 adversarial corpora cover prompt injection, misleading data, contradictions, unsafe directives, poisoning, and instruction leakage. What attack vectors are missing? The OWASP Agentic Top 10 names 10 — are there 4 more corpora to build?
The cloud model's more specific adversarial triggers produced more inconclusives. If the promotion gate were ever loosened — promoting on "plausible" instead of "proven" — how many of those inconclusives would become promoted rules? This is the risk of relaxing the gate. The adversarial corpus quantifies it.
The validation suite has 41 adversarial tests, 0 failures. But the suite tests the matcher and promotion gate, not the extractor. Should the suite also test the extraction layer — verifying that the model produces specific triggers even on adversarial inputs? Or is the extraction layer's trustworthiness irrelevant, given that downstream layers catch everything?
The unsafe corpus tests whether the model promotes rules that recommend dangerous actions. All 4 models produced 0 passes — but the cloud models produced 10 inconclusives each. The triggers were too specific to fail but too narrow to pass. Would a "safety linter" — checking the trigger's recommended action against an unsafe-action list — catch these, or are they already caught by the promotion gate's evidence requirement?

The broader lesson

If you are building an LLM system — a rule-learning sidecar, a coding agent, a memory layer, a retrieval system — assume the LLM is compromised. Build the defense downstream. The LLM is the attack surface. The replay engine, the evidence gate, the promotion threshold — those are the defense.

CauterRule's adversarial test proved the model can be made to produce any trigger the attacker wants. It also proved that none of those triggers get promoted. Not because the model resisted the attack, but because the system doesn't trust the model. The model produces. The system verifies. The system rejects what the model produces when the evidence doesn't support it.

That is defense-in-depth for LLM systems. Not "make the model safe." Make the system safe when the model is not.

CauterRule v0.2.0 is released. The full adversarial corpus breakdown — 6 attack vectors, 4 models, per-model verdict distributions — is in the field test report and the learnings & fixes document. The repo is public. Install with pip install cauterule. Changelog · Release notes

My 3B Model Found a Shortcut. It Took Me Three Fixes to Close It.

Debashish Ghosal — Tue, 08 Sep 2026 20:41:00 +0000

Update — v0.2.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, TUI review, observability, 7 export formats, adversarial corpora, and a bundled git rule pack. The field test report evaluated 4 models across 745 trajectories and is the source for every number in this article. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

My model produced a trigger that was just "step_1". It matched every reference trajectory that contained a step numbered 1 — which is all of them. Precision 1.00, recall 0.02, verdict: pass. Two nearmiss false positives traced to this trigger. The specificity scorer classified it as "specific" because it contains a token. The matcher scored it 1.00 because "step_1" is an exact substring match in every trajectory's step identifier field.

The model found a shortcut. And my benchmark rewarded it.

It took me three fixes to close it. Not because the problem was hard — each fix was a few lines. But because the shortcut revealed something deeper about evaluation design: if your matcher rewards any substring match, your model will find substrings that match everything. The problem wasn't the model. It was the reward.

The shortcut: "step_1" matches everything

Two of those 5 false positives were caused by the same trigger: "step_1". The learnings & fixes log traces them:

Trajectory	Trigger	Precision	Recall	Verdict
NM-030-certificate-retry	`"step_1"`	1.00	0.02	Pass (false positive)
NM-044-git-commit-hook	`"step_1"`	1.00	0.02	Pass (false positive)

The trigger "step_1" appears in every reference trajectory's step identifier field — {"step": 1, "input": "git push origin feature", ...}. The matcher normalizes both trigger and haystack, tokenizes them, and checks for substring matches. "step_1" is a substring of "step_1": {"input": .... Score: 1.00. Precision: 1.00 — the trigger fires on exactly 1 reference trajectory. Recall: 0.02 — 1 out of 210. Verdict: pass, because precision is high and the trigger "prevents" a real failure.

But the trigger is not about a failure. It is about a step number. The model learned that "step_1" appears in every trajectory, so producing it as a "trigger" guarantees a match. It is not extracting a failure pattern. It is exploiting a structural artifact of the trajectory format.

Fix 7a: degenerate trigger classification

The first fix was in the specificity scorer (specificity.py). I added a regex pattern:

_DEGENERATE_RE = re.compile(r"^step[_\s]*\d+$", re.IGNORECASE)

Any trigger matching this pattern — "step_1", "step 2", "STEP_3" — is classified as "generic" instead of "specific". The specificity scorer now correctly identifies these as degenerate triggers that name a step number, not a failure pattern.

But classification alone doesn't stop the match. The specificity scorer flags the trigger as generic, but the matcher still scores it 1.00 and the replay engine still produces a "pass" verdict. The flag is a signal. It is not a gate.

Fix 7b: degenerate trigger rejection in the matcher

The second fix was in the matcher (matcher.py). I added the same regex pattern and made it a hard gate:

_DEGENERATE_TRIGGER_RE = re.compile(r"^step[_\s]*\d+$", re.IGNORECASE)

def rule_matches(candidate, trajectory, threshold=0.70):
    if _DEGENERATE_TRIGGER_RE.match(candidate.trigger):
        return False  # degenerate trigger — never matches

rule_matches() now returns False immediately for any degenerate trigger. No tokenization, no scoring, no substring check. The trigger is rejected before the matcher ever runs. The replay engine gets no match, produces no "prevented" or "broken" verdict, and the candidate fails.

This eliminated the 2 nearmiss false positives (NM-030, NM-044) immediately. "step_1" no longer matches anything. The shortcut is closed.

Fix 7c: the harder question — what other shortcuts exist?

Closing "step_1" was easy. The pattern is regular, the regex is 20 characters, the fix is 3 lines. The harder question is: what other shortcuts is the model finding that I haven't noticed?

After Fix 7, I went back to the remaining 3 nearmiss false positives on Llama — the ones that were NOT caused by "step_1":

Trajectory	Trigger	What it matches	Why it's wrong
N-001-git-nm-001-auth-vs-ff	`"git push fails with authentication error"`	"git push fails with non-fast-forward" reference	Near-miss about auth, not nff — but matcher sees "git push" overlap
N-003-cosmetic	`"git push fails with cosmetic error"`	Wrong reference entirely	"git push" token overlap with unrelated reference
N-004-env-task-different-tool	`"python import fails with wrong module"`	"python ImportError" reference	Wrong tool entirely, but "python" + "import" token overlap

These are not degenerate triggers. They are real, specific, well-formed triggers — "git push fails with authentication error" is a legitimate failure description. The problem is that the matcher can't distinguish "authentication error" from "non-fast-forward" when both are "git push fails with X". The token overlap ("git", "push", "fails") is high enough to score above threshold.

This is a different class of shortcut. The model is not exploiting a structural artifact ("step_1"). It is exploiting a semantic gap in the matcher — the matcher can't tell the difference between two failure classes within the same tool. "git push fails with authentication error" and "git push fails with non-fast-forward" look similar to a token-overlap matcher because they share 3 of 6 tokens. They look different to a human because the failure class is completely different.

Fix 7c is not a regex. It requires the matcher to compare failure_class between the trigger and the matched reference — a trigger-domain mismatch check. If the trigger names "authentication error" but the matched reference has failure_class = "non-fast-forward", downgrade the match score or flag as inconclusive. This is planned for v0.3.0. The "step_1" shortcut is closed. The "wrong failure" shortcut is still open.

Why the model found the shortcut

The model is a 3B local model — Llama-3.2-3B-Instruct, quantized to 4-bit, running on OMLX. It is small, fast, and free. It is also not sophisticated enough to produce the kind of specific, error-class-aware triggers that the 8B cloud models produce. When it can't find a specific failure pattern, it finds something that matches — and "step_1" matches everything.

This is not a bug in the model. It is the model doing what models do: optimizing for the reward signal. The reward signal is "the matcher scores above 0.70." The model produces triggers that score above 0.70. "step_1" scores 1.00. By the reward signal, it is a perfect trigger.

The problem is that the reward signal is wrong. The matcher rewards substring overlap, not failure-pattern match. The model is not misbehaving — it is solving the problem the matcher defines. The matcher defines "a good trigger is one whose tokens appear in the reference trajectory." The model produces a trigger whose tokens appear in every reference trajectory. That is optimal behavior under the matcher's definition.

What I learned from this

Reward hacking is not always malicious. The model didn't "decide" to game the benchmark. It produced a trigger that matched the reward signal. The reward signal was "token overlap with reference trajectories." "step_1" has maximum token overlap. The model is optimizing the objective the matcher defines. If the objective is wrong, the model's behavior is correct under the wrong objective. This is the alignment problem in miniature — not about safety, but about evaluation.

Structural artifacts in your data format are attack surfaces. Every reference trajectory has a step field with a number. "step_1" appears in all of them. The matcher treats this as a match. The model treats this as a reward. Any structural artifact that appears in every trajectory — step numbers, timestamps, session IDs, tool names — is a potential shortcut. The fix is not to remove the artifacts (they are part of the data format). The fix is to make the matcher not reward matches on structural artifacts.

A regex fix is a band-aid. The real fix is semantic. Fix 7b closes the "step_1" shortcut with a regex. It does not close the "wrong failure" shortcut — "git push fails with authentication error" matching "git push fails with non-fast-forward" — because that shortcut is not structural. It is semantic. The matcher can't tell the difference between two failure classes within the same tool. The regex catches the easy case. The semantic gap is the hard case, and it is still open.

Every false positive has a root cause. Trace it. The 5 nearmiss false positives looked like a model quality problem — "the model can't distinguish near-misses from real failures." But tracing each FP to its trigger revealed two different root causes: degenerate triggers (2 FPs, structural shortcut) and wrong-failure matches (3 FPs, semantic gap). The aggregate "5 FPs" is meaningless. The breakdown "2 degenerate + 3 semantic" tells you exactly what to fix. Always trace false positives to individual triggers. The aggregate hides the root causes.

Open questions

The "wrong failure" shortcut (3 nearmiss FPs) requires trigger-domain mismatch detection. How should the matcher compare failure_class between trigger and reference? Keyword comparison? Embedding similarity? A separate classifier? The v0.2.0 matcher has no failure_class awareness — it only sees tokens.

Are there shortcuts I haven't found? The "step_1" pattern was obvious because it is not a failure description. What about triggers like "git" or "python" — single-tool-name triggers that match every trajectory involving that tool? The specificity scorer classifies these as "generic" (score < threshold), but the matcher might still score them above 0.70 on token overlap. Would a minimum-trigger-length check catch these?

The 3B model found the "step_1" shortcut. The cloud models (gpt-4o-mini, llama-3.1-8b) did not — they produced specific triggers like "git push fails with non-fast-forward". Is this because the cloud models are smarter, or because their tokenization is different? If I ran a weaker model, would it find shortcuts the 3B model didn't?

Fix 7b rejects degenerate triggers in the matcher. Should the gate also reject them? Currently, the gate (gate.py) checks for failure signals (exit codes, errors, failure_class). It does not check the extracted trigger for degeneracy. A degenerate trigger that reaches the matcher is already a wasted extraction. Should the gate reject degenerate triggers before the LLM call?

The nearmiss corpus has success=False — these are real failures, not recoveries. Fix 6 (nearmiss recovery detection, documented in learnings-fixes.md §1.5) drops success=True recovery patterns. Fix 7 rejects degenerate triggers. Neither addresses the "wrong failure" FPs. How many nearmiss FPs would remain if Fix 7c (trigger-domain mismatch) were implemented? The data suggests 3 on Llama, 5-7 on cloud post-Fix 8. Would Fix 7c eliminate all of them, or are some "wrong failure" matches genuinely ambiguous?

The broader lesson

If your benchmark rewards surface-level similarity, your model will optimize for surface-level similarity. That is not a model problem. It is a benchmark design problem.

The "step_1" shortcut was the most visible case. But the "wrong failure" false positives are the same lesson at a deeper level. The matcher rewards token overlap. The model produces triggers with maximum token overlap. Sometimes that means "step_1" (structural shortcut). Sometimes that means "git push fails with authentication error" matching "git push fails with non-fast-forward" (semantic shortcut). Both are the model optimizing the objective the matcher defines.

The fix is not to make the model smarter. The fix is to make the matcher's objective align with what "a good trigger" actually means. A good trigger names a specific failure class. A token-overlap matcher can't verify that. A semantic matcher — one that compares failure_class, error codes, and failure semantics — can.

CauterRule's v0.2.0 field test closed the "step_1" shortcut with a regex. The "wrong failure" shortcut is the next one. And it is the same lesson: your model is not misbehaving. Your benchmark is mis-rewarding.

CauterRule v0.2.0 is released. The full nearmiss breakdown — per-model false positives, trigger analysis, and the degenerate trigger rejection fix — is in the field test report and the learnings & fixes document. The repo is public. Install with pip install cauterule. Changelog · Release notes

The 6-Line Fix That Outperformed My Entire Matcher Week

Debashish Ghosal — Tue, 08 Sep 2026 05:39:08 +0000

Update — v0.2.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, TUI review, observability, 7 export formats, adversarial corpora, and a bundled git rule pack. The field test report evaluated 4 models across 745 trajectories and is the source for every number in this article. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

I spent a week fixing the replay matcher. Four fixes — a precision formula bug, 50+ distinctive phrases, 10 new alias entries, a raised threshold floor. 359 validation tests, all green. The golden pass rate moved from 10% to 20%. One week, four fixes, ten percentage points.

Then I found a 6-line fix in the simulator. It checked 6 keywords in failure_class. Golden jumped from 20% to 50% on both cloud models. One fix, six lines, thirty percentage points.

I was optimizing the wrong layer.

The golden corpus was stuck at 20%

The golden corpus contains 10 canonical failure scenarios — git push non-fast-forward, pip version conflict, docker build package not found, kubectl CRD missing, terraform state lock, pytest assertion, deploy timeout. Each has a known expected rule. The model extracts a trigger, the replay engine checks whether that trigger matches the reference trajectory. A pass means the trigger matches the right failure. The release gate target is ≥70%.

Before the matcher fixes, golden was at 10% — 1 pass, 8 inconclusive, 1 fail. The 8 inconclusives were all matcher_gap: the trigger was specific, but the matcher couldn't bridge it to the reference. "When git push fails with non-fast-forward" should match a trajectory whose error says "! [rejected] non-fast-forward" — but the matcher's token-F1 score was 0.50, below the 0.70 threshold.

The v0.1.0 field test report named this as the #1 engineering target: "Replay and matcher calibration is now the highest-value engineering target." So I fixed the matcher.

The matcher week: Fix 1 through Fix 4

Fix 1 was a precision formula bug. The formula was precision = weighted_hit / len(haystack_tokens) — it divided by the size of the reference trajectory, not the size of the trigger. A 6-token trigger matching 5 of them against a 17-token haystack gave precision 5/17 = 0.29, even though 83% of the trigger was covered. I changed it to divide by the trigger size. On G-001 (git push non-fast-forward), the score went from 0.50 to 0.61. Not enough to pass, but the right direction.

Fix 2 added distinctive phrase fallback. "non-fast-forward", "ModuleNotFoundError", "OOMKilled" — 50+ error codes and phrases. If a distinctive phrase appears in the raw trigger and its normalized form appears in the haystack, floor the score at 0.70. On G-001, this pushed the score from 0.61 to 0.70. It passed.

Fix 3 expanded the alias map from 11 to 22 entries — "version conflict" → "dependency resolver conflict", "package not found" → "unable to find", "state lock" → "error acquiring". Fix 4 raised the alias phrase floor from 0.65 to 0.70, so that a verbatim alias match in the haystack counts as a pass.

Here is the score progression for G-001 across all four fixes, traced in the learnings & fixes log:

Fix	What changed	G-001 score
Before all fixes	—	0.50
Fix 1 (precision formula)	`/ haystack` → `/ trigger`	0.61
Fix 2 (distinctive phrases)	"non-fast-forward" floors at 0.70	0.70
Fix 3 (expanded aliases)	Covered G-005, G-010	0.70
Fix 4 (raised floor 0.65→0.70)	Alias phrase hits now pass	0.70

Combined impact: golden inconclusive dropped from 80% to 0%. Every trigger now produces a meaningful verdict. The matcher finds matches for all 10 golden scenarios. 359 validation tests, 0 failures. The matcher was fixed.

But the golden pass rate only moved from 10% to 20%. 2 passes, 8 fails, 0 inconclusive. The 8 "fail" verdicts meant the triggers matched the reference failure — but they also matched clean successes in the reference corpus. The triggers were too broad.

The matcher was fixed. The pass rate was not.

The 6-line simulator fix

The problem was not in the matcher. It was in the simulator — the component that classifies what happens when a trigger matches a reference trajectory.

The simulator takes a candidate trigger and runs it against every reference trajectory. For each trajectory, it asks: does this trigger fire? If it fires on a trajectory that was a failure, that's prevented — the trigger would have caught a real failure. If it fires on a trajectory that was a success, that's broken — the trigger would have interfered with a correct outcome. The classification logic was simple: if trajectory.success == True and the trigger matches, classify as broken. If trajectory.success == False and the trigger matches, classify as prevented.

But some success=True trajectories are not clean successes. They are recoveries — the agent failed, then self-resolved. A trajectory with failure_class = "temp error, retry succeeded" and success = True is a near-miss, not a clean success. The agent encountered a failure, recovered from it, and the final outcome was success. But the failure was real — it just didn't stick.

The simulator was counting these as broken. A trigger that fires on a recovery trajectory is not breaking a success — it is firing on a near-miss that happened to recover. Penalizing the trigger for matching a recovery is wrong. The fix, documented in learnings-fixes.md §4.8, is 6 lines:

RECOVERY_KEYWORDS = {"temp", "near", "retry", "recover", "intermittent", "flaky"}
if trajectory.success and any(kw in trajectory.failure_class for kw in RECOVERY_KEYWORDS):
    classification = "near_miss"  # not "broken"

The numbers

I re-ran the cloud sweep on golden, failures/positive, and nearmiss after Fix 8. The full results are in the v0.2.0 field test report §2:

Corpus	Model	Pre-Fix 8	Post-Fix 8	Delta
golden	gpt-4o-mini	2P / 2F / 6I (20%)	5P / 1F / 4I (50%)	+3 passes
golden	llama-3.1-8b	2P / 2F / 6I (20%)	5P / 1F / 4I (50%)	+3 passes
failures/positive	gpt-4o-mini	15P / 6F / 29I (30%)	22P / 5F / 23I (44%)	+7 passes
failures/positive	llama-3.1-8b	15P / 7F / 28I (30%)	27P / 5F / 18I (54%)	+12 passes
nearmiss	gpt-4o-mini	2 FPs (96% precision)	5 FPs (90% precision)	+3 FPs
nearmiss	llama-3.1-8b	5 FPs (90% precision)	7 FPs (86% precision)	+2 FPs

Golden jumped 20% → 50% on both cloud models. Failures/positive jumped 30% → 44-54% — llama-3.1-8b now meets the ≥50% release threshold for the first time. Nearmiss false positives rose slightly (2→5 on gpt-4o-mini, 5→7 on llama-3.1-8b) — an acceptable tradeoff for the golden and failures gains, but a tradeoff, not a free win.

Fix 8 moved the golden pass rate more than Fix 1-4 combined. One fix, six lines, thirty percentage points. The matcher week gave me ten. The simulator fix gave me thirty.

Why the simulator was the right layer

The matcher fixes were necessary. Without them, 80% of golden scenarios were inconclusive — the matcher couldn't find matches. Fix 1-4 eliminated the inconclusive bucket. Every trigger now gets a decisive verdict. That felt like progress, and it was — but only half the progress I needed.

A decisive verdict is not the same as a correct verdict. The matcher fixes converted inconclusive to fail. The triggers now match — but they match too broadly, so the replay engine says "fail" instead of "inconclusive." The problem shifted from "the matcher can't decide" to "the matcher decides wrong."

The wrong decision was in the simulator's classification. Recovery trajectories with success=True were counted as broken — the trigger broke a success. But they weren't successes. They were recoveries. The trigger didn't break anything — it fired on a near-miss that self-resolved. Fix 8 reclassifies these as near_miss, so the trigger is no longer penalized for matching them.

This is why Fix 8 outperformed Fix 1-4. The matcher fixes addressed the matching layer — can the trigger be found in the reference? The simulator fix addressed the classification layer — when the trigger is found, what does that mean? The matching layer was necessary but not sufficient. The classification layer was where the actual pass rate lived.

Why I missed it for a week

I missed it because the matcher fixes were working. Inconclusive dropped from 80% to 0%. Every trigger got a verdict. The validation suites passed — 359 tests, 0 failures. The field test report's #1 engineering target was "replay and matcher calibration," and I was fixing the matcher. Every signal I had said I was working on the right layer.

The signal I didn't have was the attribution of the remaining 8 "fail" verdicts. The report said the triggers were "too broad" — matching successes they would break. I accepted that framing. The triggers are specific (94.4% specific or moderate per the field test report §A.9), but they match too broadly. The fix must be to make them narrower — prompt tuning, trigger specificity checks, broad-trigger penalty.

But the triggers weren't too broad. The reference corpus was misclassified. The "successes" the triggers were breaking weren't successes — they were recoveries. The triggers were matching correctly. The simulator was classifying incorrectly. The broad-trigger penalty I built — broken > prevented → fail — was working correctly on bad data.

I spent a week optimizing the matcher because the report said the matcher was the problem. The report was right about v0.1.0 — the matcher's 80% inconclusive rate was the #1 issue. But after Fix 1-4 eliminated the inconclusives, the #1 issue shifted to the simulator's classification logic. The report didn't update because I hadn't re-run the field test yet. I was working from a stale diagnosis.

What I learned from this

The biggest fix is often not in the layer the report names. The v0.1.0 report named the matcher as the #1 target. It was right — for v0.1.0. But after fixing the matcher, the #1 target shifted to the simulator. The report didn't update because the field test hadn't been re-run. I was optimizing against a stale diagnosis. Re-run the field test after every fix. Let the new data tell you where the next bottleneck is.

A decisive verdict is not the same as a correct verdict. Fix 1-4 eliminated inconclusives — every trigger now gets a pass or fail. That felt like progress. But the 8 "fail" verdicts were wrong — the triggers were being penalized for matching recoveries, not successes. Converting inconclusive to fail is only progress if the fail verdicts are correct.

Classification bugs masquerade as model quality problems. Golden pass rate was 20% across all four models — local and cloud. That looked like a model capability ceiling. Every explanation pointed to the model or the corpus. None pointed to the simulator's 6-line classification logic. Classification bugs are invisible because they affect all models uniformly — if every model gets the same wrong verdict, it looks like a system property, not a bug.

One fix can outperform four. Fix 1-4 gave me 10 percentage points. Fix 8 gave me 30. The matcher fixes were necessary foundations — without them, Fix 8 would have had no verdicts to improve. But the lever that moved the pass rate was not in the matcher. It was in the simulator. Do not over-invest in one layer just because the report named it first.

Open questions

The remaining 4 golden inconclusives on cloud (50% vs 70% target) are trigger-breadth issues — triggers match the reference but also match clean successes. Would reference corpus expansion (currently 230 trajectories, per learnings-fixes.md §7.1) to 330-430 give the simulator enough data to distinguish "this trigger matches real failures" from "this trigger matches too broadly"? Or is the issue that the triggers themselves need to be narrower?

The nearmiss false positives rose slightly after Fix 8 (gpt-4o-mini 2→5, llama-3.1-8b 5→7). These are "wrong failure" scenarios — "git push fails with authentication error" matching "git push fails with non-fast-forward". Fix 8 doesn't address them because they have success=False. Would a failure_class mismatch check in the matcher catch these without hurting the golden gains?

Fix 8 has not been re-run on local OMLX models yet. The fix is model-independent — it changes simulator classification, not extraction. But local models produce broader triggers. Will the recovery exclusion help them as much as it helped cloud, or will the broader triggers still match too many clean successes?

The 6 recovery keywords ("temp", "near", "retry", "recover", "intermittent", "flaky") were chosen from the annotated corpus vocabulary. Are there recovery patterns in the corpus that these keywords miss? A trajectory with failure_class = "transient network blip" would not match any of the 6. Would a semantic classifier (embeddings on failure_class) be more robust than keyword matching?

If I had re-run the field test after Fix 1-4 instead of continuing to Fix 5-7, would the data have pointed me to the simulator sooner? The 8 "fail" verdicts with broken > prevented were the clue — the triggers were matching recoveries, not successes. But I didn't look at which specific trajectories were being counted as "broken." I accepted the "too broad" framing and moved to the broad-trigger penalty.

The broader lesson

When a bottleneck persists across every model — local and cloud, 3B and 8B — the problem is almost never in the model. It is in the evaluation layer that classifies the model's output. A model-independent failure is a classification failure.

CauterRule's golden pass rate was stuck at 20% for a week because I was fixing the matcher when the simulator was the problem. The matcher was the named bottleneck in the v0.1.0 report, and it was right — for v0.1.0. But after the matcher was fixed, the bottleneck shifted. I didn't notice because I didn't re-run the field test. I kept optimizing the named layer instead of letting the data re-diagnose.

The 6-line fix that moved golden 20% → 50% was not a clever insight. It was a classification bug that affected all models uniformly. It looked like a model ceiling because every model hit the same 20%. It was a 6-line bug in the simulator that counted recoveries as successes.

If a bottleneck persists across every model you test, stop tuning the model. Stop tuning the matcher. Look at the layer that classifies the output — the layer that decides what a "pass" and a "fail" mean. That is where model-independent failures live. And that is where the biggest fixes hide.

CauterRule v0.2.0 is released. The full fix-by-fix breakdown — Fix 1 through Fix 8, with before/after scores on every golden scenario — is in the field test report and the learnings & fixes document. The repo is public. Install with pip install cauterule. Changelog · Release notes

When Your Judge Can't Decide

Debashish Ghosal — Mon, 07 Sep 2026 19:52:17 +0000

Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, MCP server, 7 export formats, and a bundled git rule pack. The field test report evaluated 4 models across 394 trajectories and is the source for every number in this article. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

Every benchmark produces three buckets: pass, fail, and inconclusive. Most people only read the first two. In CauterRule's field test, the third bucket was bigger than both of them combined — and it was the one that mattered most.

The majority of our results were "can't tell"

Across all four models and the full corpus, the field test produced 1,538 candidates. Here is how they split:

Verdict	Count	% of total
Pass	365	23.7%
Fail	359	23.3%
Inconclusive	814	52.9%

More than half of every result the benchmark produced was inconclusive. The replay engine could not decide whether the rule was good or bad. It did not pass the candidate. It did not reject it. It returned "maybe."

That is not a fringe bucket. It is the majority of the output. And it is the bucket that most benchmark reports skip entirely, because it is uncomfortable to talk about. A pass is a win. A fail is a loss. An inconclusive is an embarrassment — it means your evaluation tool could not do its job.

But if you ignore the inconclusive bucket, you are ignoring more than half of your data. And you are probably drawing the wrong conclusions from the half you kept.

The inconclusive rate varied by model — but not the way you'd expect

Model	Candidates	Inconclusive	Inconclusive %
Local Llama 3.2B	379	189	49.9%
Local Qwen 4B	373	209	56.0%
Cloud GPT-4o-mini	394	248	62.9%
Cloud Llama 3.1 8B	392	168	42.9%

The strongest cloud model — meta-llama/llama-3.1-8b-instruct — had the lowest inconclusive rate at 42.9%. That makes sense: better extraction produces more specific triggers, which are easier for the replay engine to verify.

But gpt-4o-mini — a paid cloud model — had the highest inconclusive rate at 62.9%. Higher than both local models. This is the kind of finding that only surfaces when you stop reading pass counts and start reading the full distribution. On raw pass count, GPT-4o-mini looked comparable to the locals (77 vs 72 vs 93). On inconclusive rate, it was the most indecisive model in the field test.

That does not mean GPT-4o-mini is a bad model. It means the model produced candidates that the replay engine could not evaluate. The model's output was not worse — it was less scorable. Those are different failures, and they point to different fixes.

What "inconclusive" actually means

The replay engine takes a candidate rule and runs it against reference trajectories. It checks whether the rule's trigger fires on the right cases (recall) and stays silent on the wrong ones (precision). If both precision and recall are high enough, the verdict is pass. If the rule clearly fires on wrong cases or misses all right cases, the verdict is fail.

If neither condition is clearly met, the verdict is inconclusive.

The matcher that decides this is not a sophisticated judge. It uses substring and token-overlap heuristics. It checks whether words in the trigger appear in the trajectory. If the overlap is in a gray zone — not high enough to pass, not low enough to fail — the matcher returns inconclusive.

This is not a secret. The field test report names it explicitly as the #1 engineering target:

Replay and matcher calibration is now the highest-value engineering target.

The report also says:

The matcher and simulator are still heuristic and can over- or under-fire on substring/token overlap.

That means the inconclusive bucket is contaminated. It contains at least two different populations:

Rules that are too vague to verify — the model extracted a generic trigger like "when a command fails" instead of "when git push fails with non-fast-forward." The matcher cannot tell whether this fires on the right cases because the trigger is too broad to evaluate. This is a model problem.
Rules that are specific and reasonable but the matcher is too weak to confirm — the model extracted a good rule, but the substring/token-overlap heuristic cannot recognize that the trigger matches the reference trajectory. This is an engine problem.

The field test data cannot cleanly separate these two populations. That is itself a finding: the benchmark is not instrumented well enough to tell you why a result is inconclusive. And without that, you cannot tell whether the inconclusive bucket is mostly model weakness or mostly engine weakness.

The curated vs raw split reveals where inconclusives concentrate

The sharpest insight comes from splitting the inconclusive rate by corpus type.

Model	Curated inconclusive %	Raw inconclusive %
Local Llama 3.2B	33.0%	55.2%
Local Qwen 4B	34.0%	63.4%
Cloud GPT-4o-mini	38.3%	70.7%
Cloud Llama 3.1 8B	14.9%	51.7%

On curated corpora, the inconclusive rates are manageable — especially for the strongest cloud model (14.9%). On raw corpora, every model jumps above 50%. GPT-4o-mini hits 70.7%.

Two raw corpora drove most of the inconclusive volume:

Raw corpus	Llama 3.2B inc %	Qwen 4B inc %	GPT-4o-mini inc %	Llama 8B inc %
`raw/ci`	45.0%	81.8%	90.0%	54.5%
`raw/sibling-repos`	90.0%	90.0%	70.0%	100.0%

raw/ci produced 110 trajectories per model. For GPT-4o-mini, 99 of those 110 were inconclusive — 90%. For Qwen 4B, 90 of 110 — 82%. The replay engine was essentially paralyzed on this corpus.

raw/sibling-repos was worse: 10 trajectories per model, and for the strongest cloud model, all 10 were inconclusive. 100%. The engine could not produce a single decisive verdict.

The report's interpretation is direct:

Raw corpora, especially raw/ci, produce many inconclusive outcomes because the current replay setup is a weak fit for that breadth.

raw/sibling-repos remains mostly inconclusive for both cloud models, meaning harness-transfer quality is still not strong.

That is an important distinction. The inconclusive problem is not evenly distributed. It is concentrated in raw corpora where the replay engine's heuristics are a poor fit for the data shape. On curated corpora — where the trajectories are clean, the failure signatures are clear, and the expected rules are known — the engine performs much better.

Why better models did not solve this

If inconclusives were mostly a model problem, stronger models should reduce the inconclusive rate. They did, but only partially.

Model	Inconclusive %	Pass %	Fail %
Local Llama 3.2B	49.9%	19.0%	31.1%
Local Qwen 4B	56.0%	24.9%	19.0%
Cloud GPT-4o-mini	62.9%	19.5%	17.5%
Cloud Llama 3.1 8B	42.9%	31.4%	25.8%

The strongest model reduced inconclusives from ~50-63% to ~43%. That is a real improvement, but it left the inconclusive bucket as the largest category. Even the best model in the field test could not get the replay engine to a decisive verdict on the majority of its output.

And the cloud model with the best parse stability — gpt-4o-mini — had the worst inconclusive rate. Perfect formatting did not help the matcher decide. That confirms what the report says: better structured output does not automatically translate to good replay outcomes. The bottleneck is not extraction. It is judgment.

This is the same conclusion the field test report reaches from a different angle. The cloud runs "proved that better structured output does not automatically translate to good safety behavior or high replay pass rates." The inconclusive data is the concrete evidence for that claim.

What the inconclusive bucket costs you

The inconclusive bucket is not neutral. It has real costs that compound if you ignore it.

It makes your pass rate look lower than it is. If 53% of your results are inconclusive, your pass rate is measured against a denominator that includes a large chunk of "we don't know." The true pass rate — the rate at which the model produces good rules — is somewhere between the reported pass rate and pass-plus-inconclusive. For Llama 8B, that range is 31.4%–74.2%. That is a massive uncertainty band, and it means the pass count is a floor, not a ceiling.

It makes your fail rate look lower than it is. The same logic applies to fails. Some inconclusives are rules that should have been rejected but the matcher could not prove it. Those are hidden failures — safety risks that do not show up in the fail count because the engine was too weak to catch them.

It blocks promotion. In a production system, inconclusive rules are not promoted. They sit in a queue waiting for human review or better evaluation. If 53% of your output is inconclusive, your promotion pipeline is choked. You are producing candidates faster than you can adjudicate them.

It hides the real product problem. When you cannot tell whether a rule is good or bad, you cannot tell whether your product is good or bad. The inconclusive bucket is a measurement gap, and measurement gaps are where product risks hide. The report is explicit: "the benchmark does not yet turn that processing into consistently actionable quality signals."

What a better judge would look like

The current replay engine is a string matcher. It checks whether words in the trigger appear in the trajectory. That is a reasonable starting point, but it is not sufficient for the problem CauterRule is trying to solve.

A better judge would need to do three things the current engine cannot:

Semantic matching. Instead of substring overlap, the matcher needs to understand whether a trigger like "when a non-fast-forward push is rejected" fires on a trajectory where the error message says "Updates were rejected because the remote contains work that you do not have locally." Those are the same event described differently. A string matcher misses that. A semantic matcher — even a lightweight embedding-based one — would catch it.

Corpus-aware calibration. The same matcher threshold does not work for all corpora. Curated corpora with clean failure signatures can tolerate stricter matching. Raw corpora with messy, multi-step trajectories need looser matching or a different matching strategy entirely. The report says the replay setup is "a weak fit for that breadth." The fix is not one better matcher. It is a matcher that adapts to the corpus type.

Inconclusive attribution. When the matcher returns inconclusive, it should say why. Is the trigger too broad? Is the trajectory too messy? Is the matcher's vocabulary too narrow? Without attribution, every inconclusive looks the same, and you cannot prioritize fixes. The current benchmark lumps all inconclusives together, which is why the report says "the benchmark still mixes several failure modes together."

What I learned from this

The inconclusive bucket is the most important column in the table. Pass and fail tell you what the system decided. Inconclusive tells you what the system could not decide. In a product where the core value proposition is "we promote only safe rules," the inability to decide is a bigger risk than a wrong decision. A wrong decision can be caught and reverted. An undecidable result sits in limbo forever.

Your benchmark is only as good as its judge. CauterRule's extraction layer improved dramatically after the parser fixes. The model comparison produced clean, parseable candidates across 1,538 rows. But the replay engine — the thing that decides whether those candidates are good — is still a heuristic string matcher. That means the benchmark's quality ceiling is set by the judge, not by the models. Investing in better models while the judge is weak is optimizing the wrong layer.

Raw breadth without evaluation depth is not evidence. The raw corpora produced 1,175 candidates across all models. 745 of those were inconclusive — 63%. The product can process raw corpora, but it cannot evaluate them. The report says this directly: "broad coverage that produces mostly inconclusives is not yet the strongest evidence of product trustworthiness." If you are benchmarking a learning system, do not confuse processing coverage with evaluation coverage.

The inconclusive rate is a product health metric, not just a benchmark metric. If your inconclusive rate is 53%, your product has a measurement problem that blocks every downstream decision — promotion, safety, trust. Tracking the inconclusive rate over time tells you whether the product is getting better at judgment, not just at extraction. It should be a first-class metric in any learning system's dashboard.

Open questions

Would a semantic matcher (even a lightweight embedding-based one) materially reduce the inconclusive rate on raw corpora, or is the problem deeper than matching strategy? The current data cannot answer this because we only tested one matcher.
The curated inconclusive rate for Llama 8B was 14.9%. Is that the realistic floor for the current matcher, or can calibration push it lower? If 15% is the floor, that means even on clean data, 1 in 7 rules cannot be evaluated.
GPT-4o-mini had the highest inconclusive rate (62.9%) despite perfect parse stability. Is that because the model produces more nuanced triggers that the string matcher cannot handle, or because it produces vaguer triggers? The data does not distinguish.
The raw/sibling-repos corpus produced 100% inconclusives for the strongest cloud model. Is that a matcher problem, a corpus problem, or a harness-transfer problem? The report says "harness-transfer quality is still not strong," which suggests the corpus itself may be the issue, not the engine.
If we split the inconclusive bucket by root cause (model vagueness vs engine weakness), what proportion would fall into each? That split would completely change the product roadmap. If 70% of inconclusives are engine weakness, the fix is replay calibration. If 70% are model vagueness, the fix is prompt engineering and trigger specificity.

The broader lesson

If you are building any system that evaluates AI output — a replay engine, a guardrail checker, a quality scorer, a promotion gate — track the inconclusive rate as seriously as you track pass and fail. The inconclusive bucket is where measurement gaps live. And measurement gaps are where product risks hide.

A benchmark that returns 53% "can't tell" is not measuring quality. It is measuring uncertainty. And uncertainty is the most expensive thing in a learning system, because it blocks every downstream decision without telling you why.

The next milestone for CauterRule is not more extraction breadth or better models. It is a judge that can decide.

CauterRule v0.1.0 is released. The full model-by-model breakdown — including pass/inconclusive/fail distributions for all 4 models across curated and raw corpora — is in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes

A Better Model Improved the Numbers. It Didn't Fix the Product.

Debashish Ghosal — Mon, 07 Sep 2026 04:50:00 +0000

Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you 25+ CLI commands, an MCP server, 7 export formats, and a bundled git rule pack. The field test report is the source for every number in this article: 4 models, 13 corpus types, 394 trajectories, 844+ deterministic tests passing. The strongest cloud model tested was meta-llama/llama-3.1-8b-instruct with 123 pass / 168 inconclusive / 101 fail — better than every other model, but still not enough to solve the safety problem on its own. Release notes · Changelog

There is a comforting fantasy in AI engineering. When a system struggles, you tell yourself the problem is the model. Spend a little more, use a stronger one, and the product gets better.

Sometimes that is true. One of the most useful outcomes from the CauterRule field test was learning exactly where that story stops being true.

Why we ran cloud models

After fixing the parser, prompt, result-reset, and corpus timestamp issues, we finally had a benchmark we could trust. That was the moment cloud comparison became worth doing. We ran two working cloud models through the full corpus: openai/gpt-4o-mini and meta-llama/llama-3.1-8b-instruct.

The point was not just to get better numbers. It was to answer a sharper question:

Are the remaining problems mostly local-model weakness, or are they product problems?

The answer turned out to be both, but mostly the second.

All-model comparison: local vs cloud

The broadest comparison is the full-corpus view.

Model	Type	Total rows	Total candidates	Pass	Pass %	Inconclusive	Fail
`omlx-openai-Llama-3.2-3B-Instruct-4bit`	local	393	379	72	19.0%	189	118
`omlx-openai-Qwen3-4B-Instruct-2507-4bit`	local	373	373	93	24.9%	209	71
`openai/gpt-4o-mini`	cloud	394	394	77	19.5%	248	69
`meta-llama/llama-3.1-8b-instruct`	cloud	394	392	123	31.4%	168	101

Three things jump out. First, the cloud models removed most of the remaining availability and formatting ambiguity — both produced near-perfect parse reliability. Second, meta-llama/llama-3.1-8b-instruct produced the strongest overall pass count of any model we tested: 123 passes from 392 candidates. Third, gpt-4o-mini was still operationally valuable because it was stable, cheap, and fully runnable across all 394 trajectories.

But there is something more subtle in this table that matters more than the ranking. Even the strongest tested model still produced 168 inconclusives and 101 hard fails. That is 269 out of 392 candidates — 69% of the output was not clearly usable. Better models improved the system. They did not settle the product.

Curated comparison: where model quality actually mattered most

The curated corpora are the cleanest lens for this comparison.

Curated corpus	Local Llama 3.2B	Local Qwen 4B	Cloud GPT-4o-mini	Cloud Llama 3.1 8B
`golden`	8P / 2F	3P / 4I / 3F	6P / 4I / 0F	7P / 0I / 3F
`failures/positive`	15P / 3I / 12F	15P / 9I / 6F	14P / 13I / 3F	22P / 5I / 3F
`failures/negative`	1P / 2I / 5F	1P / 4I / 5F	0P / 4I / 6F	3P / 1I / 6F
`successes`	1P / 16I / 2F	2P / 8I / 10F	0P / 6I / 14F	3P / 6I / 11F
`nearmiss`	4P / 4I / 6F	5P / 5I / 4F	2P / 4I / 8F	6P / 1I / 7F
`noisy`	1P / 3I / 1F	4P / 1I / 0F	1P / 3I / 1F	5P / 0I / 0F
`corrections`	2P / 2I / 1F	2P / 1I / 2F	3P / 2I / 0F	3P / 1I / 1F

If you want the shortest honest reading of that table: cloud models clearly help on quality. meta-llama/llama-3.1-8b-instruct was the strongest model tested. And the positive-case corpora improved much faster than the safety corpora.

That last point matters more than the ranking itself.

Winner-by-corpus view

This view is a better way to summarize what actually improved.

Corpus	Best model	Why it matters
`golden`	Local Llama 3.2B by raw pass count, Cloud Llama 3.1 8B by strongest cloud showing	regression anchor / specificity
`failures/positive`	Cloud Llama 3.1 8B	best practical extraction benchmark
`successes`	none were convincingly strong	safety / over-triggering check
`failures/negative`	none were convincingly strong (all 5-6 fails)	rejection quality
`nearmiss`	Cloud Llama 3.1 8B, but still mixed	trigger precision
`noisy`	Cloud Llama 3.1 8B	strongest clutter-handling
`corrections`	Cloud GPT-4o-mini and Cloud Llama 3.1 8B both strong	correction-to-rule synthesis

The cloud models won where learning-from-failure was easiest. They did not decisively win where restraint mattered most.

The raw corpus results confirm the pattern

The raw corpora add volume and show the same pattern at scale.

Raw corpus	Local Llama 3.2B	Local Qwen 4B	Cloud GPT-4o-mini	Cloud Llama 3.1 8B
`raw/opencode`	13P / 2I / 9F	13P / 7I / 5F	12P / 10I / 3F	18P / 3I / 4F
`raw/synthetic`	18P / 101I / 26F	36P / 87I / 22F	28P / 91I / 26F	38P / 81I / 25F
`raw/ci`	8P / 49I / 52F	9P / 90I / 11F	7P / 99I / 4F	11P / 60I / 39F
`raw/sibling-repos`	0P / 9I / 1F	0P / 9I / 1F	0P / 7I / 3F	0P / 10I / 0F
`raw/corrections`	2P / 2I / 1F	3P / 1I / 1F	2P / 3I / 0F	3P / 0I / 1F
`raw/cross-session`	2P / 1I / 2F	1P / 2I / 2F	2P / 2I / 1F	4P / 0I / 1F

Cloud Llama 3.1 8B led on raw/opencode (18P), raw/synthetic (38P), raw/corrections (3P), and raw/cross-session (4P). But on raw/ci, it still produced 60 inconclusives and 39 hard fails from 110 trajectories. That is a lot of noise from the strongest model in the batch.

The report makes an observation about this that is worth pulling out: raw breadth without replay clarity is not yet evidence. The product can process broad raw corpora, but if it mostly yields inconclusive judgments, that is not the same as producing trustworthy results. Broad processing coverage is good. But broad coverage that produces mostly inconclusives is not yet the strongest evidence of product trustworthiness.

What the cloud models clearly improved

The cloud runs gave us much cleaner evidence than the early local-only picture:

near-perfect parse reliability
complete candidate generation across the corpus
stronger results on golden and failures/positive
stronger performance on corrections and broad raw corpora

One model stood out: meta-llama/llama-3.1-8b-instruct was the strongest cost-effective model in the entire batch. On failures/positive, it produced 22 pass / 5 inconclusive / 3 fail — the best single result on the most practical extraction benchmark. On noisy, it produced 5 pass / 0 inconclusive / 0 fail — a clean sweep. On raw/opencode, it produced 18 pass — more than any other model.

The report's verdict: best current cost/performance benchmark model. Not because it was the most expensive or the largest, but because it consistently outperformed gpt-4o-mini on the corpora that matter most for practical rule extraction.

What better models did not fix

This is the more important half of the story.

Even with stronger cloud models, the system still struggled on the corpora that matter most for trust: successes, failures/negative, and nearmiss.

The simplest local-vs-cloud safety view:

Safety-sensitive corpus	Best local outcome	Best cloud outcome	What it suggests
`successes`	still weak	still weak	replay and restraint remain product issues
`failures/negative`	none were convincingly strong	none were convincingly strong	stronger generation does not solve rejection quality
`nearmiss`	mixed	mixed	trigger specificity remains a hard problem

On successes, the strongest cloud model produced 3 pass / 6 inconclusive / 11 fail. Those 3 "passes" are not a win — they are false positives. A "pass" on the successes corpus means the system extracted a rule from a trajectory that should not have produced one. Every pass on successes is a safety violation.

On failures/negative, the strongest cloud model produced 3 pass / 1 inconclusive / 6 fail. Every model in the field test produced 5-6 hard fails on this corpus, and no model produced more than 3 passes. The rejection gate is weak regardless of which model drives extraction.

On nearmiss, Cloud Llama 3.1 8B produced 6 pass / 1 inconclusive / 7 fail. Better than the locals, but still more failures than passes. The trigger is firing on cases that look similar but should not trigger.

If cloud models had cleaned those up, I would be writing a very different article. They did not.

The gap is not model quality — it is judgment quality

This is the insight that changed how I think about the roadmap.

Before the cloud runs, it was possible to believe the remaining weakness was mostly a model problem. Small local models, limited instruction-following, noisy output — maybe a bigger model would fix it.

After the cloud runs, that explanation is dead. A stronger model with near-perfect parse reliability, running on a now-trustworthy benchmark, still produced 11 fails on successes and could not reliably reject bad candidates on failures/negative. The problem is not that the model cannot generate rules. The problem is that the product's judgment — when to extract, when to reject, when to stay silent — is not strong enough.

The report identifies six specific gaps between the current state and a stronger release claim:

Safety gap — the system performs too weakly on successes, failures/negative, and nearmiss.
Replay-trust gap — the matcher and replay engine produce too many inconclusives, especially on raw corpora.
Promotion-confidence gap — many plausible candidates, but not enough evidence they are consistently safe to promote.
Human-judgment gap — the report leans on replay heuristics more than human-reviewed rule quality.
Release-criteria gap — no explicit, enforced pass/fail thresholds tied to safety-sensitive corpora.
Operational-comparison gap — we know which models are promising, but need a crisper policy for which model class qualifies as authoritative evidence.

The first three are the real release blockers. The others matter, but safety, replay trust, and promotion confidence are what define whether the product can honestly claim v0.1.0 field-test success in the strongest sense.

What I learned from this

Better models improve the ceiling, not the floor. The cloud models lifted the best results — more passes on failures/positive, cleaner noisy handling, stronger raw corpus extraction. But they did not lift the worst results. successes and failures/negative stayed weak. That tells you the floor is a product problem, not a model problem.

A "pass" on a safety corpus is not always a win. On successes, a pass means the system extracted a rule from a trajectory that should not have produced one. Counting passes without reading the corpus label gives you a false sense of progress. You have to know what each corpus is testing before you interpret the number.

Inconclusives are the silent killer. gpt-4o-mini produced 248 inconclusives out of 394 candidates — 63% of its output was uncommitted. Those inconclusives did not show up as failures, so they did not trigger alarm. But they also did not produce usable rules. A model that is 63% inconclusive is not producing 77 passes. It is producing 77 passes and 248 maybes. Maybes do not build trust. And the report is clear that some of those maybes are a replay-engine limitation, not a model limitation — the matcher is the named #1 bottleneck. But on the safety corpora, inconclusives are a different signal: they mean the model extracted something from a trajectory where it should have stayed silent, and the matcher could not cleanly reject it. Those are model problems hiding behind engine weakness. You have to split the inconclusive bucket by corpus before you know which ones are the matcher's fault and which ones are the model's.

The field test succeeded even though the product did not fully pass. This sounds contradictory but it is not. The field test succeeded because it forced the system through realistic comparisons, exposed tooling defects, validated that fixes improved signal, and produced a coherent map of what remains. The product improved materially. The product is not yet fully field-test complete by a strict release gate. Both of those things are true. That is a good outcome for an honest engineering report.

Open questions

Would a much larger model (70B, 405B) close the safety gap, or is the successes problem fundamentally about replay logic, not extraction quality? The 8B results suggest the latter, but we have not tested it.
The failures/negative corpus is small (8 trajectories per model). Would a larger negative corpus surface more rejection failures, or is 8 enough to see the pattern? The current results are already weak enough to be concerning at this scale.
gpt-4o-mini produced zero fails on golden (6P / 4I / 0F) but 14 fails on successes (0P / 6I / 14F). That is a striking contrast. Is the model genuinely good at extraction and genuinely bad at restraint, or is the successes corpus testing something the model was never designed to handle?
The report recommends safety-first quality gates: promote only when successes, failures/negative, and nearmiss support the claim. What happens to the promotion rate if we enforce that? If it drops to near zero, that tells us the current candidate pool is not safe enough. If it stays nonzero, that gives us a credible set of rules to build on.

Why this is actually good news

If the cloud models had solved everything, the conclusion would have been easy: just use a better model. That would be convenient, but it would teach us less.

What happened instead was better. The cloud runs improved the benchmark enough to expose the real work that remains. They showed that stronger models help a lot, but they also showed that the core challenge now lives in replay trust, safety, and promotion confidence.

That is exactly what a serious field test should surface.

The cloud models made the product look better. They also made its remaining weaknesses harder to deny. That is why we ran them.

CauterRule v0.1.0 is released. The full all-model comparison — local vs cloud, curated and raw corpora, winner-by-corpus view, and model recommendations — is in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes

You Aren't Choosing an AI Tool. You're Choosing Who Gets Paged at 2 AM.

Debashish Ghosal — Sun, 06 Sep 2026 21:43:50 +0000

Part of AI Leadership in the Real World — how leaders turn scattered pilots into governed, adopted, measurable capability.

TLDR: A support agent doing 50,000 chats a month needs ~3.5 FTE and $500k+/year just to stay accurate — while a typical 100-seat Copilot rollout sees only 20-30 seats used weekly. For SMBs, build-vs-buy isn't about features. It's about what you can afford to own for 24 months.

We thought we were choosing a tool. We were really choosing a future dependency, a support queue, a governance burden, and a second bill that arrives a year later.

Every vendor demo promised acceleration, control, and simplicity at once. Every internal proposal promised flexibility, ownership, and leverage. Nobody said both bills arrive late — one in engineering on-call, the other in consumption meters.

Good platform decisions feel a little boring at first and very smart a year later.

Why AI is special (and why old build-vs-buy math breaks)

Traditional software mostly stays still when you leave it alone. AI doesn't:

It drifts. Knowledge changes, customer language shifts, users ask harder questions once they trust it. Accuracy quietly drops from 90% to 70% with no error log.
It speaks for you — legally. A wrong Confluence page is embarrassing. A wrong chatbot answer is a commitment a tribunal can enforce.
It lives on someone else's deprecation clock. OpenAI gives at least 6 months before retiring a GA model. That's a hard deadline, not a backlog item. Prompts, evals, and output parsers all need rework.
It multiplies cost per request. One human click = one action. One agent resolution = 6 lookups, drafts, updates, and logs — each potentially metered.
It turns connectors into permanent work. Salesforce, SharePoint, Jira, Zendesk all change auth, rate limits, and APIs. Your agent keeps running while its knowledge goes stale.

Gartner predicts 40%+ of agentic AI projects will be canceled by end of 2027 on cost, unclear value, and weak risk controls. McKinsey's State of AI 2025 (Nov 5, 2025) found 88% report regular AI use in at least one function, yet only 39% report any enterprise EBIT impact (most under 5%; ~6% high performers). BCG's Widening AI Value Gap (2025, n=1,250) found only 5% generating value at scale while 60% see minimal gains. The gap isn't building. It's owning.

Here are 6 public cases every SMB CTO should know — scenario, what happened, what didn't work, and why.

1. The chatbot that created legal liability: Air Canada (2024)

Scenario: Jake Moffatt, booking emergency travel after his grandmother died, asks Air Canada's website chatbot about bereavement fares. The bot says: book full fare now, claim the discount within 90 days.

What happened: That was wrong. The real policy barred retroactive claims. Moffatt flew, applied with screenshots and a death certificate, was denied. He took it to the British Columbia Civil Resolution Tribunal — Moffatt v. Air Canada, 2024 BCCRT 149 (Feb 14, 2024).

The tribunal ordered Air Canada to pay C$812.02 (C$650.88 fare difference + interest + fees) for negligent misrepresentation. It explicitly rejected Air Canada's "remarkable" defense that the chatbot was a separate entity responsible for its own actions: "it is still just a part of Air Canada's website."

What didn't work and why: No grounding to the actual policy page, no policy-layer guardrail, no correction path. The bot hallucinated a generous policy linked to the real bereavement page that contradicted it.

SMB lesson: If you ship a customer-facing agent — bought or built — you own what it says. For a 20-person company, one such incident is a support crisis and a trust crisis. Budget the eval + human escalation before launch, not after.

2. The "AI platform" that was mostly services: Builder.ai (2025)

Scenario: Builder.ai, founded 2016 as Engineer.ai, marketed "Natasha" as AI that builds software "as easy as ordering pizza." Raised $450M+, valued at ~$1.5B in 2023 with Microsoft and Qatar Investment Authority backing.

What happened: On May 20, 2025 it entered insolvency. Audits revised claimed 2024 revenue of $220M down to ~$55M. Creditor Viola Credit seized ~$37-40M after a $50M debt facility. The US Attorney's Office for SDNY subpoenaed records. UK entity Engineer.ai Global Ltd went into compulsory liquidation.

Viral coverage said "700 engineers faked the AI." That shorthand is contested — Gergely Orosz (Pragmatic Engineer), after talking to ex-employees, corrected it: there was a real AI team doing spec, estimation, and code workflows, alongside hundreds of outsourced developers doing delivery. WSJ had flagged heavy human reliance as early as 2019.

What didn't work and why: The hybrid model can be legitimate. The failure was marketing opacity + financial misrepresentation: buyers couldn't tell which part was automated, which was human, and what would happen if the vendor collapsed. Customers who outsourced their roadmap to it lost the platform overnight.

SMB lesson: When buying an "AI platform," diligence the labor model: what % is model vs. reusable components vs. humans? What happens to your code, data, and uptime if they go under? Get escrow and export in writing.

3. The seat license nobody sat in: Microsoft 365 Copilot

Scenario: A 100-person SMB buys 100 Copilot for Microsoft 365 seats at $30/user/month ($36,000/year) — plus the required underlying M365 license upgrade many teams forget to price.

List price: Microsoft lists Microsoft 365 Copilot at $30/user/month (annual commitment) as an add-on requiring a qualifying M365 base plan — true all-in $42-$90/user/month depending on base tier. Copilot Studio is separate consumption: ~$200/mo per 25,000-credit pack.

What breaks: Independent utilization surveys consistently report only 20-30% of purchased seats see weekly active use at scale (directional, not audited). Change management (training, comms, support) is repeatedly estimated at 30-50% of license cost and rarely in the proposal.

Separate cautionary tale (consumer, not enterprise): Australia's ACCC filed Federal Court proceedings Oct 27, 2025 alleging Microsoft misled ~2.7M Personal/Family subscribers by bundling Copilot rises ($109→$159 Personal, $139→$179 Family) while hiding the cheaper Classic no-AI option in the cancellation flow. The widely cited 100%+ ROI figures trace to a Microsoft-commissioned Forrester TEI study — real data, not independent data.

SMB scale-down: At 25 seats instead of 100, the waste is smaller in dollars but identical in ratio — 5-8 active seats carry the other 17. That's why seats must be earned, not allocated.

What didn't work and why: Per-seat pricing assumes uniform adoption. In SMBs adoption is spiky: 15 power users love it, 60 try twice, 25 never enable it. You pay for 100, get value from 25. The prerequisite upgrade + idle seats kills ROI by month four.

SMB lesson that worked: Pilot with a 25-seat power cohort, instrument weekly active use from day one, and only expand when active-use >60% for 4 weeks. Disciplined teams treat seats as earned, not allocated.

4. The second meter: Salesforce Agentforce ($2/conversation)

Scenario: A mid-size team handling 50,000 service interactions a month turns on Agentforce. List price: $2 per conversation — on top of existing Service Cloud seats.

List price: $2 per conversation at list, before discounts — and before the Service Cloud seats underneath. Salesforce's current official model is Flex Credits ($500/100k; 1 standard action = 20 credits = $0.10) as the flexible alternative, with unused credits not rolling over and Flex vs. Conversations not mixable in one org.

What breaks: Conversation definitions (failed/escalated/abandoned handling) and per-workflow action counts are contract-specific — which is exactly why pre-sign modeling matters. Internal IT/HR agents with lots of short queries get the worst unit economics on per-conversation pricing — same $2 whether the answer saved $200 or $2.

SMB scale-down: At 5,000 interactions/mo instead of 50,000, the meter drops 10x (~$10k/mo at list) — but the modeling work doesn't. You still must define the billable unit in writing before signing.

What didn't work and why — usage amplification: A human resolving a ticket = 1 action. An agent resolving it = query record + search KB + draft response + update case + send email + log interaction = 6 metered steps, depending on config. Teams estimated on user requests but were billed on agent actions.

Copilot Studio has the same pattern: $200/mo for 25,000 credits, but generative actions burn credits faster than classic flows, plus separate Azure token + compute charges. Three bills, three consoles.

SMB lesson that worked: Demand a cost-modeling pilot in writing: define "conversation/credit" for your workflow, exclude tests, cap overages, negotiate rollover + 24-month price lock. Design agents to cache lookups and confirm intent before chaining actions — teams report 20-40% consumption cuts from flow design alone.

5. The build that became a hidden product: the 3.5-FTE support agent

Scenario (composite SMB, modeled on SearchUnify's 50k/mo reference — not a single named company): A 30-person SaaS company builds a RAG support agent on LangGraph + Confluence + Zendesk. Pilot hits 88% answer accuracy. CEO calls it "done."

What happened (per SearchUnify's 2026 vendor field analysis — directional, not audited): To keep a 50k-interaction/mo agent production-grade you need roughly: 1.0 AI/ML engineer + 1.0 data engineer + 0.5 platform + 0.5 security/governance + 0.5 product analyst = 3.5 FTE, ~$500k-700k/year before infra, tokens, observability, and audits. Major model migrations take 6-10 weeks each (re-benchmark, re-prompt, re-test, re-secure). Connectors drift: auth changes, endpoints retire, docs grow by thousands of pages.

Three drifts compound: knowledge drift (docs change), data drift (tickets change), behavioral drift (users ask harder questions). Without evals, you find out from angry customers.

SMB scale-down: At 5k interactions/mo the token/infra meter drops ~10x — but the governance floor doesn't. You still need someone to own drift review, connector fixes, and the model-deprecation deadline. That's the firewall that pages you at 2 AM when the model retires or the connector breaks.

What didn't work and why: The team budgeted 8 weeks to build, zero heads to operate. Two backend devs now spend Fridays fixing retrieval, tuning prompts, and patching connectors — the "temporary" bot became an unstaffed platform product. Software maintenance is 80-90% of lifecycle cost historically; AI maintenance runs 2-3x the initial build in the first few years because drift + vendor deprecation never pause.

SMB lesson: Don't ask "can we build it?" Ask "can we staff 0.5-1.0 FTE for 24 months to own it — including who gets paged?" If not, buy the boring platform and own only prompts + evals.

6. What did work: narrow scope + outcome pricing (Intercom Fin + BCG 5%)

Scenario: Same SMB instead picks one high-volume, well-logged workflow — e.g., "refund-status + plan-change" tickets — and buys Intercom Fin at ~$0.99 per resolved conversation.

What happened: Intercom's official Fin meter is $0.99 per resolution (standalone available with a 50-resolution minimum, no seats required) — you pay only when Fin resolves, not per query or token. Why this pattern works for SMBs: price maps to value (resolution, not login), the team can calculate cost-per-resolution vs. human cost, and scope stays narrow enough to govern. Industry studies widely report the same shape even where exact numbers vary by survey: BCG reported only ~5% generate substantial AI value at scale with "deep and narrow" pilots outperforming sprawl, McKinsey reported high AI usage but far lower EBIT impact, and mature portfolios skew heavily buy (commodity) with selective build (differentiating). Treat those survey percentages as directional, not audited — the mechanics (narrow scope + outcome meter) are the reliable part.

Why it worked: Narrow blast radius, verifiable outcome, and a vendor whose meter matches your P&L. The team kept ownership of KB quality + weekly refusal/deflection review — 2 hours/week, not 2 FTEs.

What TCO actually means for AI (the 4 lines everyone underestimates)

The "second bill that arrives a year later" — that's TCO. Most teams price the build and forget the other three lines:

Build / purchase — the demo number everyone quotes (Copilot $30/seat, Agentforce $2/conv, or your 8-week sprint).
Operate & maintain — 2-3x the build in the first few years (drift detection, connector fixes, model migrations at 6-10 weeks each, prompt rework).
Governance & change management — 30-50% of license cost (training, comms, support) or 0.25-0.5 FTE for evals, security review, incident response.
Meter / exit reserve — the second meter (overages, usage amplification, repricing on renewal) + the lock-in reserve (what it costs to leave: data export, re-integration, M&A repricing).

Score the rubric's "24-mo TCO" row on the sum of these four lines, not the first one. Here's a worked example for a 25-seat SMB handling 5,000 tickets/month:

Line (24 months)	Buy (Copilot + Agentforce/Fin)	Build (LangGraph + RAG)
Purchase / build	25 seats × $30 × 24 = $18k + Fin 5k × $0.99 × 24 = $119k → $137k	8-week sprint ~$40k one-time → $40k
Operate & maintain	vendor-absorbed → $0	2.5x build = $100k → $100k
Governance / change mgmt	40% of license = $55k + 0.25 FTE evals $30k → $85k	0.5 FTE × $150k × 2 = $150k
Meter / exit reserve	overages + repricing reserve → $20k	lock-in: data export + re-integration → $15k
24-mo total	$242k	$305k

Read honestly: in this scenario the build runs ~26% more over 24 months and hands you the on-call burden. Swap your seat counts, ticket volumes, and rates — the shape stays. Build wins only when the workflow differentiates you and you can staff the operate/govern lines.

An SMB CTO rubric (steal this for your next review)

Score each use case 1 (buy) to 5 (build). Default to buy unless 2+ factors score 4-5.

Factor	1 = Buy	5 = Build
Differentiation	Generic across companies (follow-ups, expense coding, support deflection)	Encodes how you win (your underwriting, your triage, your data moat)
Data sensitivity	Public/internal-low-risk, vendor has certs you lack (SOC 2, EU residency)	Proprietary + regulated, must stay in your VPC with audit trail you control
Change frequency	Stable workflow, vendor ships monthly improvements you want free	Changes weekly with your product; vendor roadmap would block you
Ownership capacity	No one to carry 0.5 FTE on-call, evals, connector fixes for 24 mo	Named owner + backfill funded for upgrades, drift review, incident response
24-mo TCO	Build TCO >2x buy on honest math (build + 2-3x maintenance + infra + governance)	Build TCO within 30% of buy and exit/lock-in reserve priced (re-pricing, M&A, seat+usage dual meter)

Non-negotiables before any demo: SSO, audit logs, data residency/boundaries, admin kill-switch + human escalation, observability (resolution, hallucination, cost/interaction). If a vendor can't meet these, they're off the shortlist — no matter how good the demo.

Run pilots in real workflows, not sandboxes. Document re-evaluation triggers (e.g., "if cost/resolution >$X or active-use <Y%, we revisit in 90 days"). Revisit on schedule, not when pain forces it.

Key takeaway

Build-vs-buy for AI is really own-vs-coordinate. Own only the layer that differentiates you and you can operate well for 12-24 months. Coordinate (rent) everything else — but model the second meter, staff the 2-hour weekly governance loop, and keep an exit path.

Honest limitations

This piece has no access to your contracts or usage logs — substitute your seat counts, volumes, and rates into the math above. It doesn't cover fine-tuning economics or on-prem GPU breakeven (competitive only at sustained 5-6+ hrs/day utilization over 3-5 years). Regulated workloads (health, finance, public sector) shift strongly toward buy-the-certified-platform or build-in-VPC — get counsel early. And vendor pricing from 2025-2026 will move; re-benchmark every 6-12 months.

Open questions for your team

What AI layer do you actually want to own at 2 AM, and why?
What are your non-negotiables before you compare vendors?
How do you stop tool sprawl before 15 pilots become permanent?
What TCO line (support, governance, change management, overages) are you most likely underestimating?
When does a tactical tool become a strategic dependency — and who decides?
If you had to standardize on one AI lane tomorrow, what would drive the choice: cost, control, or speed?

The smartest build-versus-buy decision is rarely about the best feature set. It's about choosing what your organization is prepared to own well — and refusing to rent what you can't afford to lose.

Sources

Primary (official / court / regulator / analyst press release):

Salesforce, Agentforce Pricing (official) — Conversations $2/conversation; Flex Credits $500/100k; 1 action = 20 credits ($0.10); unused Flex Credits do not roll over; Flex + Conversations cannot mix in one org
Salesforce Help, Agentforce Pricing (May 19, 2025) + Flexible pricing press release (May 15, 2025)
Microsoft, Microsoft 365 Copilot enterprise pricing (official) — $30/user/mo add-on, qualifying M365 plan required
Microsoft, Copilot Studio pricing (official) — 25,000-credit packs $200/mo; credits billed per agent action/response
Microsoft Learn, License options for Microsoft 365 Copilot
ACCC, Microsoft in court for allegedly misleading millions (Oct 27, 2025) + Concise Statement PDF — consumer Personal/Family bundling case (~2.7M subscribers; $109→$159 Personal, $139→$179 Family; Classic option hidden in cancellation flow; alleged, not yet decided)
Gartner, Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (June 25, 2025)
Moffatt v. Air Canada, 2024 BCCRT 149 — BCCRT decision / CanLII; coverage: BBC, ABA Business Law Today — C$812.02 award, "separate entity" defense rejected
Builder.ai insolvency — TechCrunch (May 20, 2025) ($1B+ valuation, $450M+ raised, insolvency proceedings); nuance correction: Pragmatic Engineer — Builder.ai did not "fake AI with 700 engineers" (hybrid AI + human delivery; WSJ flagged human reliance as early as 2019)
Intercom, Fin — $0.99 per resolution (official) + Pricing + AI agent pricing comparison (Fin vs Zendesk vs Agentforce) — outcome meter (pay only on resolution), standalone Fin with 50-resolution minimum; resolution-rate and adoption-lift figures are vendor-reported

Secondary (vendor field analysis — directional, not audited benchmarks):

SearchUnify, The Real Cost of AI Agents: Maintenance, Model Drift, and Hidden Ownership Costs — 3.5 FTE / $500-700k/yr reference model at 50k interactions/mo, 6-10 week model

We Could Have Shipped on Local Models Alone

Debashish Ghosal — Sun, 06 Sep 2026 16:13:43 +0000

Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, MCP server, 7 export formats, and a bundled git rule pack. The field test report evaluated 4 models (2 local OMLX, 2 cloud OpenRouter) across 394 trajectories. The local-model totals below — Llama 3.2B at 379 candidates and Qwen 4B at 373 — come directly from that report. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

One of the easiest mistakes to make in AI engineering is asking the wrong question. We started our field test with this one:

Are small local models good enough?

By the end of the work, the more useful question looked different:

What are small local models good enough for?

And after the cloud runs came back, a third question surfaced that I did not expect to be able to answer affirmatively:

If I only had local models, could I still ship this product with the same confidence?

The answer turned out to be yes. Not because local models are as strong as the best cloud model — they are not. But because the thing that blocks shipping confidence is not the model. It is the replay engine and the safety corpora, and cloud models do not fix those. More on that below.

Two models did not even make it to the benchmark

Before the comparison could start, two candidate models were cut. Qwen3.5-4B-4bit was too slow for practical batch work — a model that takes minutes per trajectory is not one you can run repeatedly across 394 trajectories. google/gemini-2.0-flash-001 was unavailable on the tested OpenRouter tier. The endpoint simply was not there.

That is an operational detail that benchmark papers often skip. Two out of six candidates were eliminated before a single extraction ran. Model selection needs an operational screen — speed, availability, cost — before you invest in a full corpus sweep. Skip that screen and you waste real testing time on models that were never going to work in your workflow.

The field test report puts this bluntly: not every model failure is a product signal. Some are simply deployment or availability problems. The report needs to separate those cleanly so they do not pollute the product conclusions.

The post-fix local results were better than the early benchmark implied

Most of what the early benchmark called "local model failures" were not model failures at all. They were benchmark failures. The field test report puts this bluntly: before the fixes, both local models showed severe parse failure rates on curated corpora. After the fixes, parse failures dropped to near-zero on most curated corpora. The models did not change. The benchmark did.

What was broken	Before	After
Result-file isolation	Same-day reruns appended duplicate rows, contaminating counts	Reruns reset result files at start
JSON extraction	Responses with trailing commentary or a second JSON object were rejected	First balanced JSON object is now extracted
`when.context` validation	Blank context entries caused hard parse failure	Blank entries filtered before validation
Prompt clarity	Minimal example encouraged inconsistent formatting	Concrete filled example + explicit "JSON only" instruction
Curated parse reliability	Severe parse failure rates on both local models	Near-zero parse failures across most curated corpora
Raw corpus completeness	`raw/synthetic` and `raw/sibling-repos` blocked by missing timestamps	Timestamp fix applied; both corpora run end to end

That is six separate harness defects — none of them in the models — that were collectively making the local models look worse than they were. When the early benchmark said "small local models are borderline unusable," the honest reading was "small local models are borderline unusable through this broken benchmark." Fix the benchmark and the same models produce 379 and 373 candidates across 13 corpus types.

This is the part of the story that is easy to miss if you only look at the final numbers. The early narrative was not "local models have some weaknesses." It was "local models are mostly failing." That narrative was wrong, and the reason it was wrong is that the evaluation harness was failing the models, not the other way around.

After the fixes, the OMLX models became much more meaningful engineering tools. The cleanest curated comparison:

Curated corpus	Llama 3.2B	Qwen 4B
`golden`	8 pass / 2 fail	3 pass / 4 inconclusive / 3 fail
`failures/positive`	15 pass / 3 inconclusive / 12 fail	15 pass / 9 inconclusive / 6 fail
`failures/negative`	1 pass / 2 inconclusive / 5 fail	1 pass / 4 inconclusive / 5 fail
`successes`	1 pass / 16 inconclusive / 2 fail	2 pass / 8 inconclusive / 10 fail
`nearmiss`	4 pass / 4 inconclusive / 6 fail	5 pass / 5 inconclusive / 4 fail
`noisy`	1 pass / 3 inconclusive / 1 fail	4 pass / 1 inconclusive / 0 fail
`corrections`	2 pass / 2 inconclusive / 1 fail	2 pass / 1 inconclusive / 2 fail

Not a story of "small local models solved the problem." But also not a story of "small local models were worthless." The full benchmark totals reinforce that.

Model	Total rows	Total candidates	Pass	Pass %	Inconclusive	Fail
Local Llama 3.2B	393	379	72	19.0%	189	118
Local Qwen 4B	373	373	93	24.9%	209	71

379 candidates from a 3B model running locally on a laptop. 373 from a 4B model. Those are large enough run volumes to support real engineering decisions, not just gut feelings.

But the raw pass rate only tells part of the story. To see how local models actually stack up, you have to put them next to the cloud baselines from the same field test:

Model	Type	Candidates	Pass	Pass %
Local Llama 3.2B	local OMLX	379	72	19.0%
Local Qwen 4B	local OMLX	373	93	24.9%
Cloud GPT-4o-mini	cloud OpenRouter	394	77	19.5%
Cloud Llama 3.1 8B	cloud OpenRouter	392	123	31.4%

Read that table carefully. Local Qwen 4B — a 4B model running on a laptop — outperformed openai/gpt-4o-mini on total passes (93 vs 77) and on raw pass rate (24.9% vs 19.5%). Local Llama 3.2B beat GPT-4o-mini on the golden corpus (8 pass vs 6 pass). This is not "small local models earned a place as a regression tool." This is "a 4B local model beat a paid cloud baseline head-to-head on total pass count."

The cloud Llama 3.1 8B is still the strongest model in the field test, and the safety corpora remain weak across the board. But the premise that local models are clearly inferior to cheap cloud baselines is not supported by this data. On some corpora, the local models are better.

And here is the point that matters most for anyone deciding whether to ship: the cloud runs did not move the shipping decision. The field test report says it plainly — "better models improve extraction quality, but they do not erase the product's safety and replay problems." The strongest cloud model still fails on successes, failures/negative, and nearmiss. The replay engine is still the named #1 bottleneck. The safety problem lives in the product, not in the model. So if your shipping gate is "do we have enough confidence in the safety and replay layer," cloud models do not get you over that line any more than local models do. They get you a cleaner extraction ceiling, not a safer product.

That is why the honest answer to "could I ship with local models only?" is yes. You would ship with the same unresolved safety and replay gaps either way. The cloud model does not close them.

What the pass counts are actually measuring

There is one more thing the headline numbers do not show, and it matters for how you read every table in this article. The pass / inconclusive / fail verdict comes from the replay engine, not from a human judge. And the field test report names that replay engine as the single biggest bottleneck in the whole system:

Replay and matcher calibration is now the highest-value engineering target.

The matcher uses substring and token-overlap heuristics. When it cannot decide whether a rule fires against a reference trajectory, it returns inconclusive — not fail. That means a large slice of the inconclusive bucket is not a model failure. It is a replay-engine limitation. A specific, reasonable rule can land in inconclusive purely because the matcher is too weak to verify it.

So the pass counts in the tables above are closer to a floor on model usefulness than a ceiling. The true model-only pass rate sits somewhere between the reported pass count and pass-plus-inconclusive. For Qwen 4B, that range is 24.9%–81.0% of candidates. For Llama 3.2B, it is 19.0%–68.9%.

That is a wide band, and it is not honest to claim all inconclusives as hidden passes. Some of them are real model weaknesses — over-broad triggers like "when a command fails" instead of "when git push fails with non-fast-forward", and over-triggering on the successes corpus where the model should have stayed silent. Those are model problems, not matcher problems.

But it is also not honest to present the raw pass count as the model's ceiling. The report is explicit: the replay engine is the named bottleneck, and better models did not erase the inconclusive problem. So when you read that Local Qwen 4B passed 93 of 373 candidates, the honest interpretation is "at least 93, and some unknown fraction of the 209 inconclusives are also correct extractions that the matcher could not verify."

The raw corpus split: Llama wins breadth, Qwen wins depth

The curated corpora showed Llama as the better default. The raw corpora told a more nuanced story.

Raw corpus	Local Llama 3.2B	Local Qwen 4B
`raw/opencode`	13P / 2I / 9F	13P / 7I / 5F
`raw/synthetic`	18P / 101I / 26F	36P / 87I / 22F
`raw/ci`	8P / 49I / 52F	9P / 90I / 11F
`raw/sibling-repos`	0P / 9I / 1F	0P / 9I / 1F
`raw/corrections`	2P / 2I / 1F	3P / 1I / 1F
`raw/cross-session`	2P / 1I / 2F	1P / 2I / 2F

Look at raw/synthetic: Qwen produced 36 pass vs Llama's 18 pass — double the extracted rules. But Llama had 26 hard fails vs Qwen's 22. And both produced a large number of inconclusives (101 vs 87).

Then raw/ci: Llama produced 8 pass / 49 inconclusive / 52 fail. Qwen produced 9 pass / 90 inconclusive / 11 fail. Qwen had fewer hard failures but far more inconclusives. That is a different quality profile — Qwen is more cautious, Llama is more decisive but also more wrong.

This is why raw pass count alone is not enough to pick a default local model. The pattern of pass / inconclusive / fail matters. High pass + high fail means decisive but noisy — you get more signal but more garbage. Moderate pass + high inconclusive + low fail means cautious — less garbage but less signal. For a local regression tool, you want the decisive model — false positives are cheaper because you can review them. For a trust benchmark, you want the cautious one — false positives are expensive because they can be promoted.

What local models were actually good for

After the fixes, the local models were useful for:

broad internal corpus sweeps
low-cost repeated reruns
prompt and parser regression checks
local smoke tests for the extraction loop
cheap model-to-model comparisons

That is already a meaningful role. And once you look at the local-vs-local comparison in a practical way, the split becomes clearer:

Question	Local Llama 3.2B	Local Qwen 4B
Better practical local default?	Yes	No
Better secondary comparison model?	Acceptable	Yes
Better on raw synthetic breadth?	Weaker	Stronger
Better operational feel for reruns?	Yes	Slower / heavier

This is why I do not think raw pass count alone is enough to pick the default local model. Engineering usefulness includes speed, rerun friction, and how easy a model is to keep in the loop.

Where local models were still clearly limited

The same results also make the limits obvious. The weakest areas remained the same ones that matter most for trust: successes, failures/negative, and nearmiss.

That means the local models are still not strong enough to be the only evidence behind public quality claims or autonomous promotion decisions. But neither are the cloud models — the report's central finding is that even meta-llama/llama-3.1-8b-instruct does not solve the safety corpora. So the honest caveat is not "local models are only good for regression." It is "no model in this field test is sufficient for trust on its own, and the safety problem lives in the product, not in the model." Local models are competitive with cheap cloud baselines on extraction quality; what they all still lack together is safe rule selection and replay trust.

An observation that surprised me

I expected the larger local model (Qwen 4B) to clearly outperform the smaller one (Llama 3.2B). It did not. On the curated corpora, Llama 3.2B actually produced more golden passes (8 vs 3) and was more decisive on failures/positive — 15 pass / 3 inconclusive / 12 fail, versus Qwen's 15 pass / 9 inconclusive / 6 fail. By raw pass-to-fail ratio, Qwen actually wins that corpus (15:6 ≈ 2.5 vs Llama's 15:12 ≈ 1.25). But Llama produced fewer inconclusives, which means it committed to a verdict more often. Qwen caught up on breadth — especially on raw/synthetic where it doubled Llama's pass count — but it did not translate that into better curated precision.

That suggests something I did not expect: for small local models, raw capability and extraction precision may not correlate the way you'd assume. A 3B model with good instruction-following can outperform a 4B model on specific tasks, even when the 4B model processes more data. The bottleneck at this scale is not parameters. It is instruction adherence and format stability.

What I learned from this

First impressions from a broken benchmark are worse than no benchmark at all. The early data made local models look borderline unusable. After the fixes, they produced hundreds of candidates across 13 corpus types. The models did not change. The benchmark did. If we had trusted the early data, we would have written off local models entirely and spent money on cloud APIs for every regression run.

Operational screening belongs before benchmarking, not after. We lost time on Qwen3.5-4B (too slow) and Gemini Flash (unavailable). Both failures were predictable with a 5-minute operational check. A full corpus sweep across 394 trajectories is expensive. Do the cheap screening first.

Pass count is not the same as decision quality. Qwen 4B had more total passes (93 vs 72) but also more inconclusives (209 vs 189). Llama 3.2B had more hard fails (118 vs 71). Which model is "better" depends on what you are using it for — a regression tool wants decisiveness, a trust benchmark wants caution.

Role separation beats model replacement. The temptation when a model is weak is to replace it. The better move is to assign it a role it can handle. But note what the data actually showed: a 4B local model beat a paid cloud baseline on total pass count. So "local for iteration, cloud for quality" is true, but it is not true because local models are weak. It is true because cloud models give you a cleaner quality ceiling and wider model choice, not because local models cannot compete on extraction. The cheapest useful cloud model (meta-llama/llama-3.1-8b-instruct) still beat every local model tested — but gpt-4o-mini did not.

Open questions

Would a 7B or 8B local model close the gap on successes and nearmiss, or does the safety problem live in the product regardless of model size? The cloud results in the next article suggest the latter.
Is the Llama 3.2B advantage on golden stable across reruns, or is it within noise? We ran each model once per corpus. More reruns would tell us whether 8P/2F is reliable or lucky.
The raw/synthetic split (Qwen 36P vs Llama 18P) is large enough to matter. Is Qwen genuinely better at breadth, or is it over-triggering on raw data? The inconclusive count (87 vs 101) suggests Qwen is extracting more but also hedging more.

The broader lesson

Local models were not good enough to end the evaluation conversation. But they were much better than their first benchmark impression — good enough, in fact, to beat a paid cloud baseline on total pass count, and good enough to ship the product on. The lesson is not that local models are a cheap fallback. It is that the gap between "cheap local" and "cheap cloud" is smaller than the benchmark infrastructure made it look, and the thing that blocks shipping confidence — the replay engine and the safety corpora — is a product problem that cloud models do not fix. If I only had local models, I could have shipped CauterRule with the same degree of confidence the cloud models gave. Which is to say: the same confidence, and the same unresolved gaps.

CauterRule v0.1.0 is released. The full model-by-model breakdown — including the curated corpus comparison table, raw corpus results, and model ranking — is in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes

When Your Benchmark Finally Tells the Truth

Debashish Ghosal — Sun, 06 Sep 2026 05:45:44 +0000

Update — v0.1.0 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract a candidate rule from a failure trajectory, replay-test it against history, promote only what survives. pip install cauterule gets you the full CLI, MCP server, export to 7 formats, and a bundled git rule pack. The field test report evaluates 4 models across 394 trajectories and is the source for every number in this article. Release notes · Changelog

CauterRule is an open-source sidecar that turns repeated agent failures into standing rules. It extracts candidate lessons from trajectories, replay-tests them against history, and is designed to promote only rules that look reusable and safe.

When we started field testing CauterRule, the early numbers looked bad enough that the story almost wrote itself. Local OMLX models looked weak. Parse failures were common. Some corpus slices would not run fully. It was tempting to conclude that the model layer was the main problem.

That was not the right conclusion.

The better conclusion, backed by the rerun data, was that the benchmark harness itself was manufacturing a meaningful part of the failure.

The before/after numbers were too large to ignore

The clearest evidence came from rerunning the same benchmark surfaces after parser, prompt, result-reset, and timestamp fixes.

Signal	Before fixes	After fixes
Local Llama `golden` parsed candidates	3 / 11 rows	10 / 10 rows
Local Qwen `golden` parsed candidates	3 / 10 rows	10 / 10 rows
Raw synthetic processed rows	124 / 145	145 / 145
Raw sibling-repo processed rows	0 / 10	10 / 10

Those are not cosmetic improvements. A benchmark that can only parse 3 of 10 or 3 of 11 golden rows is measuring something very different from a benchmark that cleanly parses 10 of 10.

To put that in perspective: the early benchmark was throwing away 73% of its own signal on the golden corpus. It was not measuring model quality. It was measuring parser fragility and then labeling the result as "model quality." If we had shipped a product roadmap based on that data, we would have optimized the wrong layer for months.

And it was not just golden. The raw synthetic corpus went from 124/145 to 145/145 processed rows. The raw sibling-repo corpus went from 0/10 to 10/10 — zero usable data to fully usable data. That is not a gradient improvement. That is the difference between a corpus you can reason about and a corpus that produces no signal at all.

What was actually broken

The field test exposed four harness failures. Each one looked like a model problem but was not.

Problem	What it looked like	What it actually was
duplicate rows	model inconsistency	same-day result append contamination
parse failure on valid-looking output	weak model	parser grabbed too much text
context validation failure	bad extraction	blank `context` items rejected too aggressively
blocked raw runs	model/runtime failure	missing input timestamps

This distinction matters because each of those failure modes points to a different fix path. If you treat all of them as "the model is bad," you will optimize the wrong layer.

A golden result set with 11 rows for a 10-trajectory corpus is not evidence of model instability. It is evidence that the reporting path allowed rerun contamination. Raw corpus failures caused by missing timestamps are not evidence that the LLM could not solve the task. They are evidence that the input contract was incomplete.

Each misdiagnosis has a real cost. Misdiagnose duplicate rows as model instability and you spend weeks tuning temperature for a file-append bug. Misdiagnose parse failure as a weak model and you upgrade to a more expensive model for a greedy JSON slicer. Misdiagnose blocked raw runs as runtime failure and you debug the execution environment for missing timestamps in the input data.

The cost of misdiagnosis is not just wasted time. It is that you build confidence in the wrong mental model of your own system.

The fixes changed the meaning of the benchmark

We made five specific changes:

reset results.jsonl and summary.json at run start
extract the first balanced JSON object instead of slicing from first { to last }
filter blank context items before validation
strengthen the prompt with a concrete filled JSON example
repair missing timestamps in blocked raw corpus assets

The report describes the transition in one sentence: the fixes moved the benchmark from "mostly noise" to "mostly signal." That is the single most important shift in the entire field test. Not because the numbers improved — but because the numbers started meaning something.

Here is a concrete example. Before the fixes, the local Llama model on golden produced 3 parseable rows out of 11. After the fixes, it produced 10 out of 10, and the result was: 8 pass, 2 fail. That 8P/2F is a real signal about the product's extraction quality. The earlier 3/11 was not a signal about anything except the parser.

The raw corpora told the same story at a different scale

The curated corpora are the cleanest signal, but the raw corpora are where volume lives. And the raw data told the same story at much larger scale.

Raw corpus	Before fixes	After fixes
`raw/synthetic`	124 / 145 rows processed	145 / 145
`raw/sibling-repos`	0 / 10 rows processed	10 / 10
`raw/opencode`	partially blocked	fully runnable
`raw/ci`	partially blocked	fully runnable
`raw/corrections`	partially blocked	fully runnable
`raw/cross-session`	partially blocked	fully runnable

Six raw corpora. Before the fixes, two were completely blocked and four were partially blocked. After the fixes, all six ran end to end. That is not "we improved some metrics." That is "we went from a benchmark that could not process its own data to one that could."

The report makes an observation about this that I think is underappreciated: a working runner plus a stable corpus is a compounding asset. Each future iteration starts from the accumulated knowledge of the last one, instead of rediscovering the same benchmark setup problems every time. That is the difference between a benchmark that gets better with use and one that decays.

The transition from plumbing to judgment

There is a moment in every field test where the character of the work changes. For CauterRule, that moment was the parser fix.

Before the fix, the work was plumbing: file contamination, JSON slicing, blank context items, missing timestamps. Each problem had a clear cause and a clear fix. The fixes were satisfying — numbers jumped, blocked corpora opened up, the benchmark started working.

After the fix, the work was judgment: is the trigger too broad? Is the directive specific enough? Does the replay logic correctly interpret the candidate? Does the product avoid extracting from cases where it should remain silent?

Plumbing fixes produce dramatic gains. Judgment-quality improvements require more careful scoring, better matcher design, improved negative-case handling, and human-calibrated evaluation. The project is leaving the easy gains behind and entering the more important work.

That is a healthy transition. But it is also a warning: the next set of improvements will not come from a one-line parser fix. They will come from harder, slower, more deliberate work on the product's decision quality.

What the benchmark started saying once it became honest

Once the harness stopped dominating the results, three conclusions became stable:

the product could extract many candidate rules from positive or obvious failure trajectories
the product was still much weaker on successes, failures/negative, and nearmiss
the main unresolved problem had shifted from formatting to judgment quality

The report frames this as the project's biggest win — not a benchmark score, but the moment when "the remaining weak spots now point to the product, not to the test rig." Once that happened, the real product problem became visible for the first time.

And here is the quote from the report that stuck with me most:

A system that never produces useful output is obviously immature. A system that often produces useful-looking output, but still has weak safety boundaries, is more subtle and more dangerous. CauterRule is clearly in the second category.

The benchmark fixes did not make the product look good. They made the product look honest. And honest meant: good at the easy half, visibly weak on the hard half, and dangerous in exactly the way that matters for trust.

The engineering foundation held while the benchmark broke

One detail from the report that deserves more attention: while the benchmark harness was manufacturing failure, the deterministic foundation was solid.

844+ deterministic tests passing
104/104 Docker tests passing
export/import validation passing
redaction validation passing
gold-family validation passing
performance baselines within threshold

It would have been easy to dismiss the whole system as broken. It was not. The application core — the serialization, the CLI, the Docker environment, the export formats, the redaction engine — was reliable. The noise was concentrated in the benchmark layer, not the product layer. If the substrate had also been shaky, the field test would have been unrecoverable.

What I learned from this

A few things that I think generalize beyond CauterRule:

Your benchmark is a system too. It has its own bugs, its own input contracts, its own failure modes. If you do not test your benchmark, you are trusting an untested system to judge a tested one.

Signal loss is invisible until you measure it. We did not know we were losing 73% of golden signal until we fixed the parser and saw the numbers jump. Before that, the lost signal looked like "the model is bad." The failure was silent.

Failure bucketing changes your roadmap. If you lump all failures into "model problem," you optimize the model. If you split them into model / parser / corpus / scoring, you discover that most of your early failures are not model problems at all — and your roadmap shifts.

A plausible result from a broken benchmark is worse than no result. No result forces you to investigate. A plausible result from a broken harness gives you false confidence. The 3/11 golden rows were plausible enough to interpret. They were wrong enough to mislead.

Fix the benchmark before you optimize the model. If you optimize the model while the benchmark is broken, you are fitting to noise. Every model improvement you measure through a broken harness is suspect.

Raw breadth without replay clarity is not yet evidence. The raw corpora now all run end to end, but broad sets like raw/ci still generate many inconclusive results. The product can process them, but the benchmark does not yet turn that processing into consistently actionable quality signals.

Why this matters beyond one project

The broader lesson is not "always trust local models more." It is narrower and more useful:

never let your benchmark failure masquerade as a model failure.

If you are building an agent system, memory system, or any kind of model-evaluated workflow, split failures into buckets before you interpret them:

model problem
parser/tool problem
corpus/input problem
scoring/replay problem

If you do not, the benchmark can become the noisiest component in the system while still pretending to be the most authoritative one.

That is the part I would most want other engineers to argue with, test, or improve on. Not because it is abstract, but because once the benchmark becomes believable, the next round of product work becomes much more honest too.

CauterRule v0.1.0 is released. The full field test report — including the parser-fix before/after data, 844+ deterministic tests, 104 Docker tests, and model-by-model breakdowns — lives here. The repo is public. Install with pip install cauterule. Changelog · Release notes

I Thought Role Separation Would Fix the Optimizer. It Didn't.

Debashish Ghosal — Sat, 05 Sep 2026 16:03:24 +0000

Previously: 9 Bugs That All Looked Like a Working System · I Built an AI That Rewrites Its Own Prompts · The Edit That Fixed 4 Tasks and Broke 1 · I Let an LLM Rewrite Its Own Prompt. The Real Win Was the Gate That Rejected It. · I Tried 4 Models to Save My Self-Improving Agent. All 4 Failed.

AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.
Repo: github.com/deghosal-2026/agent-self-edit/tree/v0.3.0
Release notes: docs/release/v0.3.0/release-notes.md
Field test report: docs/field-test/v0.3.0/FIELD_TEST_REPORT.md

Role separation felt like the obvious upgrade.

Stop forcing one model to do everything. Let one model execute tasks, another analyze failures, another judge outputs when needed. Use the right model for the right job.

That sounds so reasonable it almost feels boring.

So when v0.3.0 shipped a separated-role runner, I expected the first result to be at least directionally better.

Instead, the first real run produced zero proposals.

That stung a little. It also taught me more than a clean win probably would have.

Why I Thought Role Separation Would Help

The single-model story already had a clear problem.

The local 4B model was cheap and mechanically useful, but weak as an analyzer. The stronger cloud Mistral model produced the first small positive signal, but still got stuck making narrow, local edits.

So the natural next move was: keep a decent executor, use a stronger analyzer, maybe keep a separate judge for tasks that need it.

That idea is attractive because it matches how people talk about agent systems in general. Planner here. Critic there. Specialist over there. Split the roles and improve the whole.

I bought that story too.

What We Shipped in v0.3.0

v0.3.0 added separated-role support so executor, analyzer, and judge can come from different provider configs with fallback.

That was a real feature, not a paper design.

The first separated-role classification run used:

executor: qwen/qwen3-30b-a3b-instruct-2507
analyzer: mistralai/mistral-small-3.2-24b-instruct
iterations: 3
held-out sample: 5
promotion sample: 10

Top-line result:

baseline held-out: 60.0%
final held-out: 60.0%
promotions: 0
proposals: 0 in every iteration

The zero is the whole story.

Zero Proposals Is a Different Kind of Failure

The local 4B run failed one way: weak edits.

The single-model Mistral run failed another way: small positive movement, still not enough to promote.

The separated-role run failed earlier.

It did not even generate proposals.

That distinction matters.

The artifacts made the failure point pretty clear:

analysis.json existed in every iteration
prompt-a.md existed
accuracy.json existed
prompt-b.md did not exist
ab-comparison.json did not exist
prompt-after.md did not exist
no error.txt files were written

That means the run started normally, collected failures, entered the analyzer path, produced an analysis artifact, and then stopped because there were no proposals to continue with.

This was not a weak-promotion problem. It was a no-proposal problem.

The Part That Surprised Me Most

The separated-role run did not fail because the executor was obviously broken. Baseline was plausible at 60.0%.

It did not fail because the runner crashed. The directories and artifacts were there.

It failed because the analyzer produced nothing.

That changed how I thought about role separation.

I had treated it like a model-quality upgrade. Better model in the analyzer slot, problem improves.

The run pushed back on that assumption hard.

The analyzer doesn't reason over abstract task IDs. It reasons over the concrete failure traces coming from the executor. So changing the executor changes the analyzer's input surface. Different failures, different clustering, different patterns, different chance of seeing something edit-worthy.

That means role separation is not just a routing improvement. It is a learning-surface change.

Why a Stronger Analyzer Wasn't Enough

This is the part I think is easiest to miss.

The analyzer model in the separated-role run was not worse than the one that produced weak positive signal in the single-model cloud run. In fact it was the same stronger analyzer candidate: mistralai/mistral-small-3.2-24b-instruct.

So why did one setup give small positive edits and the other give none?

The best explanation from the evidence is that the analyzer is sensitive to the shape of the failures it sees.

Three possibilities seem plausible:

executor-output shift: the 30B executor produced a different kind of mistake distribution, one the analyzer did not turn into proposals
smoke-size underexposure: with only 5 held-out and 10 promotion samples, the analyzer may not have seen enough recurring structure
staged-analyzer brittleness: the pipeline may still be too sensitive to exact failure phrasing coming from a different executor model

I don't think the lesson is "role separation is bad."

I think the lesson is that role separation is not a free win.

What This Taught Me About Agent Architecture

Agent diagrams make roles look cleaner than they are.

Planner, executor, critic, judge. Nice boxes. Nice arrows.

In real systems, those roles are coupled through the artifacts they produce for each other. Change one box and you change the texture of the evidence flowing into the next one.

That coupling is exactly what this run exposed.

The analyzer was not just "a better model waiting to help." It was a model reacting to a different failure landscape, and in this case that landscape was apparently much less fertile.

That is a more interesting architectural lesson than a simple benchmark gain.

What Worked Anyway

Even though the run produced zero proposals, I don't see it as a wasted experiment.

The separated-role runner itself worked.

The unique output directory logic worked.

The baseline wasn't nonsensical.

And, more importantly, the run ruled out an overly simple story I was half ready to believe: "just keep the executor cheap and swap in a stronger analyzer."

The project is better off not believing that story anymore.

What I Learned

First, role separation changes more than cost and quality. It changes what the system can learn from.

Second, stronger models are not enough if the surrounding evidence surface changes in ways the analyzer can't use well.

Third, zero-proposal runs deserve as much attention as weak-positive runs. They are different failure classes. If you lump them together under "no promotion," you throw away the most useful part of the diagnosis.

And there is another lesson here that I did not expect to matter this much: the artifact shape tells you almost as much as the numeric result. analysis.json without prompt-b.md or ab-comparison.json is a very different story from a run that reaches A/B and then fails confidence. Those should be treated as different architectural events, not just different rows in the same score table.

Why Developers Should Care

If you're building multi-model agent systems, I think this is the important warning label: specialization creates interaction effects.

Swapping a better model into one role does not just improve that role. It can change the whole system's learning behavior, because the roles are coupled through real outputs, not abstract interfaces.

That is the kind of thing you only really appreciate once a run fails in a new way.

I shipped role separation expecting an upgrade.

The first real run gave me a negative result instead.

I trust that result enough now to say this without hedging: that was still progress.

If you are splitting roles across models in your own system, how are you checking whether the roles are actually helping each other instead of just making the architecture look smarter? And when a stronger analyzer produces no proposals at all, do you treat that as a model problem, a dataset problem, or a sign that the executor changed the learning surface too much?

I Thought the Optimizer Was the Product. I Was Wrong. The Gate Was.

Debashish Ghosal — Fri, 04 Sep 2026 20:50:20 +0000

AgentSelfEdit is an open-source sidecar that rewrites its own system prompt from execution feedback. It A/B tests edits and promotes only statistically-proven winners.
Repo: github.com/deghosal-2026/agent-self-edit/tree/v0.3.0
Release notes: docs/release/v0.3.0/release-notes.md
Field test report: docs/field-test/v0.3.0/FIELD_TEST_REPORT.md

When I started this project, I thought the star of the show would be the optimizer.

The part that reads failures, rewrites the prompt, and gets smarter over time.

That is still the part people notice first.

By the time v0.3.0 shipped, it was no longer the part I trusted most.

That honor went to the gate.

A Self-Editing System Can Quietly Destroy Its Own Baseline

That is the problem in one sentence.

If you let a system change the prompt that shapes its own future behavior, then a bad edit is not just a one-off bug. It becomes part of the baseline. It gets inherited by every later run.

That is why I stopped thinking about the gate as a feature. It is the blast door.

If the blast door is weak, the rest of the system becomes dangerous no matter how clever the optimizer looks.

What the Gate Actually Does in v0.3.0

By v0.3.0, the promotion gate had seven deterministic checks:

sample floor
effect size
confidence threshold
frozen sections
edit distance
drift detection
Oracle Drift Guard

No LLM decides whether an edit gets promoted.

That was a deliberate line in the sand. If an LLM judges its own edits, the whole system starts grading its own homework. I wanted the final promotion boundary to be code, not vibes.

The Numbers That Made Me Trust It

The best numbers in v0.3.0 were not about improvement. They were about restraint.

0% false positives across the shipped field-test evidence
8/8 adversarial edits blocked
0 false negatives observed in adversarial validation
no bad promotions in local synthetic, cloud synthetic, or Docker-backed flows

If a self-editing system is going to fail while it learns, this is how I want it to fail.

Conservatively. Verbosely. With receipts.

Oracle Drift Guard Ended Up Mattering More Than I Expected

This was one of the quieter v0.3.0 features, but I think it is one of the most important.

Oracle Drift Guard exists for a nasty kind of failure where the optimizer, the scorer, and the benchmark all share the same wrong idea of success.

That is more dangerous than an obvious bug because the whole system can look internally consistent while still drifting in the wrong direction.

The prompt starts optimizing for the wrong thing. The scorer rewards it. The corpus keeps reinforcing it. Everything lines up and says "looks good."

That is not learning. That is coordinated self-deception.

I think more AI systems need explicit defenses against that class of failure.

The Safety Story Was Bigger Than Statistics

People hear "gate" and think p-values. Fair enough. The confidence threshold matters.

But the real safety story in v0.3.0 was wider than that.

materialize_candidate_prompt() replaced raw str.replace(), which means missing old_text fails loudly instead of pretending an edit happened.

Frozen sections draw a hard line around prompt content the analyzer should never touch automatically.

Edit distance limits keep the system from calling a wholesale rewrite "one edit."

Rollback with lineage means a promoted change is not just stored. It is traceable and reversible.

That is what I mean when I say the gate became the product boundary. It is not one check. It is a safety stack.

The Optimizer Still Didn't Win, and That Made the Gate More Valuable

The optimizer still did not produce a promotable edit in v0.3.0.

The best cloud candidate from mistralai/mistral-small-3.2-24b-instruct showed weak positive movement, but nowhere near enough confidence.

The local 4B analyzer mostly produced null edits.

The first separated-role run produced zero proposals.

That can sound disappointing if your only success metric is promotion.

It looks very different if your first success metric is "do not let the system quietly corrupt its own baseline while the optimizer is still immature."

On that metric, the gate absolutely earned its keep.

The Surprise Was How Hard Safe Failure Actually Is

I used to think safety would be the boring part. Add thresholds. Add checks. Move on.

That was naive.

The hard part was not inventing names like confidence or drift. The hard part was making sure the system measured the right thing, at the right point in the flow, against the right baseline prompt, with enough attached evidence that a rejection meant something concrete.

Once a system edits itself, every shortcut gets more expensive.

That was one of the main lessons of v0.3.0.

What I Learned

The optimizer is replaceable. The gate is not. If I swap in a stronger analyzer tomorrow, the gate still has to be the thing I trust more than the model.

I also learned that statistical rigor is necessary but not sufficient. P-values and effect sizes matter, but so do provenance, materialization correctness, adversarial testing, rollback, and drift checks. Safety is a stack, not a number.

I also came away thinking the gate has become a better diagnostic tool than I expected. It is no longer just a blocker. In v0.3.0, it is separating different failure classes: null edits, locally plausible net-zero edits, and weak-positive but underpowered edits. That makes the gate more useful than a binary "promote/reject" label suggests.

And I learned the hardest thing emotionally: a system that refuses to move can still be healthy. Sometimes a long string of rejections is not proof that the project is broken. It is proof that the project still has a boundary.

Why Developers Should Care

If you are building any loop where an AI system can revise its own behavior, ask this before you ask anything else:

what stops the system from rewarding its own mistakes?

If the answer is basically "another LLM will tell us if it looks good," you do not have a safety story yet.

I started this project thinking the smartest part would matter most.

v0.3.0 taught me the opposite. The part that says no is the reason the rest of the system gets to keep trying.

If you were building a self-modifying system, how conservative would you make the safety boundary before the product became useless? Would you rather ship a system that almost never promotes, or one that moves faster but occasionally corrupts its own baseline? Where do you draw that line?