Debashish Ghosal

Posted on Sep 13 AI-assisted

My Extraction Score Was 0.08 and the Model Was Innocent: Rebuilding the Ruler

#ai #llm #measurement #agents

Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

The last article ended on an open question: at what point does a rule that is right-but-reworded deserve credit, and who decides? For v0.3.1 we built the instrument that question deserves — a direct extraction-accuracy metric, scoring the extracted rule against the expected_rule ground truth the corpus already carried but we had never used.

The first reading was 0.08.

We stared at it the way we stared at every low number in this project's history: reach for the model. Then we looked at the ruler. The ruler was broken — in the same way we had accused the replay gate of being broken, which makes this article feel a little like we are grading our own homework on a rubric we just wrote. It is also the most satisfying fix of the release cycle.

What we built

Four of the 40 corpora carry an expected_rule field — the ground-truth rule a trajectory should produce: golden (60, backfilled with the authored scenarios in #735), failures/positive (23 trajectories carry one), reference-expansion (303), and its paraphrase-diversity slice (15). For every extracted candidate we now compute, per corpus:

Metric	Comparator	What it grades
`token_f1`	literal token F1, extracted vs `expected_rule`	surface form
`semantic_f1`	MiniLM cosine on the full rule	meaning, whole rule
`directive_f1`	literal token F1 on the `do` half only	the directive phrase
`agreement`	trigger-only semantic match (after J6)	the headline

The metric's job is narrow and it matters: it answers "did the model extract the right rule?" independently of replay, which answers "can we verify it against history?" One number, two questions — the split from the previous article, finally instrumented.

The first reading: 0.08, and why it lied

The first version gated agreement on the directive's literal token F1. A directive like "pull latest changes before pushing" is a short phrase. A correct rewording — "run a rebase or fetch-merge first" — shares almost no tokens with it and scores ~0.22. Gate a composite on that, and the headline agreement on golden collapses to 0.08–0.10.

Read that number naively and the story is "the model is bad at extracting directives." Read the four sub-metrics side by side and the story is different: the trigger — the part of the rule that identifies the failure — was matching at semantic similarity of 0.6+. The model knew when. The gate was punishing it for rephrasing what to do.

That is the replay sin from the last article, committed by the new metric: a lexical comparator wearing a validation costume. We had diagnosed the disease in one half of the pipeline and then prescribed the same drug for the other half.

The fix (J6): agreement is trigger-only, and the parts are reported, not fused

Two changes:

agreement is now the trigger-only semantic match. The directive stops gating the headline; it is reported alongside as directive_f1 so nobody loses the signal — it just stops deciding it.
Every sub-metric is reported next to the composite. The spread is the finding. A composite that hides its parts cannot be diagnosed; a set of parts can.

The regression test that pins this is test_agreement_is_trigger_only_directive_not_gated in tests/measurement/test_extraction_accuracy.py. A metric, once it gates a release, is load-bearing code. It gets tests like anything else.

Two smaller honesty fixes in the same commit

J10 — 0.0 is a value; null is a different value. The first version of the metric emitted extraction_f1: 0.0 for corpora with no expected_rule at all (raw/ci). But 0.0 means measured, and the answer was zero — which on a corpus with no ground truth is a lie. It is indistinguishable from total failure. to_dict() now returns null when n == 0, and the runner emits None. The report can now tell no ground truth from ground truth said zero.

J13 — label the rate for what the corpus actually is. raw/opencode was reporting a "false_accept_rate" of 0.68 / 0.72 across the two models. It is a promotion corpus, not a rejection corpus — that number is the acceptance rate, i.e. working as intended. Calling it "false accept" makes a pass rate read like a safety violation in every summary file we shipped. Rejection corpora (nearmiss, adversarial/*) report false_accept_rate; silence and extraction corpora report acceptance_rate.

Neither fix changes a model or a threshold. Both change whether the report means what it says.

The data

Cloud models, v0.3.1, from the committed artifacts:

Corpus (n)	Model	token_f1	semantic_f1	directive_f1	token_agree	agreement
golden (60)	gpt-4o-mini	0.419	0.611	0.205	0.250	0.783
golden (60)	llama-3.1-8b	0.484	0.670	0.220	0.367	0.850
failures/positive (23)	gpt-4o-mini	0.527	0.664	0.302	0.304	0.739
failures/positive (23)	llama-3.1-8b	0.556	0.652	0.294	0.478	0.783
reference-expansion (303)	gpt-4o-mini	0.608	0.734	0.244	0.548	0.917
reference-expansion (303)	llama-3.1-8b	0.654	0.749	0.205	0.630	0.917
paraphrase-diversity (15)	gpt-4o-mini	0.512	0.718	0.311	0.267	0.933
paraphrase-diversity (15)	llama-3.1-8b	0.509	0.711	0.388	0.200	0.933

Look at the spread, not the single column:

agreement 0.74–0.93 — the trigger the model extracted is semantically the right trigger most of the time.
token_f1 0.42–0.65 — in literal wording, the same rule looks mediocre.
directive_f1 0.20–0.39 — the short "do" phrase is where literal comparison dies, which is exactly why it should not gate the headline.

Same model, same corpus, three comparators, three very different stories. The model extracts the right rule in different words. That was the v0.3.0 open question (plan Q2) — "is the extraction quality gap the model, or the matcher proxy?" — and this is the empirical answer: the proxy.

What worked

The ground truth was already in the corpus. expected_rule needed no new labeling — it was being parsed out and dropped at load. Picking it up is the cheapest high-signal metric we have, and we waited a full release to pick it up.
Reporting the parts, not just the composite. The agreement/token/directive spread is the entire finding. A single fused number would have been another 0.5-something to argue about.
null as a first-class value. Distinguishing unmeasurable from zero is a one-line change that prevents an entire class of misread reports.
The metric got regression tests the day it gated a release decision. test_extraction_accuracy.py pins the trigger-only definition; J6's behavior is a test, not a comment.

What didn't work

The first version of the metric. It was the replay bug reborn: a literal-token gate on the shortest, most freely-reworded part of the rule. A new metric ships with the same assumptions as the old one until someone forces it to look at its own sub-scores.
0.0 for "not measured." It read as total failure in summary.json and would have polluted any downstream average silently.
Borrowed labels. false_accept_rate on a promotion corpus is a rate with the wrong sign of connotation. Labels are interpretation; interpret the corpus, not the habit.

Questions we still can't answer

Is trigger-only agreement too generous? If the trigger is right and the directive is nonsense, agreement says the extraction was good. Is a right-when/wrong-do rule "extracted correctly"? We report directive_f1 but do not act on it. That is a decision we have deferred, not solved.
The small-n problem. failures/positive has 23 ground-truth trajectories and paraphrase-diversity 15. Agreement at n=15 is a direction, not a claim. The Wilson CIs we now publish for pass rates do not yet cover these extraction numbers.
Does reference-expansion's 0.917 overfit to our authored references? The 303 trajectories are paraphrase-expansions of our own reference pool. High agreement there partly means "the model reproduces our phrasings." How much of 0.917 transfers to rules nobody in the pool wrote? We do not yet know.

What I learned

A metric is untested code until it is tested. We wrote a measurement instrument with the same confidence we would write a load-bearing feature — and it shipped a lying first version. The fix was structural (trigger-only, parts reported) but the lesson is procedural: the ruler gets tests before the reading gets believed.

null is a value, and so is a label. 0.0 vs null and false_accept_rate vs acceptance_rate are not cosmetics. A report that cannot distinguish unmeasured from zero, or working as intended from safety violation, is a report that will be misread by the next person — usually under time pressure, which is when reports are read.

The spread between comparators is the diagnosis. token_f1 0.5, semantic 0.7, directive 0.2 — read together, that triple says exactly where a rewording-resistant rule lives. A single composite says "medium." Composites compress; sub-metrics localize.

When a new metric contradicts the old narrative, suspect the metric first — including the new one. 0.08 implied the model was bad. 0.78 implied the model was good. Both readings were one comparator choice away from being wrong. The data did not change between the two; the ruler did.

The broader lesson

If you score a generated artifact against ground truth, you are not adding a number — you are adding a second system with its own bugs, sitting between you and the truth. Test it against cases where you know the answer (a near-verbatim extraction must score high; a reworded-but-correct one must not collapse), publish its parts, and give "unmeasurable" its own value. Otherwise you will spend a cycle fixing the thing the metric was designed to exonerate.

The honest ruler pointed at the next problem: two corpora had sat at a flat zero for a full release, and the reason was not in the model at all.

References

CauterRule v0.3.1 is released. The extraction-accuracy tables, the J6/J10/J13 issue-journal entries, and the full 40-corpus × 2-model sweep are in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes

DEV Community