Debashish Ghosal

Posted on Sep 15 AI-assisted

A Floor of 0.80 and a Ceiling of 0.63: The Semantic Channel That Never Fired

#cli #github #opensource #python

Update — v0.3.1 released. CauterRule is now live on GitHub and PyPI. It turns repeated agent failures into permanent standing rules — extract, replay-test, promote. pip install cauterule gives you the full CLI, framework adapters, rule lifecycle, pack ecosystem, and official rule packs. The v0.3.1 field test report evaluated 2 cloud models across 40 corpora and 4,742 trajectory-runs and is the source for every number below. Release notes · Changelog

CauterRule is an open-source sidecar that learns standing rules from repeated agent failures. It extracts lessons from trajectories, replay-tests them, and tries to separate reusable guidance from noisy overgeneralization.

The previous release's article on recall ended with a named to-do: "the floor has to move, and the cosine has to be computed against the failure signature, not the whole trajectory." This release we did exactly that. This is what the gap between a 0.80 floor and 0.63 reality cost, and what moving it bought.

The mechanism: a channel that existed and never fired

The replay matcher blends three signals:

score = 0.5·token-F1 + 0.3·bigram + 0.2·semantic (MiniLM cosine)

The semantic term was the channel meant to carry paraphrases — "non-fast-forward" versus "Updates were rejected because the remote contains work that you do not have locally." Two problems sat on top of it:

The semantic floor was 0.80. The semantic channel only carried a match when the cosine cleared 0.80. A paraphrase with zero shared tokens scored at most ~0.2 on the blend (0.5·0 + 0.3·0 + 0.2·capped-semantic) — below every promotion threshold. The channel was structurally unable to change a verdict.
The signature was diluted. The haystack embedding included the failure_class label ("ci/lint"). On a short paraphrase trigger, the measured cosine was 0.631 with a clean signature — already far under the 0.80 the channel needed — and 0.547 with the class label mixed in, under every floor we have ever used. The label was tax on every semantic match, paid in the exact cases the channel existed to save.

So the state of the world through v0.3.0 was: a semantic channel that was correct, unit-tested, and off. Every paraphrase silently fell back to lexical scoring. The tests were green because the tests set up clean, high-similarity cases — the kind of case that never happens on real paraphrases.

The fix (#721/#722 + J11): move the floor to where matches actually live

Three changes, all in the matcher:

Semantic floor 0.80 → 0.62 — set against the measured distribution of true paraphrase cosines, not against an instinct for "safe."
Class-free signature view. The trigger is embedded against the failure signature without the class label, and the matcher takes the max similarity over the views. The label can inform routing; it stops taxing the cosine.
Structured failure signature. Matches are grounded on a structured failure signature rather than the whole trajectory's prose.

No model change. No prompt change. No change to the blend weights (that one is still open — see below).

The data

Metric	v0.3.0 (gpt / llama)	v0.3.1 (gpt / llama)
Golden recall	0.170 / 0.228	0.377 / 0.427	~1.9×
Golden pass rate	30–50% (n=10, no CI)	82% [0.70, 0.89] / 83% [0.72, 0.91] (n=60, Wilson CI)	clears the ≥70% gate
`adapters`	0/60	60/60	rides on this + references
`raw/ci`	0/110	21/47 / 26/47	rides on this + corpus repair
`reference-expansion`	19/303	201/303 / 198/303	~10×

(The adapters and raw/ci rows are the haystack story from the previous article — the floor and the references landed together, and both numbers ride on both fixes. I am not splitting the attribution we did not instrument.)

Two things about the golden row deserve their own paragraph. First, the recall nearly doubled — 0.170→0.377 (gpt) — which is the direct signature of a semantic channel that can finally carry a match. Second, the v0.3.0 golden "30–50%" was an n=10 point estimate with no confidence interval; v0.3.1 expanded golden to a powered n=60 with 54 authored replay references and expected_rule backfill, so "82% [0.70, 0.89]" is a claim with a lower bound that clears the gate, not a range we are waving at. Same kind of honesty fix as the ruler article: the number got a denominator and a CI before it got a title.

What worked

Measuring the score distribution before setting the floor. The 0.80 floor was never calibrated against a single measured paraphrase cosine. Once we had the distribution (true matches clustering ~0.63, with label-dilution dragging some under 0.55), 0.62 was a decision, not a guess.
Max over class-free vs class-included views. Cheap, reversible, and it removed the label tax without discarding the label's routing value.
CI-backed golden claims. n=60 + Wilson turned "golden improved" from an anecdote into a gate-passing statement.
Naming the fix in the issue journal (J11, #721/#722) so the report's before/after trace back to a diff, not a vibe.

What didn't work

The 0.80 floor itself. It was set from a "safe" instinct — high bar, few false matches — with no measurement of where correct matches actually score. A threshold above the ceiling of the distribution is a kill switch, not a safety measure. It kills the false matches and all the true ones, silently, and it is indistinguishable from "the model is bad at paraphrase" in a report that doesn't log raw similarities.
Class labels inside the embedded signature. The label is metadata; embeddings do not know that. On short texts, a few constant tokens move the cosine enough to cross a floor.
Unit tests that only test the high-similarity case. The channel was "tested" and dead. A matcher test suite needs a paraphrase fixture — zero shared tokens, correct semantics — or it will certify exactly this kind of dead channel.

Questions we still can't answer

Is 0.62 portable? It is calibrated on this corpus, these two models, this embedding model (MiniLM). Swap any one of the three and the distribution moves. We have no re-calibration procedure that doesn't amount to re-fitting to the test set — which would be cheating by one commit.
Should the floor be per-corpus? The live threshold for raw/* is already 0.35 (the "0.45→0.40" option from the plan turned out to target dead code — traced in the previous article). Curated corpora run at 0.70. The floor and the corpus threshold are two numbers doing related jobs; whether they should be one number with a per-corpus table is an open refactor.
Is the 0.2 semantic weight the last of this bug? The long-term plan raises it to 0.3–0.4. If the floor was half the disease, the weight is the other half — and we have not measured the split.

What I learned

A threshold you never measured against the real distribution is a guess with a number on it. The 0.80 floor survived a full release cycle because nothing in the pipeline logged "this match scored 0.63 and the floor said no." Adding raw-similarity logging to the inconclusive attribution is what finally made the gap visible.

"Safe" thresholds fail by subtraction, not addition. A loose threshold adds visible false positives you can see and argue about. An unreachable threshold subtracts true matches you cannot see. The failure is quieter and the misdiagnosis ("the model is weak at paraphrase") is more plausible.

Labels are not embeddings. Anything you append to the text you embed is a token, and short texts have no slack. If a label must inform matching, use it for routing and scoring — not as seasoning in the vector.

A recall doubling with no model change is the fingerprint of a dead channel. When the same model, same corpus, and same prompts produce ~2× recall after a config change, the previous config was not tuning the model. It was muting it.

The broader lesson

Every similarity floor is a bet about where real matches sit in the score distribution. If you have not plotted that distribution, you are not calibrating a threshold — you are guessing one, and the system will behave as if the feature you built does not exist. Log the raw scores, find where the known-correct matches land, and set the floor below them. Then watch the attribution column: when matcher_gap stops being the top reason for inconclusive, the channel is alive.

The channel was alive, the haystack was full, the ruler was honest. What was left was the judge — a scorer that was still killing correct rules for the crime of sharing a word with an unrelated success. The word, in the case that started it all, was "git."

References

CauterRule v0.3.1 is released. The before/after recall, the golden n=60 Wilson CIs, and the class-dilution measurements are in the field test report. The repo is public. Install with pip install cauterule. Changelog · Release notes

DEV Community