Technical source: R2.1 results, R3 results, R5 results, R3/R4/R5 reconciliation
Companion post: Context Is Part of an Agent's Authority
I built a benchmark family to test whether a governed metadata layer earns its cost when an agent selects enterprise context. I have now run it four times under four frozen contracts, redesigning the catalog and changing the acceptance ceiling along the way.
The claim that governance earns its cost against a cheap baseline has been rejected in every round that tested it. The round before those was rejected too, on a different rule.
| Round | Claim | Failed rule or controlling result | Verdict |
|---|---|---|---|
| R2.1 | Overall comparative claim | Governed holdout F1 0.24065, below the 0.245533 floor | REJECTED |
| R3 | Governed value vs. lexical | F1 CI [-0.371, 0.00005]; token ratio 2.11x vs. 1.10x ceiling |
REJECTED |
| R4 | Governed value vs. lexical | Quality passed; token ratio 9.86x vs. 3.0x ceiling | REJECTED |
| R5 | Governed value vs. lexical | Quality passed; token ratio 6.94x vs. 3.0x ceiling | REJECTED |
R3 through R5 are value-claim verdicts. R2.1 was the earlier overall-contract rejection that led me to separate the claims. Other claims passed: R3 and R5 accepted governed routing against full-context stuffing, while R4 returned those claims inconclusive.
Every round ran under a contract frozen before collection. The claim-scoped outcomes are documented publicly; R3 and R5 include public decision packs, while R4's later-derived pack remains held and is disclosed as such below.
R2.1 failed first. At its 3,000-object holdout the governed route scored 0.24065 against a prespecified floor of 0.245533, while the lexical prefilter scored 0.588.
Round 3: the simple baseline won the observed comparison
R3 compared three routes on the same local model: raw full-context stuffing, a cheap lexical prefilter, and a governed metadata route.
Background. This benchmark reproduces the structure of McKnight Consulting Group's study, "Stop the Token Bleed: Benchmarking the Benefits of Governed Metadata for Enterprise AI" (Jake Dolezal and William McKnight, August 2026; sponsored by Informatica, a Salesforce company). Their study held the model constant and found governed metadata access won on both cost (up to roughly 89x fewer tokens at scale) and accuracy (F1 1.000 against 0.29 to 0.66 ungoverned). Read their article for their full methodology and figures. This work does not reproduce their exact numbers. It lets you generate your own, on your own model.
Disclosure. The study reproduced here was sponsored by Informatica, a Salesforce company. I am employed by Salesforce. That is a reason to run this harness yourself rather than take my output on trust, which is the entire point of publishing it. Contradicting results are welcome.
Mean F1 across 20 seeds, at the prespecified 0% classifier-miss condition:
| Catalog size | Governed F1 | Full-context F1 | Lexical F1 |
|---|---|---|---|
| 300 | 0.780 | 0.261 | 0.660 |
| 1,500 | 0.632 | 0.253 | 0.737 |
| 3,000 (holdout) | 0.447 | 0.177 | 0.631 |
Read the two columns downward. The governed route degrades monotonically as the catalog grows: 0.780, then 0.632, then 0.447. The lexical route does not show the same pattern: 0.660, 0.737, then 0.631.
At the holdout, the keyword filter beat the governed route on the observed means and used less than half the prompt tokens. The paired F1 interval was [-0.371, 0.00005], which does not exclude zero in governance's favor. The token ratio was 2.11x against a frozen ceiling of 1.10x.
Governed routing crushed full-context stuffing. That claim passed. It lost to the cheapest thing in the room.
What I changed between rounds, said out loud
I ran R4 and R5 on redesigned catalogs, and I relaxed my own cost ceiling.
R3 used a lexically tractable synthetic catalog, where a keyword filter has real signal to match. R4 and R5 moved to semantic-access catalogs built on opaque physical names. In R5 the lexical route scored 0.000 F1 at every catalog size.
The redesign favored my method on quality: this name-only lexical baseline no longer had matching signal. But it also made the cost comparison harder, because the lexical route now produced an extremely small prompt. At the R5 holdout it averaged 110.0 prompt tokens against the governed route's 763.5, which is why the ratio rose to 6.94x even as the ceiling was relaxed.
Separately, between R3 and R4 I raised the maximum governed:lexical prompt-token ratio from 1.10x to 3.0x, which made the cost rule easier to pass.
Both changes were declared in new frozen contracts before their respective collections.
State that plainly, because a reader who diffs the contracts will find it anyway.
It still failed
In R5, governed selection used 6.94 times the lexical route's prompt tokens. The frozen ceiling was 3.0x. The preregistered claim that governance earns its cost against lexical filtering was rejected.
This happened on a task where the baseline scored zero. Governed context won the quality comparison but failed the prespecified prompt-token cost rule. The frozen contract required both.
R4 failed the same rule at 9.86x. R4 also returned INCONCLUSIVE on its full-context claims because one holdout request contained 128,256 input tokens plus a reserved 3,000-token completion budget, putting it 184 tokens beyond the verified 131,072-token window. It made no model call. It was retained as a preflight refusal rather than silently dropped.
The ceiling could have been relaxed again after seeing 6.94x. Moving a bar you already missed converts a result into a press release.
A benchmark that cannot reject its author is marketing with a methodology section.
What this changes for anyone buying or building a context layer
Ask three questions of any governed retrieval, semantic layer, or context-governance component.
What is the cheap baseline, and did you run it? Not merely full-context stuffing. At minimum, test the cheapest credible selective baseline: a keyword filter here, but potentially BM25, a cached lookup, or another simple retrieval route. If the only comparison is against full-context stuffing, the result may justify selective context, but it does not show that the sophisticated route earns its place over cheaper alternatives.
Was the acceptance rule written down before collection? A cost ceiling chosen after seeing the numbers is a description, not a test.
Where does the method lose? A vendor who cannot name the configuration where their layer is the wrong choice has not measured it hard enough.
My own answers: the baseline is a lexical prefilter, it beat the governed route on observed mean F1 and prompt-token use in R3, and the governed route has never cleared its cost bar in the three rounds that tested it. The route remains worth evaluating where opaque physical names make semantic selection necessary, or where an evidence trail has independent value. These runs establish the quality advantage in that narrow synthetic configuration; they do not yet establish end-to-end economic value. That is a narrower claim than the one I set out to prove.
Evidence boundary
These are synthetic, named-endpoint runtime characterizations on a single local model, qwen3-coder:30b. They are not production results, ROI claims, customer-data results, or a replication of any third-party study. R2.1 remains visibly rejected in the public record rather than buried.
R3 and R5 publish a full public packet: frozen contract hash, complete preflight, claim-scoped decision pack, and artifact digests. R4's contract is public, but its decision pack was derived after the fact from the archived report and is held rather than published. The raw reports stay private because they embed host identifiers, and their hashes are committed so you can tell if they ever change.
Four rounds in, the most useful thing this benchmark has produced is the boundary it refuses to cross.
Top comments (0)