Four Models Cited My Numbers Perfectly. One Still Misread Them.

Originally published on hexisteme notes.

I recently froze a table of behavioral metrics computed from 281 of my own AI-agent sessions and ran it through two separate checks before I let myself trust any conclusion drawn from it. Both checks passed. Neither one verified the thing I actually needed to know — and both failures turned out to have the exact same shape.

The underlying table is observational, closer to a photograph of my own routing policy than a fact about the models sitting in it, and I've already written about that failure mode on its own. This post assumes that caveat and goes one layer past it: what happens after you've accepted it, built real checks anyway, and watched them both come back green.

Act one: a citation audit that passed 70 of 70

The corpus was 281 sessions, broken into 4,818 threads and 4,415 rows of behavioral metrics, frozen so nothing in it could shift under me mid-analysis. I had my own relaunched main model read the frozen table and write down its claims first, sealed before any outside model saw the data. Then I sent the identical table to six outside vendor families and asked each one to surface claims of its own. Only three of those six made it back cleanly — one hit a free-tier quota of zero, one threw an internal error, one had its response stream truncated mid-read. Between my own sealed read and the three outside reads that did return, four independent families ended up looking at the same numbers: my relaunched model, xAI's Grok, DeepSeek, and a Google open model.

Then I ran the part of the check that actually matters: a deterministic auditor that pulled every number any of the four models had cited, recomputed it directly against the frozen database, and flagged anything off by more than 5%, or any number that didn't correspond to anything in the data at all. 70 citations went in. 70 came back PASS. Zero hallucinated numbers, across four models with no shared training lineage.

That's a genuinely good result, not a strawman I'm setting up to knock down. It rules out a real failure mode — models inventing statistics that merely sound plausible. What it doesn't rule out is a model reading a real, correctly cited number and drawing the wrong conclusion from it. That happened once, cleanly enough to use as the example.

One row in the table belonged to Claude Sonnet 5's main-role threads — a small population, nine of them — where four behavioral metrics, including tool_error_rate, all read exactly 0.000. DeepSeek's model cited that 0.000 correctly and concluded it meant precise, error-free tool use, then built a routing suggestion on top of that reading. My own model, looking at the same row, flagged it as a likely measurement artifact instead. Sonnet 5 barely acts in a main-role thread in this corpus, and most of these rates are derived from file edits it almost never makes there: with no edits underneath the ratio, the rate isn't low, it's undefined and defaults to zero. A median tool-error rate of zero across those nine sparse threads is the same kind of non-signal — tie-degenerate, not a track record. The zero meant "not measured," not "no errors."

The audit never had a chance to catch this, because there was nothing wrong with the citation to catch. DeepSeek quoted 0.000 accurately. The mistake happened one inferential step later, in the word "therefore" — and a citation-accuracy audit has no way to see that step. It checks arithmetic, not reasoning.

Act two: the test built to be rigorous, and still not enough

The fan-out audit was casual by design: let several models free-associate over a frozen table and grade the arithmetic afterward. One metric got stricter treatment. Before looking at the data, I pre-registered a formal equivalence test — TOST, two one-sided tests — on same-file re-edit rate, comparing my relaunched main model's main-role threads against Claude Opus 4.8's, the older flagship. I committed to the margin in advance, 0.75 of the pooled standard deviation, and to a simple rule: the test only passes if the whole confidence interval around the observed difference sits inside that margin.

Quantity	Value
Relaunched model, main-role sessions (n)	54
Claude Opus 4.8, main-role sessions (n)	145
Observed mean difference	+0.089
Pre-registered equivalence margin	±0.222 (0.75 × pooled SD)
90% confidence interval	[+0.015, +0.163]
Pre-registered test verdict	Pass
Adversarial review verdict	Unsupported

By the rule I'd committed to, this passed clean: the full interval sat inside the margin. That's the good kind of check — a number graded against a threshold I picked before I knew whether it would be convenient. So I treated "passed" as license to say the two models were close enough on this axis to stop worrying about it.

I sent the result to two adversarial reviewers from two different outside model families — one instructed to attack the statistics, one instructed to attack the operational reasoning. Both came back with the same verdict: unsupported. Not wrong about the arithmetic — wrong about what I'd let the arithmetic mean. Three reasons, and all three held up under a second look:

• The margin was wide enough to pass almost anything. A zone of 0.75 pooled standard deviations is generous. A pass against a margin that loose proves less than a tighter, harder-won pass would have.
• The interval that cleared the margin also excludes zero. [+0.015, +0.163] never crosses 0. That means there is a real, directional difference — the relaunched model re-edits the same file measurably more often than the flagship did — and "equivalent" is the wrong word for a gap with a confirmed sign.
• The metric was never the decision. Same-file re-edit rate is one proxy for editing behavior. It says nothing about which model was more often correct, whether the tasks either one touched actually finished, or how long either took to get there — the axes that would actually justify routing one over the other.

The shape both checks share

Line the two failures up and they match. The citation audit's definition of "pass" was: did you copy the digits. The equivalence test's definition of "pass" was: does the gap sit inside a zone I chose in advance. Neither one asked the question I actually wanted answered — did you understand this, and are these two models actually interchangeable. A passing score on either check told me something true and narrow, and I filled in a broader claim myself that the check never made.

The honest version of where this leaves me: the observed behavioral signal for telling my relaunched main model and Claude Opus 4.8 apart is weak — one metric, one axis, a real but small and bounded gap. That's the whole claim that survives. "They're interchangeable" is overreach. "Route on price alone" is overreach. Both were the conclusions I was reaching for before the adversarial pass, and both got killed by reviewers doing the one thing neither check I'd built was designed to do.

One more detail worth keeping: the reviewer that attacked the operational reasoning in Act Two was DeepSeek's model — the same family that misread Sonnet 5's zeroed row in Act One. Same family, opposite outcome, two different jobs. The lesson isn't "trust this vendor less." It's that a blind spot belongs to the check, not permanently to whichever model happens to be running it. A citation audit will always be blind to interpretation, no matter which model passes it. An adversarial refute pass will catch what a citation audit can't, no matter which model runs it either.

What I'm changing

Two changes, both narrow on purpose:

1. Any analysis I delegate to an LLM now gets an interpretation pass, not just a citation pass. A different-family model reads the same claims specifically to argue against them — is the number correctly cited, and separately, is the conclusion drawn from it the only conclusion the number actually supports. Those are different questions, and a single audit can't answer both.
2. Any equivalence claim has to report the direction, not just the verdict. The margin gets tied to a real decision threshold before I look at the data, not a statistical convention. And whichever way the test goes, I report whether the confidence interval excludes zero. A pass with a wide margin and a directional interval is a different finding from a pass with a tight margin and an interval that straddles zero — but the word "PASS" alone can't tell you which one you got.

More notes at hexisteme.github.io/notes.