Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

My regression detector alerted on 4.3 points. The model moves 8.57 between runs.

The detector had a threshold. It was the wrong quantity, it had been the wrong
quantity for months, and the reason nobody noticed is a sentence in my own
documentation claiming the suite was deterministic.

What happened

On 2026-09-23 my drift tracker reported Grok 4.3 down 4.3 points, to 91.4%. The
pipeline drafted it automatically, as designed: a finding clears a threshold, a
note gets written, a human merges or does not.

The draft argued the move was real because it cleared the resolution floor
of 2.94 points. That floor is 100 / graded_calls. It is the smallest
difference a single run can express, because accuracy is a count over a
denominator and you cannot resolve a third of a question.

Then I ran the same frozen suite against the same model three more times.

median accuracy over 3 runs	97.06%
reliability of the median run	1.0
spread across the three runs	8.57 points

The move I was reporting was 4.30 points. The model moves 8.57 points against
an unchanged suite. The finding was less than half the noise, and the median of
three clean runs came in higher than either number the note compared.

The two floors, which are not the same floor

This is the whole defect and it fits in two sentences.

Resolution floor: the smallest difference ONE run can print, given its
denominator. 100/34 = 2.94. Pure arithmetic. It is a fact about counting.

Noise floor: how far the SAME model moves between runs of the IDENTICAL
suite. Measured, 8.57. It is a fact about the model and the stack under it.

My detector thresholded on the first and reported anything above it. The second
was roughly three times larger. So the detector was alerting, confidently and
in public, on moves it had no ability to distinguish from taking the sample
again.

This is not a one-model problem. Across the fleet, 10 of 20 models have a
recorded run-to-run spread larger than the arithmetic floor.

Why I never looked

The draft template opened every regression note with this:

against the live models, same questions, same graders, temperature 0

Nine of the eighteen entries in the model registry carry temperature: null,
and the provider layer omits the parameter for them because those models reject
it. The other nine, the mock control among them, default to temperature: 0. So
half the board had never been pinned at all, and the other half was pinned only
as far as each provider honours a zero.

That sentence is why the noise floor was invisible. A reader told a suite is
deterministic has no reason to ask what it varies by, and I was the reader. The
number was not hidden: the runner has been recording acc_spread on every
point for months, because it already takes three samples per model per day. I
had the measurement the entire time and never consulted it, because my own
documentation told me it should be zero.

The fix, and how the fix was also wrong

A move now has to clear both floors: the arithmetic one, and the model's own
spread taken as the median of its last ten recorded runs. On the live board that
suppressed exactly one finding, the Grok note above, and left the other four.

Then I checked the estimator against fresh measurements, and it does not hold up
as well as the commit message I wrote for it implied.

model	board estimate	measured over 3 fresh runs	error
grok-4.3	5.72	8.57	under by 1.5x
grok-4-fast	5.71	2.86	over by 2x
claude-sonnet-5	3.11	3.28	within 5%

It missed in both directions, which is worse than missing consistently,
because a bias can be corrected and a scatter cannot. It was accurate on exactly
the model I expected it to fail on.

That last row also retired something else. probe_repeated's docstring has
claimed for months that three runs half an hour apart moved Sonnet 5 by 9
points. Measured today: 3.28. The repository's own documented noise figure
does not reproduce.

What this run could and could not have seen

A spread taken from three samples is itself a noisy estimate. Max minus min on
n=3 has large variance, so 8.57 and 2.86 are not precise numbers and I am not
going to pretend they are. What survives is the ordering: these models sit
somewhere in the 3 to 9 point range, and a 2.94 point threshold is below all of
them.

What it CAN see is whether the instrument itself is the source. If the harness
or the grader were adding the variance, every model would scatter. So I ran the
same three-probe procedure against GPT-5, same suite, same grader, same
afternoon:

median accuracy	100.00%
reliability	1.0
graded	35 of 35
spread across three runs	0.00 points

Zero. The same runner that recorded 8.57 points of movement on one model
recorded none at all on another, hours apart. The variance is in the models,
not in the instrument, which is what makes a per-model threshold the right
shape and a single global floor the wrong one. A global threshold set high
enough for Grok 4.3 would blind the board to a real three point move on GPT-5,
which this control shows it could resolve.

One confound, and it is mine to state rather than yours to find: GPT-5 scored
100.00%. A model that answers everything correctly every time has no room to
move downward, so some of that 0.00 is a ceiling rather than stability. The
control establishes that the harness does not ADD noise. A mid-scoring stable
model would have been the stronger test, and I do not have one on this board.

What I might have wrong

The fix is directionally right and numerically soft. I replaced a threshold
that was definitely wrong with one that is roughly right, and I shipped it
before checking its estimator. The order should have been reversed. The honest
status is that the arithmetic floor was too low by a factor of about three, and
that the median-of-ten is the best number I currently have rather than the
correct one.

Median-of-ten is a choice I cannot yet defend with data. I picked the median
over the max because Grok 4.5's worst recorded spread is 91.43 points, which is
an outage signature rather than sampling, and taking the max would have muted
every model that ever had a bad morning. That reasoning is sound. Whether ten is
the right window, and whether the median is the right statistic, is unmeasured.

The whole thing rests on one fleet and one suite. Thirty-five questions,
seventeen LLMs and a mock control, one operator, one grader. I have shown that this detector
alerted below its own noise. I have not shown that anyone else's does, and the
interesting version of this claim is the general one.

If you run an eval suite that reports regressions: find the threshold, and then
ask which of the two floors it is. If the answer is 100/n, or a constant
somebody picked, run your suite three times without changing anything and
compare.

---

I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. https://erikhill.dev

Live board: https://egnaro9.github.io/model-drift/
Suite, graders and runner: https://github.com/egnaro9/model-drift

The retired finding, the threshold change and every number above are in public commits. The confirming runs are GitHub Actions 35945537690, 35946570084 and 35946669164.