A stable aggregate is not a stable measurement

We ran the same benchmark five times against our own production system and got 752, 750, 750, 749 and
749 out of 800.

That is a spread of three items across five runs. Less than half a percentage point. If you saw those
five numbers you would conclude the measurement was essentially deterministic, quote the mean, and
move on. I nearly did.

Then I compared the runs item by item, and 38 of the 800 items had changed answer between one run
and another. Not three. Thirty eight, which is 4.8 percent of the set.

The totals barely moved because the flips cancelled. Something that was right in run 1 and wrong in
run 3 is invisible in a sum, and there were enough in each direction that the aggregate came out
almost identical every time. The number was stable. The measurement underneath it was not.

What five runs actually showed

run	score	of 800
1	752	94.00%
2	750	93.75%
3	750	93.75%
4	749	93.62%
5	749	93.62%
mean	750.0	93.75%

Same 800 fixed items, same configuration, same scorer, five separate passes through the production
path, an hour and a half apart end to end. Temperature was zero, which people reasonably expect to
mean deterministic. It does not.

Partitioning the 800 by how they behaved across all five runs:

across all five runs	items	share
all five correct	730	91.2%
all five wrong	32	4.0%
changed at least once	38	4.8%

And the pairwise view, counting items where two runs disagree with each other:

run pair	items that disagree	rate
1-2	16	2.00%
1-3	22	2.75%
1-4	19	2.38%
1-5	21	2.62%
2-3	20	2.50%
2-4	23	2.88%
2-5	17	2.12%
3-4	21	2.62%
3-5	17	2.12%
4-5	16	2.00%

The sharpest example is the pair that disagrees most. Runs 2 and 4 differ on 23 items. Their totals
differ by one. If those had been the only two runs I did, I would have written down that the system
reproduces to within a single item, and I would have been wrong by a factor of twenty three.

Why this matters if you are comparing anything

The reason to care is not academic. It is that almost every model comparison you read, including the
ones I have written, is a difference between two totals.

If your measurement has a 2.88 percent chance of flipping any given item between identical runs, then
two systems whose totals differ by less than roughly twice that are not distinguishable by the run you
did. You need the gap to clear the noise, and you cannot know the noise without measuring it.

We learned this the embarrassing way first. Earlier we ran a three model comparison twice, one hour
apart, same prompts, same temperature zero. The model that came first in run one came third in run
two, and the one that came third came first. The gaps were around one to two points. The noise was
larger than the gaps. We had a ranking, and it was a coin.

The cheap fix, which is one extra run

You do not need a statistical apparatus. You need a second pass of the identical set.

1. Run your benchmark twice on the same items. Do not change anything.
2. Count the items that changed answer. That percentage is your noise floor. Ours is 2.88 percent on this workload.
3. Refuse any comparison whose gap does not clear roughly twice the floor. Not "note the caveat." Refuse it, in code if you can, so a tired person cannot quote it later.

We put that last rule in the tool. It computes the floor alongside the result and will not bless a
claim that sits inside the noise. It has told us no more than once, including on a result that
flattered us.

And the thing worth keeping instead

If aggregates are fragile, something has to replace them, and the replacement is better anyway.

Report which items you disagree on, not who scored higher. When we compared three models by task
rather than by total, the partition was stable across runs even while the ranking was not: the same
problems came out contested each time, far more often than chance would put them there. Which problems
a system gets wrong is a property of the system. Who scored higher on the total is a property of
the afternoon.

That is also the more useful finding for anyone actually building something. A leaderboard position
tells you which one to pick. A partition tells you where picking one is not enough.

The uncomfortable part

I want to be honest about where this leaves our own numbers, because it would be convenient to publish
this as a lesson for other people.

It applies to us. Every benchmark result we have quoted from a single run is a number with an unstated
uncertainty, and some of the differences we have found interesting in the past are inside the floor we
just measured. The 2.88 percent is not a result about other people's evaluations. It is the error bar
on ours, and we did not have it until this week.

The five runs cost about two hours and a few dollars. The thing they bought was not a better score. It
was knowing how much of the score was real.

---