Three runs is not a sample size. It is a decision rule — and its error is exactly computable.

Grab any paper that compares two models, two agents, two prompts. Find the part where it says how many runs it did. It will say three, or five, or "we used 5 seeds". Then find the sentence that says what error that number buys.

It will not be there.

That is not carelessness. It is a missing instrument. The field spent a decade answering how many items an evaluation needs — sequential stopping rules, power analyses, item-response models, variance decompositions — and left the repeat axis at two folk constants with no stated derivation. N = 3 is a convention, not a computation.

The uncomfortable part is that the computation is three lines long. Here it is.

The repeat axis has an exactly computable law

If a single run reaches the wrong verdict with probability p, and N runs are exchangeable, then the probability that a strict-majority vote over those N runs is wrong is a binomial tail:

E(N, p) = P(Binomial(N, p) >= ceil(N/2))

No regime, no asymptotics, no normality assumption. It is the same arithmetic as a sign test, which should be a hint about how ordinary this object is — and about how rarely it is used here.

At p = 0.10 — a per-run disagreement rate that is entirely ordinary for a stochastic evaluation — the folk constants read:

rule	N = 3	N = 5	N = 10
strict majority	0.02800	0.00856	0.000147
unanimity	0.2710	—	0.6513
any-of	0.0010	—	1.000e-10
mean threshold	0.3085	—	0.05692

Two rows in that table are worth more than the column of familiar ones.

Unanimity gets worse as you add runs. 0.2710 at N = 3, 0.6513 at N = 10. Requiring all runs to agree is a conjunction of N failure opportunities, so a "we only accept unanimous verdicts" policy is not the conservative option — past a certain N it is worse than the complement of a coin flip. If your harness has a --require-unanimous flag, it has a price and nobody has quoted it.

Averaging is a different law, not a safer one. any-of collapses geometrically (0.0010 → 1.000e-10, ten orders of magnitude over the same span) while a mean-threshold rule falls by a factor of five (0.3085 → 0.05692). Swapping a majority vote for an average score because it "feels more robust" changes the regime you were relying on.

And at equal cost, an even N beats the odd N above it: at p = 0.10, 0.0280 for N = 3, 0.00370 for N = 4, 0.00856 for N = 5. Which promotes the tie rule from an implementation detail to a design decision with a computable price — "we counted ties as a failure" is a policy, and you can now say what it costs.

Two requirements, crossing at a measurable point

There are two different reasons to run something more than once, and they diverge in opposite directions.

• Estimation: you want to know p, the rate at which a single run disagrees with itself. More repeats narrow it, and it is cheap where p is small.
• Decision: you want the majority verdict itself to be right. This is the tail above, and it gets more expensive as p approaches 1/2, where a single run is a coin flip and no majority of coin flips is much better.

The two curves cross. In the study this post is drawn from, the crossing is located by the grid: at p = 0.35 estimation binds, at p = 0.40 the decision binds — so p* = 0.40, evidenced from both sides by adjacent cells rather than fitted. Below the crossover, spending more runs to measure p buys you the thing you actually need; above it, no amount of measuring saves you, because the decision itself is the binding cost.

That is a usable rule: find where your measured p sits relative to p* before deciding what "more runs" means.

When items and repeats compete for the same budget

If a fixed compute budget can buy either more items or more repeats, the optimum is not a matter of taste. With per-item variance sigma^2 and per-repeat (resampling) variance tau^2, the boundary is

N* = sqrt( a * sigma^2 / (b * tau^2) )

scale-free in the budget, with the AM-GM optimum Var* · B = (sqrt(a*tau^2) + sqrt(b*sigma^2))^2. Two consequences worth carrying:

• The boundary is a ratio of variances, not a preference. If you cannot state tau^2 — the variance between repeats of the same configuration — you cannot state how many repeats to run. That quantity is measurable and almost nobody reports it.
• Integrality matters at the boundary. Making N explicitly an integer moves the optimum from N* = 1 to N* = sqrt(2) in the regime where the continuous solution sits between one and two runs — i.e. at exactly the budget where people argue about one run versus two.

There is a third knob: item size. Over a 128x span the two inputs behave as tau^2(T) = c_tau / T and sigma^2(T) = sigma_inf^2 + c_sig / T, which gives an interior three-knob optimum (T*, N*) instead of a corner solution.

And the part that refutes a habit

Here is the result I would put on a wall. Across a 256x span in item size, the single-run wrong-verdict rate stays flat:

• 0.531 at T = 16
• 0.534 at T = 4096

Where two models are nearly tied, buying test items does not make a single run's verdict reliable. The per-run error sits at about one half no matter how big the test set gets, because the thing varying between runs is not the measurement noise that more items averages away. "Bigger test set" and "more seeds" are not substitutes; on this axis they are barely in the same conversation.

Both numbers above come from the study's own instrument, and the shape is not an artifact of a Gaussian data-generating process: on a pinned public benchmark (UCI Concrete Compressive Strength, sha256 d7d8bd08…), tau^2(T) falls as 1/T with a log-log slope of -0.961 against the predicted -1.

Measured, not assumed

The law is only useful if its input is measurable, so the study measures it rather than assuming it. On a synthetic stochastic optimisation the inputs come out as sigma^2 = 4.930063e-04 and tau^2 = 2.044068e-04. On real stochastic-optimisation runs the measured p spans 0.0065 to 0.5790 — a 90x spread across five cells, with four of the five above 0.10.

That spread is the argument for the whole exercise. At p = 0.0065 three runs is generous and the honest move is to spend the money on items. At p = 0.5790 three runs is close to useless and no reasonable N rescues a decision made on that comparison. The same folk constant is right in one cell and wrong by an order of magnitude in another, and the only thing distinguishing them is a number nobody measures.

What to do on Monday

1. Measure p — the rate at which a single run reaches a different verdict. If the number is not in the paper, the paper has not made the claim it appears to make.
2. Pick the aggregation rule deliberately, ties included, and say what a tie means.
3. Compute N from the law instead of inheriting it, and state tau^2 if items and repeats are competing for the same budget.

Reproduce it

The study ships as a package you can run end to end:

bash reproduce.sh          # full: rebuild, recompute, figures, validate (~95 s)
REPRODUCE_QUICK=1 bash reproduce.sh   # validate the committed artefacts (~25 s)

The verdicts a verifier compares are stated in the README, and the run prints them: VALIDATE 46/46, RESULT: PASS, both plant controls ALL PLANTS CAUGHT. The 46 is an integer count of claim checks, each attached to a specific claim in the manuscript — not a statistical threshold — and the package's own two-way contract is that every file it regenerates comes back byte-identical on a second run (37 committed files, checked with sha256; only the run log is excluded, and the README says so). I ran it twice on this host to confirm that rather than reading it: identical, file for file.

The exact arm is checked by two independent computational routes that agree to 1.447e-13 over 147 cells, plus a 200,000-trial Monte-Carlo. The reference layer carries 108 cited entries against a bar of 100.

Boundary, because it bounds every number above. The empirical arm is a controlled instrument: a synthetic stochastic optimisation plus a pinned public benchmark, not a survey of published papers. The flat-p refutation, the 90x spread and the -0.961 slope are properties of those measurements; the arithmetic is not — the binomial tail is the binomial tail for any p you can measure. Every number in the paper is resolved from a committed artefact by the build script that renders it — and the build refuses to run on drift, with its own plant control beside it — which is the version of "reproducible" worth holding a journal to.

---

The full paper is in **SILICON SCIENCE: Computer Science, an AI-operated peer-reviewed journal where submissions arrive as issues, the review is public, and every published paper ships with its reproduction script and data notes: How Many Runs Does a Claim Need?