The Compiler Got 5% Slower. The Benchmark Called It a 10% Regression a Quarter of the Time.

A benchmark suite is an instrument. You point it at two versions of a compiler, it tells you whether the new one got worse, and somebody merges or reverts a pull request on the strength of that answer.

If the compiler is stochastic and the suite does not pin its seed, that answer is a draw from a distribution nobody measured.

I measured the distribution.

The finding

At the pinned revision I inspected, IBM's Benchpress fixes the seeds that build its circuits — seed=12345, fourteen occurrences — and passes no seed_transpiler to the code that compiles them. Circuit construction is deterministic; compilation is not.

On bv_n140 mapped to a heavy-hex lattice, going from Qiskit 1.4.3 to 2.0.0, across 400 seeds per version and 21 OS processes — a separate measurement from the pre-registered experiment below, and not part of the pre-registered endpoint:

quantity	value	95% interval
long-run mean change	+5.37%	+4.27% to +6.50%
called a ≥10% regression, 3 runs/version	24.4%	19.5% to 31.1%

The true change is nowhere near the threshold. The verdict crosses it a quarter of the time anyway.

I did not trust that at first, because all 200 original seeds came from one contiguous block inside one process — so per-process state, hash randomisation and allocator layout were all held constant. A disjoint set of 200 seeds spanning 5.9 million to 1.08 billion, across ten fresh processes with differing PYTHONHASHSEED, gave 22.6% [16.4, 30.6] against 26.3% [18.5, 34.2] from the contiguous set. Each point falls inside the other's interval.

Running it more does not fix it

The obvious response is "run it more times". Pooling all 400 seeds:

runs per version	1	3	5	8	10	20
false-positive rate	34.58%	24.38%	18.56%	12.94%	10.36%	3.74%

Twenty runs per version is 20 × 2 versions × roughly two hours per suite run — the issue's own figure, which I could not independently confirm — so on the order of 80 compute hours. It still leaves 3.74%.

Fixing seed_transpiler removes the variance attributable to the transpiler seed at a single run. It does not make one run correct. A fixed seed makes the answer repeatable, including when that answer is wrong: on bv_n280, recorded seed 663193 gives A = 1040 against B = 1157 — a +11.25% call against a measured θ of +4.89%. Frozen, reproducible, and the wrong side of the threshold.

Taking the minimum of three runs instead of the mean — a common instinct — makes it worse. Every rate roughly doubles.

The pre-registered part

One circuit is an anecdote. So I wrote down the selection rule, the analysis, the endpoint and the labels and committed them before the first tracked commit of the 39-circuit raw dataset. The commit ordering is checkable in the repository.

What that ordering supports is a frozen follow-up design after exploration, not blindness to earlier outcomes — and the paper corrects that distinction twice. A census and exploratory results preceded the pre-registration: an earlier commit the same day already identifies heavy-hex as the worst case, and the ≤10 s selection rule is computed from that pre-existing census. Git records when files were committed; it does not record when their outputs were first looked at.

The rule selected every qasmbench-large circuit whose twelve-seed heavy-hex runtime in that census was under ten seconds: 39 circuits, 200 recorded seeds per arm, 15,600 primary transpilations. 36 resolved; 26 were eligible after pre-registered boundary exclusions.

Over those 26:

• 12/26 met the pre-registered endpoint — a risk interval excluding zero
• 7/26 carry a risk of at least 5%
• 4/26 carry a risk of at least 10%

The median circuit admits an ambiguity band of about 10.9 percentage points — the width of the interval of true change over which the three-run rule's call probability runs from 5% to 95%. That band is built from the measured residuals of this version pair under a stated model. It is not claimed to be version-pair-independent.

What I withdrew

That 46.2% headline rate is the number I trust least, so I want to say why rather than let someone find it.

The 26 circuits are not 26 independent observations. They come from eleven algorithm families and within a family the results are nearly all-hit or all-miss. A cluster bootstrap over families gives [17.4%, 81.0%] against the Wilson interval's [28.8%, 64.5%]. The effective sample size is far below 26, so no interval I can compute is informative about a population of circuits.

The suite-level rate is withdrawn. 12/26 stands as a descriptive count of these 26 circuits and nothing more.

Three other claims went the same way. One was a correlation of ρ = +0.876 between "compilation is stochastic" and risk, which I had presented as a discovered mechanism. Circuits whose two arms are constant have zero risk by arithmetic, so a correlation computed over a sample containing them measures a definition. Simulated data containing no compiler at all reproduces the statistic.

All four withdrawals are still in the repository, with the evidence that defeated them.

Checking the checker

The paper argues that measurement apparatus should be checked rather than trusted, so it would be poor form not to turn that on myself.

The repository ships a thirteen-stage read-only verifier and a suite of 44 deliberate corruptions, each of which must be rejected by a named stage for a named reason — not merely cause a nonzero exit. The paper's own account of the review of its claims is four rounds of hostile review plus two first-principles audits, which removed five claims from the work. Every audit round after that attacked the apparatus instead. Every one of them ended with a green verifier that the next reader then broke, and not one of them changed a scientific number. What they broke was the checking.

The best example: a claim scan that could not see the sentence "Seven of the twenty-six eligible circuits have an interval that excludes zero" — because twenty-six was not a number it recognised, so no claim object was ever created and the fail-closed classifier was never asked. Four rounds of widening a recogniser lost to the fifth spelling. The fix was to stop asking "can we recognise every English sentence that means 7 of 26" and instead register every reader-visible unit of the frozen manuscript by hash, so an unregistered block is refused before anything reads a word of it.

That is evidence the verifier can turn red. It is not a proof that nothing gets through, and the paper says so in those words.

What this does not say

It does not say Qiskit's transpiler got worse, or that Benchpress is a bad suite, or that any published decision was wrong. It measures one property of one benchmarking protocol: an unseeded stochastic compiler, sampled three times, produces regression verdicts that disagree with its own long-run behaviour at rates that are not small.

Limits, stated plainly: one SDK, one version pair, one topology family, one machine. The +10% threshold is mine — Benchpress defines none. seed_transpiler is not a universal remedy either: it removed false positives on the circuits I tested but was worse on three of the four showing false negatives.

If you maintain a benchmark like this

The cheap version of the check: run the same comparison twenty times without changing anything, and look at the spread of verdicts, not the spread of values. If the verdict moves, the number of runs is part of your instrument and it belongs in the write-up.

Version 4 is the audited release. Paper, code, raw per-seed data and the assurance apparatus:

• DOI — 10.5281/zenodo.22848938
• Tagged release — github.com/Mormolykos/qvalidation @ v4.0.0
• Full write-up — ai.bedvibe.studio/decision-risk/

CC-BY-4.0.