Two teams run the same model, scaffold, and temperature on SWE-bench Verified, once each. Team A reports 32.4%; Team B reports 28.8%. Both post their logs. Neither is lying.
The gap comes from one distribution. Table 1 of On Randomness in Agentic Evals has this row: DeepSWE-preview on nano-agent, temperature 1.0, ten independent runs, mean 31.4% ± 1.0%, minimum 28.8%, maximum 32.4%. Team A landed at the top of the window; Team B landed at the bottom. They were comparing two samples, not two systems.
The paper, by Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus, appeared on arXiv as 2602.07150 on February 6, 2026 (v3: March 25) and is a preprint.
What the experiment did
Most papers run each configuration once. This one ran three models (Qwen3-32B, DeepSWE-preview, Devstral-2) × two temperatures × two scaffolds (nano-agent, R2E-Gym), ten times each, on 500 SWE-bench Verified tasks: 60,000 trajectories, 25.58B tokens, 1.88M tool calls.
One definition matters. The headline range of 2.2 to 6.0 percentage points is the spread between the minimum and maximum single-run score within the same configuration. It is not the gap between two models. As the paper puts it: "single-run pass@1 estimates vary by 2.2 to 6.0 percentage points depending on which run is selected." The number measures repeatability, not capability.
Three sources of variance
The first is sampling. Above temperature zero, every token is drawn from a distribution. One different token changes the context for the next one, and the run can diverge from there.
The second is the scaffold. It changes both the level and the spread. DeepSWE-preview scores 31.4% ± 1.0% on nano-agent at temperature 1.0, but 34.4% ± 1.5% on R2E-Gym. Devstral-2 scores 63.5% ± 1.1% on nano-agent at temperature 0.2, but 34.9% ± 1.5% on R2E-Gym. The model did not change; the wrapper did.
The third is the runtime, and it only becomes visible at temperature zero.
Temperature zero is not a determinism switch
Greedy decoding sounds deterministic: always pick the most likely token. The paper shows it is not. Floating-point addition is not associative, and parallel reduction order, hardware-specific kernels, and batching can produce tiny differences in the same logits. Autoregressive decoding amplifies them.
At temperature 0, the standard deviations shrink but never reach zero. Qwen3-32B on nano-agent is 16.4% ± 1.2%; DeepSWE-preview on nano-agent is 20.4% ± 1.0%; Qwen3-32B on R2E-Gym is 22.3% ± 1.8%. The largest standard deviation at temperature 0 in the table is 1.8 points.
Lower temperature does not reliably shrink variance either. Qwen3-32B on nano-agent has a standard deviation of 0.7% at temperature 0.6 and 1.2% at temperature 0. Temperature zero delays divergence; it does not remove it.
So when a paper says temperature=0 makes its result reproducible, ask: over how many runs, with what standard deviation, on which inference engine and hardware?
Trajectories split early
For DeepSWE-preview on nano-agent at temperature 1.0, the median position of the first differing token between two runs was token 5, about 0.5% of the trajectory. At temperature 0 it moved to token 56. The paper's conclusion is that the first divergence almost always occurs within the first 1% of a trajectory. The next few hundred steps are the consequence of that one token.
The paper gives a concrete case: django__django-9296, Qwen3-32B, nano-agent, temperature 0. Two runs matched for 93 tokens. At token 94, one searched for the Paginator class; the other read the Django source. One patch landed in the wrong place; the other passed.
How to read a leaderboard
Before the ranking, ask five questions:
- How many runs? Is this the mean of n independent runs, or the best single run?
- Is there a standard deviation or interval? Without one, the score is a point, not an estimate.
- What temperature? Zero is not deterministic; higher temperatures usually widen the spread.
- What scaffold? The same model can move by tens of points when the harness changes.
- Is this pass@1 or pass@k? DeepSWE-preview on R2E-Gym scores 34.4% pass@1, 52.9% pass@5, and 15.5% pass^5. All three say "pass"; they answer different questions.
The paper supplies a contrast. DeepSWE-preview on nano-agent scores 31.4% ± 1.0% at temperature 1.0 and 20.4% ± 1.0% at temperature 0, an 11.0-point difference the authors treat as real. Devstral-2 on the same scaffold scores 63.5% ± 1.1% at temperature 0.2 and 63.8% ± 1.6% at temperature 0, a 0.3-point difference they treat as noise. The deciding factor is the gap relative to the standard deviation.
How many runs you need
The paper uses a power calculation. It assumes single-run scores are approximately normal; Shapiro-Wilk tests passed for 11 of 12 configurations at p ≥ 0.05. Using the observed standard deviations of 0.7% (minimum), 1.5% (median), and 1.8% (maximum):
- Detecting a 2-point improvement at p < 0.05 takes 2, 9, or 13 runs.
- Detecting a 1-point improvement takes 8, 36, or 51 runs.
- Detecting a 5-point improvement takes 2 runs at the median variance; 10 points takes 1 run at any of the three.
The paper states: "Detecting a 2% improvement at p < 0.05 with 80% power requires approximately 9 runs per agent under test, while detecting a 1% improvement requires 36 runs." The number depends on the effect you need to detect and the variance of your own setup.
What this study cannot show
First, it is a preprint: v1 on February 6, 2026, v3 on March 25, 2026, with no peer review.
Second, it covers one benchmark, three models, two scaffolds, and two temperatures. SWE-bench Verified has long software-engineering trajectories; the 2.2–6.0-point window may not transfer to short tasks or much longer ones.
Third, the paper does not link a code or data artifact. Reproducing the distribution means building the scaffolds and running 60,000 trajectories yourself.
Fourth, there is no calendar window. The paper reports ten runs per configuration but not the dates when the data was collected.
Fifth, variance at temperature 0 depends on the inference stack. The finding is that temperature 0 does not guarantee determinism, not that every temperature-0 evaluation has a standard deviation of at least 1.5 points.
Sixth, run-to-run variance and task-sampling variance are separate sources. Too few tasks and too few runs produce different errors, and they should be budgeted separately. This paper addresses only the second.
The cost of a single run
An evaluation costs more than the price of running it once. It also costs the price of believing the result. 25.58B tokens is expensive, but it buys a measurement of how unstable a single score is, which is usually cheaper than making a decision on one sample.
The next time you see "this model improved by 3 points," ask whether that is a difference between two ten-run means, or two single runs that landed at opposite ends of the same window.
This piece is part of a series on how to read AI evaluations. More writing is indexed at https://hhfa008.github.io/ai/writing/.
Disclosure: drafted with AI assistance and edited by the author.
Top comments (0)