Two Runs, Six Points Apart: Measure the Re-Run Spread Before You Compare Endpoints

You run the same 60-task suite twice against the same endpoint. First pass: 71%. Second pass: 77%. Nothing changed — not the model, not the prompts, not the tasks.

Now someone drops a screenshot of a different endpoint at 74% and asks which one wins. You cannot answer that with single runs. Before you compare anything, you need to know how much your number moves when nothing changes. That spread is your noise floor, and any delta smaller than it is a coin flip wearing a percentage sign.

This post is a repeatable method for measuring that spread, computing the smallest difference your suite can actually resolve, and deciding when a comparison is worth reporting. The reference implementation is deliberately boring. Boring is the point.

Why a single run cannot answer a comparison question

A pass rate is a summary of Bernoulli trials. Each task either passes or fails, and each repeat is a fresh draw. With 60 tasks and a true pass rate near 0.75, the run-to-run standard deviation alone is roughly 6 percentage points — so a 71% and a 77% are entirely compatible with an unchanged system.

Add harness drift on top: a timeout that clipped one task, a retry that got counted as a pass, a container that came up with a cold cache on the first run. Those do not show up in the headline number. They show up as movement you cannot explain.

The fix has two halves. First, repeat each task K times so you have a distribution instead of a point. Second, compute the minimum detectable difference (MDD) for your suite, so you know which deltas are even worth discussing.

Step 1: Freeze the task set, the harness, and the scoring rule

Write down the task IDs, the harness commit, the max-turn or max-timeout setting, and the exact command that produces a pass/fail verdict. Hash the task directory and store the hash next to your results.

# Freeze the inputs once, per experiment
find tasks/ -type f -name '*.json' | sort | xargs sha256sum | sha256sum > tasks.sha256
git -C harness rev-parse HEAD > harness.rev

Any rerun that does not reproduce tasks.sha256 is a different experiment. Do not merge its numbers into the same table — file it separately and label it. This one habit kills most "the score went up and we don't know why" tickets.

Step 2: Run K repeats per task, not K repeats per suite

This is the step people skip. If you run the whole suite five times, you get five suite-level numbers and a standard deviation computed on 5 samples — too few to say anything, and blind to which tasks are unstable.

Run each task K times instead. K=5 is a reasonable start for a 40–80 task suite; K=3 tells you almost nothing about per-task instability but is still better than 1. Write results as JSON Lines so an interrupted run can resume instead of restarting from zero.

#!/usr/bin/env bash
# run_spread.sh — illustrative driver; adapt the run command to your harness
set -euo pipefail
K=${K:-5}
OUT="runs/$(date -u +%Y%m%dT%H%M%SZ).jsonl"
mkdir -p runs

for task in tasks/*.json; do
  id=$(basename "$task" .json)
  for k in $(seq 1 "$K"); do
    start=$(date +%s)
    if timeout 900 ./harness/run --task "$task" --json > /tmp/out.json 2>/tmp/err.log; then
      status=pass
    else
      status=fail
    fi
    elapsed=$(( $(date +%s) - start ))
    printf '{"task":"%s","repeat":%d,"status":"%s","seconds":%d}\n' \
      "$id" "$k" "$status" "$elapsed" >> "$OUT"
  done
done
echo "wrote $OUT"

Two details matter more than they look. First, timeout must be a hard ceiling so a hung task cannot silently skew your wall-clock budget. Second, write the log line after the verdict, never before — a crash mid-run should leave a missing record, not a fake pass.

Step 3: Score per task, with an interval

Aggregate to a per-task pass rate p_i = passes / K, then report the suite mean with a Wilson interval. Wilson behaves sanely near 0 and 1, where the normal approximation falls apart — and agent suites always have a few tasks sitting at 0/5.

# spread.py — reference implementation (Python 3.10+), unexecuted here
import json, math, statistics
from collections import defaultdict

def load(path):
    per = defaultdict(list)
    with open(path) as f:
        for line in f:
            r = json.loads(line)
            per[r["task"]].append(r["status"] == "pass")
    return {t: sum(v) / len(v) for t, v in per.items()}

def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    c = p + z * z / (2 * n)
    m = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return ((c - m) / d, (c + m) / d)

def suite_interval(per_task, k):
    # average of per-task Wilson bounds, a conservative suite-level band
    los, his = zip(*(wilson(round(p * k), k) for p in per_task.values()))
    return statistics.mean(per_task.values()), statistics.mean(los), statistics.mean(his)

Report the band, not the point. "74.2% (95% band 69.8–78.1)" is a claim you can defend. "74.2%" invites a fight you will lose.

Step 4: Compute the minimum detectable difference

Now the number that actually governs your comparisons. Take two endpoints, A and B, both run with K repeats over the same N tasks. For each task, compute d_i = p_A_i - p_B_i. The MDD is roughly 2.8 standard errors of the mean difference — that is the two-sided alpha=0.05, 80% power threshold.

def mdd(per_a, per_b):
    tasks = sorted(set(per_a) & set(per_b))
    d = [per_a[t] - per_b[t] for t in tasks]
    n = len(d)
    se = statistics.stdev(d) / math.sqrt(n) if n > 1 else float("nan")
    return statistics.mean(d), 2.8 * se, n

A worked illustration, using made-up numbers: if mean(d) = +0.03 and MDD = 0.09, you have measured a 3-point edge on a suite that cannot resolve anything under 9 points. The honest conclusion is "no detectable difference at this sample size," not "B is slightly better."

Three levers shrink the MDD: more tasks, more repeats per task, and tasks that are less flaky. Only the first two are fully under your control.

Step 5: Use a paired rule before you call a winner

Paired designs are far more sensitive than comparing two independent suite means, because they cancel task difficulty. Count the tasks where A beats B and the tasks where B beats A; ties drop out. Then test those discordant counts.

def sign_test(wins_a, wins_b):
    n = wins_a + wins_b
    if n == 0:
        return 1.0
    k = min(wins_a, wins_b)
    tail = sum(math.comb(n, i) for i in range(k + 1)) / 2 ** n
    return min(1.0, 2 * tail)

A decision table keeps the rule out of your gut:

Observation	Report
|mean(d)| < MDD	"No detectable difference at N tasks, K repeats"
|mean(d)| ≥ MDD and sign test p < 0.05	"A ahead by X points, in the same direction on W of W+L discordant tasks"
|mean(d)| ≥ MDD but sign test p ≥ 0.05	"Led by a few heavy tasks; not a suite-wide effect"
Per-task rates cluster at 0/5 and 5/5	Suite is easy or saturated; widen difficulty before comparing

That last row is the one people ignore. A suite where most tasks are 0/5 or 5/5 has almost no resolving power, no matter how many repeats you run.

Step 6: Publish the spread, not just the mean

Ship a results table with columns for suite pass rate, Wilson band, MDD, discordant pair counts, and the input hashes from Step 1. Anyone can recompute your conclusion from that table. Nobody can recompute it from a screenshot.

Keep the raw JSONL. It is the only thing that lets you answer "was that run before or after the timeout change?" three weeks later.

Controls that quietly break repeatability

Four knobs account for most unexplained movement:

1. Retry accounting. Decide once whether a retried task counts as one attempt or two, and apply it to every endpoint.
2. Timeouts. A 900s ceiling that clips 8% of runs is a scoring parameter, not an implementation detail.
3. Temperature and sampling. Whatever you set, log it in the result file, because it is not visible in the aggregate.
4. Cold starts. The first run against any fresh runner includes setup cost the later ones do not.

Where a free runner fits

The whole method is cheap in thinking and expensive in wall-clock: N tasks × K repeats × 2 endpoints. A 60-task suite at K=5 across two endpoints is 600 runs, and most of that time is the harness waiting on a model.

That is the honest case for using free model access and a free server option to run the loop, rather than building runner infrastructure first. MonkeyCode exposes both — free model access and a free server option — as of this writing; quota, capacity, and terms are the operator's to state and change, so verify them yourself before you plan a 600-run experiment around them. I used the free server as the place to run a spread check like the one above, which keeps a noisy experiment off my laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two caveats if you go that route. Shared capacity means wall-clock variance is higher than on a box you control, so log seconds per run and treat outliers as a harness signal. And free-tier terms are a moving target: pin the terms you relied on in your run manifest alongside the task hash.

Limitations and who should skip this

K=5 over 60 tasks is not a certification. It resolves differences in the high single digits at best, and it says nothing about tasks your suite does not contain. If you need a procurement-grade claim, this method is a screening tool, not evidence.

Skip it if your test suite is itself nondeterministic in ways you cannot pin (live third-party APIs in assertions, wall-clock-dependent tests, network flakiness you have not isolated). Repeats will measure your test noise and you will confidently report it as model noise. Fix the suite first.

Also skip it if a 10-point difference is obvious to the naked eye and you only need a direction. Methodology has a cost; spend it on the close calls, where the argument actually lives.

Copy-paste checklist

• [ ] Hash the task set and record the harness commit
• [ ] Run K repeats per task, JSONL output, hard timeout
• [ ] Compute per-task pass rates and a Wilson band
• [ ] Compute MDD from paired per-task differences
• [ ] Apply the sign test before naming a winner
• [ ] Publish spread, MDD, discordant counts, and input hashes

If you run this once, you will stop arguing about three-point gaps. If you run it twice, you will start asking why your suite cannot resolve the differences you care about — which is the more useful question anyway.