Your Benchmark Forgot the Box: Attribute the Wobble Before You Blame the Model

You ran the same twelve tasks against two coding agents last month. One won by four points. You put the table in a doc and moved on.

Now ask the harder question. How much of that four-point gap belonged to the runtime rather than the model? A different machine. A cold dependency cache. A workspace that still had yesterday's build artifacts in it. None of that appears in your score, and all of it moves it.

Here is a cheap way to find out: hold the model fixed, flip only the box, and count how many tasks change verdict. If the box flips more tasks than your two candidates differ by, you do not have a model comparison yet. You have an environment comparison wearing a model's name.

The bug: you measured spread, not cause

Most of us now accept that agent scores wobble between runs. Good. That was the first lesson, and plenty of teams still have not learned it.

But knowing the wobble exists is not knowing what produced it. A re-run spread of six points could come from sampling temperature, from the grader being fuzzy, or from the machine underneath. Those three have completely different fixes, and averaging them together hides all three.

Attribution needs a controlled flip. Change one thing, hold everything else, and count verdicts that move. That is it. No statistics degree required, no leaderboard, no thirty-run sweep.

The experiment: flip the box, hold the model

You need three things: a task set small enough to run twice in an afternoon, a pass test that returns exit code 0 or not, and two environments that differ in one dimension you can name out loud.

Start with twelve tasks from your own repo. Twelve is not statistically impressive and it does not need to be. It is enough to detect a loud environment, which is the only thing this pass is for.

Step 1 — Write the pass test before you look at any model

The pass test is the whole contract. If it needs a human to read the diff and nod, it is not a pass test, it is a vibe.

bench/tasks/t01_parse_offsets/
├── prompt.md            # what the agent is asked to do
└── repo/
    ├── run_tests.sh     # exit 0 = pass, anything else = fail
    └── ...              # the checkout the agent patches

Two rules keep this honest. First, the test script must not reach the network. Second, it must clean up after itself, or your second run is really your first run plus leftovers. If your tasks cannot satisfy both rules today, fix that before reading any further.

Step 2 — Define two environments that differ in one dimension

A tab-separated file is enough, and it removes the temptation to hide a second variable in a config file nobody reads.

# bench/envs.tsv
name    base_url    workspace   reset   timeout_s
local-container http://127.0.0.1:8080   /tmp/bench/ws   wipe    120
remote-free https://YOUR-SECOND-ENV-HOST    /home/bench/ws  wipe    120

Name the dimension you are flipping in your notes. "Container vs remote host" is a dimension. "Container vs remote host, also different timeout, also different lockfile" is three dimensions and one useless experiment.

Step 3 — Run the probe

The harness below wipes the workspace per run, executes only the pass test, and appends one JSON line per (task, env, attempt). Label: not executed against your repo as written, and it will not be until you wire your own run_tests.sh.

#!/usr/bin/env python3
# bench/env_probe.py - flip the runtime, hold the model, record the verdict.
import argparse, csv, hashlib, json, pathlib, shutil, subprocess, time, uuid

def load_envs(path):
    with open(path, newline='') as f:
        return list(csv.DictReader(f, delimiter='\t'))

def env_sha(env, task):
    blob = json.dumps([env['name'], env['base_url'], task.name,
                       (task / 'repo' / 'run_tests.sh').stat().st_size],
                      sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def fresh_ws(env, task):
    ws = pathlib.Path(env['workspace']) / task.name
    if ws.exists():
        shutil.rmtree(ws)
    shutil.copytree(task / 'repo', ws)
    return ws

def run_one(env, task, attempt, run_id):
    ws = fresh_ws(env, task)
    t0 = time.monotonic()
    proc = subprocess.run(['bash', 'run_tests.sh'], cwd=ws,
                          capture_output=True, timeout=int(env['timeout_s']))
    wall_ms = int((time.monotonic() - t0) * 1000)
    return {'run_id': run_id, 'task': task.name, 'env': env['name'],
            'env_sha': env_sha(env, task), 'attempt': attempt,
            'exit_code': proc.returncode, 'passed': proc.returncode == 0,
            'wall_ms': wall_ms}

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--tasks', required=True)
    ap.add_argument('--envs', required=True)
    ap.add_argument('--out', required=True)
    ap.add_argument('--repeat', type=int, default=2)
    args = ap.parse_args()

    run_id = uuid.uuid4().hex[:12]
    tasks = sorted(p for p in pathlib.Path(args.tasks).iterdir() if p.is_dir())
    out = pathlib.Path(args.out)
    out.parent.mkdir(parents=True, exist_ok=True)
    with out.open('a') as fh:
        for attempt in range(args.repeat):
            for env in load_envs(args.envs):
                for task in tasks:
                    rec = run_one(env, task, attempt, run_id)
                    fh.write(json.dumps(rec) + chr(10))
    print('run_id=%s rows=%d' % (run_id, len(tasks) * args.repeat * 2))

if __name__ == '__main__':
    main()

Run it twice, not once. --repeat 2 gives you a within-environment signal for free: if attempt 0 and attempt 1 disagree on the same box, the box is already noisy and no cross-environment comparison will save you.

python3 bench/env_probe.py --tasks bench/tasks --envs bench/envs.tsv \
  --out runs/2026-11-08.jsonl --repeat 2

Where a free second environment honestly fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The expensive part of this experiment is not the analysis, it is standing up the second box. MonkeyCode advertises free model access and a free server option, which is exactly the two pieces you need here: a model you can hold constant while you flip the environment, and a second environment you can reach without a procurement ticket. The free token allowance was reported to me as 10 million tokens at the time of writing — treat any specific number as a snapshot with an expiry date and re-read the current terms the day you run, because quotas move.

Be precise about what that buys you, though. A shared free server is a second environment, not a reference environment. Its load is not yours to control, its disk state may persist in ways you did not choose, and nothing about its wall-clock is a latency baseline. Use it to detect verdict flips. Do not use it to publish p95 numbers, and do not treat it as the fast side of a "free is faster/slower" claim.

Step 4 — Build a flip table instead of a mean

Comparing means across environments throws away the only interesting information: which tasks moved.

#!/usr/bin/env python3
# bench/flip_table.py - paired verdicts in, flip table out.
import collections, json, sys

def main(path, env_a, env_b, attempt=0, gate=0.10):
    rows = [json.loads(l) for l in open(path) if l.strip()]
    pair = collections.defaultdict(dict)
    for r in rows:
        if r['attempt'] == attempt:
            pair[r['task']][r['env']] = r['passed']

    flips, n = 0, 0
    for task in sorted(pair):
        a, b = pair[task].get(env_a), pair[task].get(env_b)
        if a is None or b is None:
            continue
        n += 1
        flipped = a != b
        flips += flipped
        print('%-28s %-5s %-5s %s' % (task, a, b, '<- flip' if flipped else ''))

    rate = flips / n if n else 0.0
    print('tasks=%d flips=%d flip_rate=%.3f gate=%.2f' % (n, flips, rate, gate))
    sys.exit(0 if rate <= gate else 1)

if __name__ == '__main__':
    main(sys.argv[1], sys.argv[2], sys.argv[3])

python3 bench/flip_table.py runs/2026-11-08.jsonl local-container remote-free

The output shape looks like this. These numbers are placeholders to show the format, not a measurement of anything:

t02_retry_budget              True  False <- flip
t05_null_guard                True  True
t09_pagination_cursor         True  False <- flip
tasks=12 flips=2 flip_rate=0.167 gate=0.10

Two flips out of twelve is a loud box. Now compare that to your model gap: if agent A beats agent B on two tasks and the environment flips two tasks on its own, you cannot tell those apart, and neither can your reader.

Step 5 — Gate on the table

Pick the rule before you see the data, or you will rationalize whatever comes out.

Flip count (12 tasks)	Reading	Next action
0	Box is quiet enough for this task set	Proceed to the model comparison
1–2	Box noise is the same order as your effect	Widen the task set or pin harder before claiming a winner
3+	Box dominates	Fix the environment; publish nothing about models yet
Any	Same-env attempts disagree	Your pass test is flaky — fix the test first

The script exits non-zero above the gate so CI can hold the line for you. A flipped verdict that nobody reruns is just a bug with good manners.

Why these numbers are not marketing

The difference between a benchmark and a slide is what you ship alongside it. Ship the raw JSONL, the task list, the envs.tsv, and the env_sha on every row. A reader who has those can regenerate your table without asking you anything.

Also ship what is missing. Your twelve tasks came from one repo. Your environments differed in ways you may not have fully enumerated. Your pass test is binary, so it cannot see a patch that passes while breaking something else. Say all of that in the article, in the same section as the table, not in a footnote.

And state the direction of your incentive. If you sell the thing being measured, publish the harness so someone with the opposite incentive can run it. A number without a harness is an advertisement.

Limitations and who should not use this

• Latency claims. The flip table deliberately ignores timing except as a smoke signal. If you need p95 wall-clock, you need dedicated capacity and a load generator, not this.
• Security-sensitive code. Do not push a proprietary repo to a shared environment you have not read the data terms for. Use your own second box.
• Stateful workloads. Wipe-per-run environments cannot tell you anything about migrations, caches, or long-lived sessions. Different experiment.
• Teams without a binary pass test. If your tasks still need human review, spend the week writing the test. The flip table is worthless on top of a fuzzy oracle.

The method is also deliberately underpowered. Twelve tasks will catch a loud environment and will miss a quiet one. That is the correct trade for a pre-flight check you run in an afternoon, and the wrong trade for a published ranking.

What to do on Monday

Pick twelve tasks, write run_tests.sh, stand up a second box, run the probe twice. You will have a flip table before lunch and a much better question than "which agent is better": which of my numbers survive a change of machine?

If you run this, the artifact worth sharing is the flip table, not the mean.