Your Coding-Agent Benchmark Needs a Run Manifest, Not Another Leaderboard Row

A colleague pasted a screenshot into our channel last month: "agent X solves 78% of our internal suite." Someone asked which suite. Nobody knew. The screenshot had a bar, a percentage, and a logo, and it took three people an hour to establish that it was not comparable to anything we had ever run.

That is the normal failure mode of agent benchmarking. Not fraud. Not even sloppiness. Just a number published without the envelope that produced it.

You cannot fix that with a better leaderboard. You fix it by making the number's preconditions a machine-readable artifact that a reviewer can diff. Below is the contract I now require before I read anyone's percentage, including my own.

The number is the last line of the report, not the first

A pass rate is a summary of a run. A run is a bundle of decisions: which tasks, which snapshot, which retry policy, which token budget, which scoring script. Change any of them and the percentage means something else.

So write the bundle down first, hash it, and only then compute the number. If two runs disagree on any envelope field, the correct output is not a delta. It is a refusal.

Step 1: Freeze the slice before you run anything

Write your task IDs to a file, sort them, and hash the file. Not the directory. Not "the SWE tasks we usually use." A single file with stable, sorted content.

# tasks.txt — one stable task id per line, no comments
printf '%s\n' task-0011 task-0023 task-0047 > tasks.txt
sort -o tasks.txt tasks.txt
sha256sum tasks.txt | cut -d' ' -f1
# -> 3f9a...  paste this into the manifest

This one habit kills most benchmark disputes. If a task is added, the hash changes, and the hash change is visible in review. Nobody can quietly improve their score by dropping the tasks that fail.

Pick the slice for a stated reason and write that reason down too. A slice chosen because it was convenient is still fine — as long as the manifest says so and the conclusion stays inside that boundary.

Step 2: Declare the envelope in a manifest

Here is the file I use. It is deliberately boring, because boring files survive contact with reviewers.

{
  "run_id": "2026-10-18-acme-v3-r1",
  "started_at_utc": "2026-10-18T09:12:00Z",
  "operator": "casey",
  "task_slice_sha256": "3f9a...",
  "agent_id": "acme-cli",
  "harness_commit": "9c1b7de",
  "model_id": "provider/model-family",
  "model_snapshot": "<the pinned version string the provider returns>",
  "temperature": 0.0,
  "seed": 7,
  "max_output_tokens": 4096,
  "budget_tokens_per_task": 60000,
  "tool_timeout_s": 120,
  "retry_policy": "no retries on test failure; 2 retries on HTTP 5xx",
  "env_image_digest": "sha256:...",
  "scoring_script_sha256": "...",
  "runner": "free-server",
  "provider_region": "unspecified"
}

Two details earn their keep. model_snapshot is whatever version string the provider exposes that day — a family name alone is not an envelope field, because families move under you. retry_policy matters more than temperature in practice; an agent that gets two extra attempts at a failing test suite can look materially better than the same agent with one.

Step 3: Make the checker refuse bad comparisons

This is the whole artifact. Everything else is supporting cast.

#!/usr/bin/env python3
"""check_comparable.py — refuse to diff two runs with different envelopes."""
import json, sys
from pathlib import Path

ENVELOPE = {
    "task_slice_sha256", "agent_id", "harness_commit", "model_id",
    "model_snapshot", "temperature", "seed", "max_output_tokens",
    "budget_tokens_per_task", "tool_timeout_s", "retry_policy",
    "env_image_digest", "scoring_script_sha256",
}
PROVENANCE = {"run_id", "started_at_utc", "operator", "runner", "provider_region"}

def load(p):
    return json.loads(Path(p).read_text())

def main(a, b):
    ma, mb = load(a), load(b)
    blocking = sorted(k for k in ENVELOPE if ma.get(k) != mb.get(k))
    drift = sorted(k for k in PROVENANCE if ma.get(k) != mb.get(k))
    for k in blocking:
        print(f"BLOCK {k}: {ma.get(k)!r} != {mb.get(k)!r}")
    for k in drift:
        print(f"note  {k}: {ma.get(k)!r} != {mb.get(k)!r}")
    if blocking:
        print(f"\nnot comparable: {len(blocking)} envelope field(s) differ")
        return 3
    print("\ncomparable envelope: diff the outcomes")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2]))

Exit code 3 is the important line. In CI it fails the job. In review it is the comment that says "these two numbers are different experiments."

Field that differs	Verdict
task_slice_sha256	Not comparable — different question
model_snapshot	Not comparable — different subject
temperature, seed, budget_tokens_per_task	Not comparable — different conditions
env_image_digest, scoring_script_sha256	Not comparable — different referee
runner, provider_region, started_at_utc	Comparable, note it in the write-up

The last row is where people over-correct. Recording that run A used a laptop and run B used a free server is useful context, not a disqualification — unless your conclusion depends on latency or hardware.

Step 4: Report discordant pairs, not just averages

Two runs over the same slice produce paired outcomes. Averages throw that structure away.

#!/usr/bin/env python3
"""paired.py — per-task deltas with a seeded bootstrap interval."""
import json, random, sys

def load(p):
    return {r["task_id"]: r for r in json.load(open(p))}

def main(pa, pb, iters=10000, seed=7):
    a, b = load(pa), load(pb)
    shared = sorted(set(a) & set(b))
    if set(a) != set(b):
        print(f"WARN task sets differ; using {len(shared)} shared tasks")
    deltas = [int(b[t]["passed"]) - int(a[t]["passed"]) for t in shared]
    a_pass_b_fail = sum(1 for t in shared if a[t]["passed"] and not b[t]["passed"])
    b_pass_a_fail = sum(1 for t in shared if b[t]["passed"] and not a[t]["passed"])
    n, rng = len(deltas), random.Random(seed)
    boots = sorted(sum(deltas[rng.randrange(n)] for _ in range(n)) / n
                   for _ in range(iters))
    lo, hi = boots[int(0.025 * iters)], boots[int(0.975 * iters)]
    print(f"n={n} delta={sum(deltas)/n:+.3f} 95%CI=[{lo:+.3f},{hi:+.3f}]")
    print(f"A passed / B failed: {a_pass_b_fail}")
    print(f"B passed / A failed: {b_pass_a_fail}")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

With 20 tasks and a +0.150 delta, expect something like 95%CI=[-0.050,+0.350] and a discordant split around 2 / 5. That interval straddles zero. The honest sentence is "directionally positive, not established" — and the discordant counts tell you which tasks to read first.

The numbers above are an expected output shape for the script, not measurements from a real run. Run it on your own outcomes before quoting anything.

Step 5: Ship the manifest next to the number

A pass rate without a manifest is a rumor with a percent sign. Publish run.manifest.json, outcomes.json, and the exact command line. A reviewer should be able to check comparability in under a minute without asking you a question.

One practical shortcut: you need a second environment to test whether your envelope survives a change of runner. That is a cheap check with a free-tier setup — run the same slice on your laptop and on rented-or-free infrastructure, then feed both manifests to the checker. MonkeyCode's free model access and free server option are one way to get that second environment without a budget conversation; the operator tells me the free tier is described as a 10M-token allowance plus a free server, which you should re-verify on the current pricing page before you size a run around it, because allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What this does not prove

A matching envelope proves comparability, not quality. Your 20 tasks are still 20 tasks, and the bootstrap interval is indicative rather than inferential at that size.

Slice selection bias survives every hash. If you picked tasks your agent happens to be good at, the manifest will faithfully document a flattering and useless number. Negative controls and pre-registration address that; the manifest only stops you from lying by accident.

Free tiers and hosted runners also add variance you cannot fully pin: quota throttling, shared capacity, and provider-side version drift between two runs on different days. Record the timestamp and rerun the manifest check before you combine them.

Who should skip this

Skip the manifest if you are doing a one-off smoke test to decide whether an agent is worth an afternoon. Skip it for a demo. Skip it if your conclusion is "this looks promising, let's try it on real work."

Adopt it the moment a number leaves your terminal. The first time someone quotes your percentage in a planning meeting, the manifest is the difference between a measurement and a slogan.