Replay the Same 24 Tasks Twice Before You Rank Two Coding Agents

You drop two coding-agent scores into a Friday review. One slide says 71 percent. The other says 68 percent. Someone asks which stack to keep paying for, and the room goes quiet because nobody can name the unit behind those percentages.

Was a task one failing test, one repository, or one overnight session with hidden retries? If you cannot replay the same 24 tasks tomorrow and land inside a tiny drift band, you do not have a benchmark. You have a screenshot.

This piece is a methodology you can run yourself. It is not a leaderboard. It will not tell you which model is “best.” It will tell you whether a comparison is even legal.

Start from a scene, not a metric

Picture a checkout service with a flaky discount rule. You give Agent A and Agent B the same failing test, the same fixture files, and the same wall clock. Agent A edits three files and the test goes green. Agent B rewrites a helper, the test stays red, then a second attempt passes after it deletes an assertion.

If your scoreboard counts both as a win, you just mixed a repair with a vandalized oracle. Marketing loves that mix. You should not.

Name three objects before you write a single number.

1. The dataset: a frozen slice of tasks, each with a fixture directory and a hidden test command.
2. The oracle: the exact command that decides green, plus a checksum of the files it is allowed to read.
3. The unit of analysis: one attempt on one task, not a chat session with unbounded retries.

Until those three are pinned, pass rate is a mood.

Build a 24-task slice you can actually replay

You do not need 500 tasks to practice honesty. You need a slice small enough to re-run twice on one machine. Twenty-four is enough to feel variance and small enough to read every failure by hand.

Use a directory layout like this:

slice24/
  TASKS.md
  controls.yaml
  tasks/
    T01_discount_rule/
      prompt.md
      repo/
      oracle/
        test_discount.py
        sha256.txt
    T02_date_parse/
      ...
  runs/
    replay_a.jsonl
    replay_b.jsonl

TASKS.md is the human contract. Each row names the bug in one sentence, the language, and whether the oracle is allowed to see the production file the agent may edit. If the oracle reads a file the agent can rewrite, your “pass” is untrusted.

Keep the slice boring on purpose. Prefer bugs you could explain on a whiteboard: off-by-one, timezone parse, nil map, wrong status code, missing unique index. Skip anything that needs a live network. Skip anything whose gold patch is already in a public training dump you cannot audit.

A decision table helps you reject tasks before they contaminate the slice.

Candidate task	Keep?	Why
Failing unit test, local fixtures only	Yes	Oracle is a command you can pin
“Improve this repo” with no test	No	No oracle, only vibes
Needs GitHub auth or a paid API	No	Control plane leaks into the score
Agent may edit the test file	No	Oracle is no longer independent
Passes only after a second hidden retry	No	Unit of analysis drifted
Output depends on current date	No	Replay will move

If a row fails that table, it does not enter tasks/. It does not get a footnote. It is out.

Pin controls like they are part of the dataset

A dataset without controls is a prompt dump. Write controls.yaml first, then refuse to change it mid-bake-off.

# Proposal: example controls for a 24-task replay slice.
# Label this as a template until you fill every field on your machine.
slice_id: slice24-checkout-2026-09-13
unit_of_analysis: one_attempt_per_task
max_attempts: 1
wall_clock_sec: 180
cpus: 2
memory_mb: 2048
network: denied
writable_paths: ["repo/"]
oracle_cmd: "python -m pytest oracle/ -q"
oracle_sha256_file: oracle/sha256.txt
seed_policy: fixed
language_toolchain: "python3.11"
forbidden_edits: ["oracle/", "controls.yaml"]
replay_required: 2
max_replay_drift: 0

Read that file out loud. The important lines are max_attempts, network, and replay_required. One attempt means you are scoring the first patch, not the best patch after a human babysits the loop. Denied network means the agent cannot shop for answers. Two replays with zero allowed drift means the environment is a scientific instrument, not a laptop that happened to be free that afternoon.

If you later raise max_attempts from 1 to 3, you started a new study. Do not append those runs onto the old JSONL and call it an update.

Score paired deltas, not a headline percent

You will still compute a pass count. You will not lead with it.

For each task t, store four facts:

• pass_a: 1 if Agent A’s attempt made the pinned oracle green
• pass_b: 1 if Agent B did
• oracle_intact: 1 if oracle/ still matches sha256.txt
• replay_match: 1 if attempt 1 and attempt 2 agreed for that agent

Then report the paired picture:

• Wins for A: tasks where A passed and B failed, oracle intact
• Wins for B: the reverse
• Ties: both passed or both failed
• Void: oracle mutated, timeout disagreement, or replay mismatch

A void is not a loss. A void is a broken instrument. If more than two of 24 tasks void, stop ranking and fix the harness.

The number you may quote is narrow: “On slice24, with one attempt and a frozen oracle, A won 5, B won 3, 14 tied, 2 void.” That sentence is ugly. It is also hard to turn into an ad.

A harness you can run twice

The following is a labeled proposal. It does not claim production metrics. Swap run_agent with your real CLI. Keep the oracle check. If you skip the checksum, the rest of the file is theater.

# proposal_harness.py — replay gate for a 24-task slice
from __future__ import annotations

import hashlib, json, subprocess, time
from pathlib import Path

SLICE = Path("slice24")
WALL_CLOCK = 180

def sha256_tree(root: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(root.rglob("*")):
        if p.is_file():
            h.update(p.relative_to(root).as_posix().encode())
            h.update(p.read_bytes())
    return h.hexdigest()

def oracle_intact(task: Path) -> bool:
    expected = (task / "oracle" / "sha256.txt").read_text().strip()
    return sha256_tree(task / "oracle") == expected

def run_oracle(task: Path) -> bool:
    proc = subprocess.run(
        ["python", "-m", "pytest", str(task / "oracle"), "-q"],
        cwd=task / "repo",
        capture_output=True,
        timeout=WALL_CLOCK,
    )
    return proc.returncode == 0

def run_agent(agent_id: str, task: Path) -> None:
    prompt = (task / "prompt.md").read_text()
    # Replace this stub with your agent CLI. Keep the timeout.
    subprocess.run(
        ["your-agent", "--id", agent_id, "--prompt", prompt, "--cwd", str(task / "repo")],
        timeout=WALL_CLOCK,
        check=False,
    )

def one_attempt(agent_id: str, task: Path) -> dict:
    if not oracle_intact(task):
        return {"task": task.name, "void": "oracle_before"}
    t0 = time.monotonic()
    run_agent(agent_id, task)
    elapsed = round(time.monotonic() - t0, 3)
    if not oracle_intact(task):
        return {"task": task.name, "agent": agent_id, "void": "oracle_mutated", "sec": elapsed}
    passed = run_oracle(task)
    return {"task": task.name, "agent": agent_id, "pass": int(passed), "sec": elapsed, "void": None}

def replay_agent(agent_id: str) -> list[dict]:
    rows = []
    for task in sorted((SLICE / "tasks").iterdir()):
        first = one_attempt(agent_id, task)
        second = one_attempt(agent_id, task)
        if first.get("pass") != second.get("pass") or first.get("void") or second.get("void"):
            rows.append({"task": task.name, "agent": agent_id, "void": "replay_drift"})
        else:
            rows.append(first)
    return rows

def paired_report(a: list[dict], b: list[dict]) -> dict:
    by_a = {r["task"]: r for r in a}
    wins = {"A": 0, "B": 0, "tie": 0, "void": 0}
    for r in b:
        t = r["task"]
        left, right = by_a[t], r
        if left.get("void") or right.get("void"):
            wins["void"] += 1
        elif left.get("pass") and not right.get("pass"):
            wins["A"] += 1
        elif right.get("pass") and not left.get("pass"):
            wins["B"] += 1
        else:
            wins["tie"] += 1
    return wins

if __name__ == "__main__":
    a_rows = replay_agent("A")
    b_rows = replay_agent("B")
    Path("slice24/runs/replay_a.jsonl").write_text(
        "\n".join(json.dumps(r) for r in a_rows) + "\n"
    )
    Path("slice24/runs/replay_b.jsonl").write_text(
        "\n".join(json.dumps(r) for r in b_rows) + "n"
        if False else "\n".join(json.dumps(r) for r in b_rows) + "\n"
    )
    print(json.dumps(paired_report(a_rows, b_rows), indent=2))

Run it twice on purpose. The outer replay_agent already doubles each task. You still re-launch the process after a reboot. If the JSONL changes, your machine is part of the score. Fix the machine before you argue about models.

Useful commands while you debug the instrument:

find slice24/tasks -name sha256.txt -print
python -m pytest slice24/tasks/T01_discount_rule/oracle -q
sha256sum slice24/controls.yaml
python proposal_harness.py
diff -u slice24/runs/replay_a.jsonl slice24/runs/replay_a.jsonl.bak

If diff is not empty, you do not discuss Agent A versus Agent B. You discuss clocks, caches, and pytest plugins.

Why these numbers are not marketing

A vendor deck wants one percentage and a bold winner. This slice refuses that in four ways.

First, n equals 24 paired tasks, not 24 independent coin flips. Several tasks will share a language and a style of bug. You may still count wins. You may not pretend you have a population study.

Second, the unit is one attempt. Hidden retries, extra tool calls after timeout, and “I nudged it in chat” are protocol violations. They belong in a different paper.

Third, void tasks exist. Marketing absorbs voids into the denominator or drops them. You surface them. A method that cannot fail in public is not a method.

Fourth, replay drift is a stop condition, not a footnote. If the same agent on the same fixture flips pass/fail, the score is environmental noise. Publishing a mean over noise is how leaderboards rot.

You can add cost later. You can add failure taxonomy later. None of that is legal until replay is boring.

Where a free server actually helps

Methodology work burns tokens on scaffolding: broken prompts, oracle mistakes, path bugs. That traffic should not touch a production budget.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need model access and a machine without standing up your own GPU, MonkeyCode’s free model access and free server option are enough to host this 24-task slice while you harden the harness. Use that only for the replay loop. Move any claim you would show a finance team onto hardware you control and a slice you can checksum.

That is the whole product note.

Limitations, and who should not use this

This approach is wrong for several honest jobs.

Do not use a 24-task slice to pick a company-wide coding agent. The interval is wide, the tasks are local, and language coverage is thin. Do not use it for multi-hour refactors, GUI work, or anything that needs the public internet. Do not use it if your oracle is “a reviewer liked the patch.” Human taste is not a sha256.

Do not compare agents that do not share a tool surface. If Agent A can run pytest and Agent B can only emit a diff, you measured the toolchain. Say that, or stop.

Do not publish the pass count without the controls file, the slice id, the void count, and the replay rule. A number without those four is a caption.

If your goal is a press quote by Monday, this workflow will slow you down. That is the point. Rankings are cheap. Replay is the filter that keeps you from buying a percentage that cannot sit still.