A Reproducible Harness for Comparing Coding Agents Before You Trust Their Numbers

Agent benchmarks are everywhere this week, and most of them are not comparable to anything. A vendor posts a solve rate, a blog post posts another, and neither tells you the model, the retry policy, the token budget, or whether the agent was allowed to run the tests it claimed to fix.

I got tired of arguing about numbers I couldn't reproduce, so I built a small harness that scores any coding agent against the same fixed task set, the same time budget, and the same grading script. This post walks through the harness so you can run it against whatever agent or model you have access to — including free tiers — and get numbers that at least mean something within your own setup.

The rules that make numbers comparable

A comparison between agents (or between models behind the same agent) is only honest if:

1. Same task set, frozen. Write the tasks down, pin them in a repo, never edit them mid-experiment.
2. Same environment. Same container image, same dependency lockfile, same network access (ideally none).
3. Same budget. Same wall-clock limit and same max tool-call count per task.
4. Same grader. An automated script, not your eyeballs. pytest exit codes beat vibes.
5. N > 1 runs. Agents are nondeterministic. One run is an anecdote.

Most published agent numbers violate at least three of these.

The harness

The artifact: a ~60-line Python script that runs a task directory through an agent CLI, captures the patch, and grades it. It's agent-agnostic — you provide a shell command template.

#!/usr/bin/env python3
"""agent-harness.py — minimal reproducible coding-agent scorer.
Usage: python agent-harness.py --cmd "myagent run {task_file}" --tasks tasks/ --runs 3
"""
import argparse, json, shutil, subprocess, tempfile, time
from pathlib import Path

def run_task(cmd_tpl: str, task_dir: Path, timeout: int) -> dict:
    workdir = Path(tempfile.mkdtemp(prefix="agent-run-"))
    shutil.copytree(task_dir / "repo", workdir / "repo")
    task_file = workdir / "TASK.md"
    shutil.copy(task_dir / "TASK.md", task_file)

    t0 = time.monotonic()
    proc = subprocess.run(
        cmd_tpl.format(task_file=task_file, workdir=workdir / "repo"),
        shell=True, cwd=workdir / "repo", capture_output=True, text=True,
        timeout=timeout,
    )
    elapsed = time.monotonic() - t0

    grade = subprocess.run(
        ["bash", str(task_dir / "grade.sh")],
        cwd=workdir / "repo", capture_output=True, text=True, timeout=120,
    )
    result = {
        "task": task_dir.name,
        "passed": grade.returncode == 0,
        "elapsed_s": round(elapsed, 1),
        "agent_exit": proc.returncode,
    }
    shutil.rmtree(workdir, ignore_errors=True)
    return result

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--cmd", required=True, help="command template with {task_file} and {workdir}")
    p.add_argument("--tasks", default="tasks")
    p.add_argument("--runs", type=int, default=3)
    p.add_argument("--timeout", type=int, default=600)
    args = p.parse_args()

    results = []
    for task_dir in sorted(Path(args.tasks).iterdir()):
        if not task_dir.is_dir():
            continue
        for i in range(args.runs):
            r = run_task(args.cmd, task_dir, args.timeout)
            r["run"] = i
            results.append(r)
            print(json.dumps(r))

    passed = sum(r["passed"] for r in results)
    print(f"\npass rate: {passed}/{len(results)} = {passed/len(results):.1%}")

if __name__ == "__main__":
    main()

Each task is a directory:

tasks/
  fix-off-by-one/
    TASK.md        # the prompt the agent sees
    repo/          # broken code + hidden test expectations
    grade.sh       # runs tests, exit 0 = pass

A minimal grade.sh:

#!/usr/bin/env bash
python -m pytest tests/ -q --tb=no

Crucially, tests/ lives in the task directory and gets copied in after the agent finishes — not shown here to keep the snippet short, but the point stands: the agent never sees the grading tests. That one detail eliminates the most common form of accidental benchmark cheating.

Where free models and a free server fit

Running 3–5 runs per task across a decent task set burns a lot of tokens, which is exactly why most people never do controlled comparisons — they try an agent twice and form an opinion.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access and a free server option, and that's the combination that made this harness practical for me: I pointed the --cmd template at a MonkeyCode-driven run, let the free server absorb the repeated executions, and reserved paid usage for the final confirmation pass. I am deliberately not quoting model names, quotas, or throughput here, because those change and any number I printed would be stale by the time you read this — check the current offering before designing around it.

A workflow that worked well:

Stage	Where it runs	Why
Task authoring	Local	You want full control of the frozen task repo
Bulk scoring (N runs)	Free models on the free server	Cost of iteration is ~zero; nondeterminism needs volume
Final confirmation	Your production model/setup	Validates the free-tier result generalizes

If the free-tier result and the production result disagree, that disagreement is a finding — it usually means the task is sensitive to model capability, and you should report both numbers rather than averaging them.

Limitations, and who shouldn't bother

• Pass/fail hides quality. An agent can pass tests with an unmaintainable patch. Extend grade.sh with a lint gate or a diff-size cap if that matters to you.
• Your task set is your bias. Ten tasks you wrote yourself measure what you care about, not universal agent quality. That's fine — just don't publish the number as if it were universal.
• Free tiers change. Treat free access as a way to iterate on methodology, not as infrastructure to depend on. If your comparison pipeline breaks when the free option goes away, the pipeline was the product, not the comparison.
• If you only have one agent and one model, skip the harness. You don't need statistics to answer "does this help me," you need a week of real usage.

Closing

The useful output of this exercise isn't a leaderboard — it's a personal baseline: given my tasks, my budget, and my grading, which setup actually finishes the work? If you build your own task set from bugs you've actually hit, I'd genuinely like to hear what your pass rates look like and where the agents fell over.

Comments

[Alex Shev]

The phrase "before you trust their numbers" is the key. Agent comparisons are easy to make look scientific while hiding prompt leakage, unequal context, flaky tests, or different failure-handling rules.