Add a Cost Axis Before You Trust an Agent Score

A pass rate without a cost axis is an advertisement, not a measurement, and that distinction decides whether a benchmark survives review. Teams comparing coding agents publish one percentage, then discover later that the cheapest configuration reached the same number for a fraction of the spend. The repair is rarely a stronger model; it is a harness that records what each solved task actually consumed.

Scores compress a messy run into one digestible number, which is precisely why they mislead so easily. Two configurations can converge on identical pass rates while diverging by an order of magnitude in input tokens, retries, and wall-clock time. A reader who sees only the percentage cannot tell whether the result reflects capability or budget, and neither can the team that published it.

Freeze the dataset before the first run

Every task should exist as a frozen file with a stable identifier, and the manifest should carry a hash of the prompt bytes so a silent edit cannot slip into a comparison. A holdout split, untouched during tuning, is what separates a measured result from a fitted one. Without that split, every reported gain contains an unknown amount of optimization against the test set.

{"id":"t-014","prompt_sha":"9f2c1ab7d004","check":"python grader.py --case 14","expect":"PASS","timeout":180}

The manifest above is deliberately boring, because boring manifests are the ones that still reproduce six weeks later. A reviewer should be able to regenerate the exact task set from version control without asking anyone for a private dataset. If that is impossible, the comparison is anecdotal and the article describing it is marketing.

Metrics that survive a skeptical reader

Three numbers belong in every published table: solved rate on the holdout split, tokens per solved task, and median wall-clock seconds per task. The middle one is the column most teams omit, and it is the one that changes conclusions. Tokens per call rewards agents that give up early, while tokens per solved task penalizes them, which is why the denominator deserves an explicit sentence.

import json, statistics

rows = [json.loads(line) for line in open("runs/arm_a.jsonl")]
solved = [r for r in rows if r["solved"]]

report = {
    "tasks": len(rows),
    "solved_rate": round(len(solved) / len(rows), 3),
    "tokens_per_solved": round(
        sum(r["tokens_in"] + r["tokens_out"] for r in solved) / max(len(solved), 1)
    ),
    "wall_median_s": round(statistics.median(r["wall_s"] for r in rows), 1),
}
print(json.dumps(report, indent=2))

None of these metrics requires a novel statistical method, and that restraint is the point. A benchmark becomes credible when a stranger can recompute it from raw rows rather than trusting a dashboard screenshot. Publish the JSONL file behind the table, or accept that the table is unverifiable.

Controls that make two arms comparable

The harness below is untested scaffolding rather than a finished tool, so treat it as a template and adapt the usage parser to whichever provider is being measured. Its only ambition is to attach a hash and a timestamp to every row, so that later disagreement becomes a diff instead of an argument. Two arms, one task set, one grader version.

# bench.py -- untested scaffolding; adapt before trusting any printed number.
import hashlib, json, subprocess, time
from pathlib import Path

def sha(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:12]

def grade(task: dict, cwd: Path) -> bool:
    proc = subprocess.run(task["check"], cwd=cwd, shell=True,
                          capture_output=True, text=True, timeout=task.get("timeout", 180))
    return proc.returncode == 0 and task["expect"] in proc.stdout

def run_task(task: dict, arm: dict, workdir: Path) -> dict:
    started = time.monotonic()
    agent = subprocess.run(arm["command"], cwd=workdir, shell=True,
                           capture_output=True, text=True, timeout=arm.get("timeout", 600))
    wall = time.monotonic() - started
    usage = parse_usage(agent.stdout)  # provider-specific: tokens_in, tokens_out, retries
    return {
        "task_id": task["id"], "arm": arm["name"], "seed": arm["seed"],
        "prompt_sha": sha(Path("tasks") / f"{task['id']}.md"),
        "grader_sha": sha(Path("grader.py")),
        "solved": grade(task, workdir), "wall_s": round(wall, 2), **usage,
    }

Holding prompt bytes, task order, grader version, and host constant matters more than holding model temperature, because the first four are invisible failure modes. A host change alone can move median wall-clock time enough to reverse a ranking, and nobody notices when the machine swap is unrecorded. The grader_sha field costs one line and prevents an entire class of accidental cheating, including well-intentioned grader edits.

A two-arm protocol you can repeat

A minimal protocol runs the same task list through a cheap arm and a strong arm, twice each, and compares only holdout rows. Two consecutive runs of the same arm expose variance that a single run hides, and variance larger than the gap between arms means no ranking should be published at all.

python bench.py --arm cheap  --tasks tasks/ --out runs/cheap-1.jsonl --seed 7
python bench.py --arm cheap  --tasks tasks/ --out runs/cheap-2.jsonl --seed 7
python bench.py --arm strong --tasks tasks/ --out runs/strong-1.jsonl --seed 7
python bench.py --arm strong --tasks tasks/ --out runs/strong-2.jsonl --seed 7
python summarize.py runs/*.jsonl --holdout-only

Claim someone wants to make	Column that must be published	Control that must hold constant
"faster on our tasks"	median wall seconds	host, task order, timeouts
"cheaper per result"	tokens per solved task	grader version, retry policy
"better at the job"	solved rate on holdout	prompt bytes, task manifest

A table like that turns a vague boast into a falsifiable sentence, and falsifiable sentences are the only ones worth shipping to a review channel. When a claim cannot fill all three cells, it belongs in a changelog rather than a benchmark post.

Where free access changes the arithmetic

This protocol is budget-hungry for an unglamorous reason: it wants repeated runs across at least two arms, and per-run spend is the usual justification for deleting the control arm entirely. The MonkeyCode project is presented by its operator as an open-source coding agent offering free model access and a free server option, which removes both excuses at once for someone who just wants to validate the method. A free server option also eliminates the laptop-versus-CI host difference that quietly corrupts wall-clock comparisons, provided the run records where it executed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability terms, allowances, and supported models change, so any reader planning a long experiment should confirm the current terms on the project's own documentation before designing a multi-week run around them. Nothing in the harness above depends on a specific provider, which is deliberate: the method should outlive whichever free tier made it affordable this month.

Limits, and who should skip this

A cost axis systematically favours cheap configurations, so a team whose real constraint is latency or correctness on a narrow, high-stakes task may find the metric actively unhelpful. Token counts also ignore human review time, which is often the largest line item and the hardest to attribute per task. Graded synthetic tasks remain proxies, and a high solved rate on twelve frozen prompts says very little about a two-million-line repository.

Teams operating under regulated audit, certified environments, or contractual throughput guarantees should not substitute this harness for their formal validation process. The method measures relative behaviour between two configurations on one task set, and it offers no absolute claim about production reliability. Anyone needing a signed attestation should look elsewhere, because a JSONL file is not an attestation.

The practical takeaway is narrow and durable: publish the cost column, freeze the dataset, pin the grader, and run each arm twice before ranking anything. A benchmark that survives those four habits will still be quoted a year later, while a lone percentage quietly expires. If the harness above helps someone replace a suspicious number with a reproducible one, it has done its job.

Comments

[Sal Parvez | ML Systems]

The cost column is the one I run my agents on. I call it MVE, Minimum Viable Expense: the smallest spend that produces a claim a person will stamp. Your tokens-per-solved-task denominator is right for the same reason: an agent that gives up early looks cheap per call and expensive per stampable claim. Two columns I would add: who graded the row and at what evidence grade, because a grader.py PASS is a modeled claim about the task, not a measured one. And you are right that the JSONL is not an attestation. In my record the attestation is two human keys bound to a hash of the row; the file is what they signed.