Write the Benchmark Card First: Dataset, Metrics, and Controls for Coding Agents

You drop a slide into standup: “Agent B hit 71% on the internal coding suite.” Someone in the back asks which suite version, how many tasks died in the toolchain, and whether retries counted as extra tries or extra luck. You do not have those answers in the deck. You have a number that already left the lab.

That is the moment a coding-agent score turns into marketing. Not because you meant to sell anything. Because the methods were still in a scratch buffer while the headline walked into a meeting.

This article is a protocol for stopping that leak. You write a benchmark card before you quote a pass rate. The card names the dataset, the metrics, and the controls. The number is allowed to travel only as a pointer to that card.

The failure mode you are actually debugging

Most leaked scores share the same shape. A mixed bag of tickets. A runner that retries on timeout. A pass rate with no denominator. A model swap mid-week that nobody recorded.

You cannot audit that. You can only argue about it. The fix is not a prettier chart. It is a document that makes the chart unreadable without the methods.

Treat the card as the publishable object. The pass rate is a derived field. If the card is missing, the derived field is not a result. It is a caption.

What belongs on the card

Keep the card short enough to paste into a PR. Long enough that a skeptic can reproduce the claim without Slack archaeology.

1. Dataset identity: name, version, hash, license, and how items were sampled.
2. Task contract: language, test command, timeout, and what “pass” means in one sentence.
3. Metric set: every number you might quote, each with a denominator and an exclusion rule.
4. Controls: model endpoint, decoding settings, token budget, seed, retry policy, hardware class.
5. Exclusion log: infra failures, missing tests, and tasks pulled after freeze.

If a field is unknown, write unknown. Do not silently omit it. Omission is how 71% becomes a brand.

Step 1 — Freeze a dataset you can name

Do not point at “our internal tickets.” Point at a directory with a manifest. You want a reader to check out one commit and get the same tasks.

bench/
  CARD.yaml
  tasks/
    001_parse_csv/
      prompt.md
      repo/
      tests/
    002_fix_off_by_one/
      prompt.md
      repo/
      tests/
  MANIFEST.sha256

Hash the tree after you freeze it. If you add a task later, that is a new version, not a silent boost.

find bench/tasks -type f | sort | xargs sha256sum > bench/MANIFEST.sha256
sha256sum bench/MANIFEST.sha256

Sample on purpose. A 12-task grab bag from last sprint is a demo. A 12-task set with strata is a study. Write the strata on the card: language, bug class, and whether tests existed before the prompt.

Proposed strata (label these as a design choice, not as measured truth):

• lang: python | ts | go
• kind: fail-to-compile | wrong-output | missing-edge
• tests_preexist: yes | no

You do not need hundreds of items to be honest. You need a frozen identity and a sampling rule a stranger can repeat.

Step 2 — Choose metrics that refuse a single headline

One pass rate is a poster. A metric set is a lab notebook. Define at least four numbers, and refuse to print any of them alone.

# proposed metric definitions — not executed claims
METRICS = {
    "n_attempted": "tasks the harness started",
    "n_scored": "attempted minus infra exclusions",
    "pass_strict": "scored tasks whose tests exited 0 with no retry",
    "timeout_rate": "scored tasks that hit the wall clock",
    "compile_fail_rate": "scored tasks that never reached tests",
    "tokens_per_pass": "sum(tokens) / max(pass_strict_count, 1)",
}

Notice what is missing: a blended “score.” You can compute one later for a appendix. You do not lead with it. Leading with a blend is how unequal tasks, unequal budgets, and retries get laundered into a rank.

Write the exclusion rule in English on the card:

• Toolchain crash, runner OOM, and lost network: exclude from n_scored, count in n_infra.
• Agent produced a patch that fails tests: scored failure. Not infra.
• Agent asked for a retry after timeout: still one attempt unless the card pre-declares max_retries.

If you cannot state the denominator, you do not have a metric. You have a vibe.

Step 3 — Record controls as data, not as folklore

Controls are the difference between a result and a story about a result. Put them in the same JSONL row as the outcome.

# proposed row schema for results.jsonl
row = {
    "card_id": "coding-mini-2026-09-22",
    "card_hash": "…",
    "task_id": "001_parse_csv",
    "stratum": {"lang": "python", "kind": "wrong-output"},
    "endpoint": "unknown-until-run",
    "temperature": 0.0,
    "max_output_tokens": 4096,
    "timeout_s": 180,
    "max_retries": 0,
    "seed": 7,
    "outcome": "fail_tests",  # pass | fail_tests | fail_compile | timeout | infra
    "tokens_in": None,
    "tokens_out": None,
    "exit_code": 1,
}

Fill endpoint at run time. Do not hard-code a model name you have not actually called. If the free endpoint you used has no public model string, store the URL class and the date, not a nickname from memory.

Retry policy belongs here, not in a wrapper script you forgot. max_retries: 0 is a control. we reran the flaky ones is contamination.

Step 4 — Run the harness where the methods can live

You need a box that can clone the frozen tree, call a model, run tests, and write JSONL. A laptop works until the lid closes. A small always-on server works better because the card, the logs, and the run live together.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access and a free server option. Those two facts are the only product claims this protocol needs. If you are already paying for an endpoint, keep it. The card does not care who serves the tokens. It cares that the endpoint is written down.

A boring run looks like this:

python harness.py --card bench/CARD.yaml --out runs/2026-09-22.jsonl
python summarize.py --in runs/2026-09-22.jsonl --require-card-hash

The second command should refuse to print a pass rate if card_hash is missing or if n_scored is zero. That refusal is the whole point.

Artifact: a card and a runner that will not print a poster

Save this as bench/CARD.yaml. It is a template. Replace every unknown before you quote a number.

card_id: coding-mini-2026-09-22
dataset:
  name: coding-mini
  version: "0.1.0"
  manifest_sha256: unknown
  sampling: "manual strata; not a random sample of production tickets"
  n_tasks: 12
  license: unknown
task_contract:
  pass: "repository tests exit 0"
  timeout_s: 180
  max_retries: 0
metrics:
  - pass_strict
  - timeout_rate
  - compile_fail_rate
  - n_scored
  - n_infra
controls_required:
  - endpoint
  - temperature
  - max_output_tokens
  - seed
exclusions:
  infra: ["runner_oom", "clone_fail", "network_lost"]
publication_rule: "do not print pass_strict without n_scored and card_hash"

Then a small harness. This is proposed code. It does not report a measured leaderboard. It shows how a run becomes unprintable until the card is complete.

#!/usr/bin/env python3
"""Proposed coding-agent harness. Not a published score."""
from __future__ import annotations

import argparse, hashlib, json, subprocess, sys, time
from pathlib import Path

try:
    import yaml
except ImportError:
    yaml = None


def load_card(path: Path) -> dict:
    text = path.read_text()
    if yaml:
        card = yaml.safe_load(text)
    else:
        raise SystemExit("install pyyaml or convert CARD.yaml to json")
    required = ["card_id", "dataset", "task_contract", "metrics", "publication_rule"]
    missing = [k for k in required if k not in card]
    if missing:
        raise SystemExit(f"card missing fields: {missing}")
    if card["dataset"].get("manifest_sha256") in (None, "unknown"):
        raise SystemExit("refuse to run: freeze MANIFEST.sha256 first")
    return card


def card_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]


def run_tests(task_dir: Path, timeout_s: int) -> dict:
    tests = task_dir / "tests"
    if not tests.exists():
        return {"outcome": "infra", "reason": "missing_tests", "exit_code": None}
    try:
        proc = subprocess.run(
            ["pytest", "-q"],
            cwd=task_dir,
            timeout=timeout_s,
            capture_output=True,
        )
    except subprocess.TimeoutExpired:
        return {"outcome": "timeout", "reason": "wall_clock", "exit_code": None}
    if proc.returncode == 0:
        return {"outcome": "pass", "reason": "tests_ok", "exit_code": 0}
    return {"outcome": "fail_tests", "reason": "tests_nonzero", "exit_code": proc.returncode}


def summarize(rows: list[dict]) -> dict:
    scored = [r for r in rows if r["outcome"] != "infra"]
    n_scored = len(scored)
    def rate(name: str) -> float | None:
        if n_scored == 0:
            return None
        return round(sum(1 for r in scored if r["outcome"] == name) / n_scored, 4)
    return {
        "n_attempted": len(rows),
        "n_scored": n_scored,
        "n_infra": sum(1 for r in rows if r["outcome"] == "infra"),
        "pass_strict": rate("pass"),
        "timeout_rate": rate("timeout"),
        "compile_fail_rate": rate("fail_compile"),
    }


def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--card", type=Path, required=True)
    p.add_argument("--out", type=Path, required=True)
    p.add_argument("--endpoint", default="unknown")
    args = p.parse_args()

    card = load_card(args.card)
    digest = card_hash(args.card)
    timeout_s = int(card["task_contract"]["timeout_s"])
    rows = []
    for task_dir in sorted((args.card.parent / "tasks").iterdir()):
        if not task_dir.is_dir():
            continue
        result = run_tests(task_dir, timeout_s)
        rows.append({
            "card_id": card["card_id"],
            "card_hash": digest,
            "task_id": task_dir.name,
            "endpoint": args.endpoint,
            "timeout_s": timeout_s,
            "max_retries": card["task_contract"].get("max_retries", 0),
            **result,
            "ts": int(time.time()),
        })

    args.out.parent.mkdir(parents=True, exist_ok=True)
    with args.out.open("w") as f:
        for row in rows:
            f.write(json.dumps(row) + "\n")

    stats = summarize(rows)
    if stats["n_scored"] == 0 or stats["pass_strict"] is None:
        print("unprintable: n_scored is 0", file=sys.stderr)
        sys.exit(2)
    print(json.dumps({"card_hash": digest, **stats}, indent=2))


if __name__ == "__main__":
    main()

Run it dry against an empty tree and you should get a refusal, not a zero. That is the behavior you want in CI.

python harness.py --card bench/CARD.yaml --out runs/dry.jsonl; echo exit:$?

Why the numbers are not marketing

Marketing needs a rank, a round number, and a missing asterisk. This protocol attacks all three.

The dataset is versioned, so you cannot quietly add easy tasks. The metrics travel as a set, so “71%” has to drag n_scored and timeout_rate with it. The controls sit on every row, so a mid-run endpoint swap shows up as two experiments, not one lucky curve. The harness will not print if the card is incomplete.

You still can misuse the output. People will screenshot the pass rate. Your job is to make that screenshot look unfinished: no hash, no denominator, no strata. Unfinished numbers are easier to challenge in review than polished ones.

Do not average strata into a single trophy. Report python and TypeScript separately if both exist. A combined mean is how a six-task toy language subsidizes a hard systems bug.

Limitations, and who should not use this

This is a methods scaffold. It is not a substitute for a reviewed benchmark, and it does not produce a vendor ranking.

Skip it if you need statistically tight comparisons across labs. Twelve frozen tasks will not give you that, even with perfect logging. Skip it if your tests are non-deterministic or require licensed cloud APIs you cannot replay. Skip it if you want a public leaderboard more than you want a paper trail.

Free model access and a free server will not make a small card into a large study. They only keep the runner and the notebook in one place. Token quality, rate limits, and hardware class are still controls you must record. If those controls are unknown, the score stays unprintable.

The protocol also will not stop a reader who ignores the card. It only makes ignoring the card visible.

What you do on Monday

Pick twelve tasks you already own. Freeze the tree. Fill every unknown you can. Run the harness once with retries disabled. Paste the JSON summary into the PR, not into the slide.

If the summary refuses to print, that is success. You found a hole in the methods before the number found an audience.

When you need an overnight box so the card and the JSONL sit together, MonkeyCode’s free model access and free server option are a reasonable place to park the runner. Keep the headline off the homepage until card_hash and n_scored travel with it.