DEV Community: Casey Zhang

Write the Benchmark Card First: Dataset, Metrics, and Controls for Coding Agents

Casey Zhang — Tue, 22 Sep 2026 14:30:56 +0000

You drop a slide into standup: “Agent B hit 71% on the internal coding suite.” Someone in the back asks which suite version, how many tasks died in the toolchain, and whether retries counted as extra tries or extra luck. You do not have those answers in the deck. You have a number that already left the lab.

That is the moment a coding-agent score turns into marketing. Not because you meant to sell anything. Because the methods were still in a scratch buffer while the headline walked into a meeting.

This article is a protocol for stopping that leak. You write a benchmark card before you quote a pass rate. The card names the dataset, the metrics, and the controls. The number is allowed to travel only as a pointer to that card.

The failure mode you are actually debugging

Most leaked scores share the same shape. A mixed bag of tickets. A runner that retries on timeout. A pass rate with no denominator. A model swap mid-week that nobody recorded.

You cannot audit that. You can only argue about it. The fix is not a prettier chart. It is a document that makes the chart unreadable without the methods.

Treat the card as the publishable object. The pass rate is a derived field. If the card is missing, the derived field is not a result. It is a caption.

What belongs on the card

Keep the card short enough to paste into a PR. Long enough that a skeptic can reproduce the claim without Slack archaeology.

Dataset identity: name, version, hash, license, and how items were sampled.
Task contract: language, test command, timeout, and what “pass” means in one sentence.
Metric set: every number you might quote, each with a denominator and an exclusion rule.
Controls: model endpoint, decoding settings, token budget, seed, retry policy, hardware class.
Exclusion log: infra failures, missing tests, and tasks pulled after freeze.

If a field is unknown, write unknown. Do not silently omit it. Omission is how 71% becomes a brand.

Step 1 — Freeze a dataset you can name

Do not point at “our internal tickets.” Point at a directory with a manifest. You want a reader to check out one commit and get the same tasks.

bench/
  CARD.yaml
  tasks/
    001_parse_csv/
      prompt.md
      repo/
      tests/
    002_fix_off_by_one/
      prompt.md
      repo/
      tests/
  MANIFEST.sha256

Hash the tree after you freeze it. If you add a task later, that is a new version, not a silent boost.

find bench/tasks -type f | sort | xargs sha256sum > bench/MANIFEST.sha256
sha256sum bench/MANIFEST.sha256

Sample on purpose. A 12-task grab bag from last sprint is a demo. A 12-task set with strata is a study. Write the strata on the card: language, bug class, and whether tests existed before the prompt.

Proposed strata (label these as a design choice, not as measured truth):

lang: python | ts | go
kind: fail-to-compile | wrong-output | missing-edge
tests_preexist: yes | no

You do not need hundreds of items to be honest. You need a frozen identity and a sampling rule a stranger can repeat.

Step 2 — Choose metrics that refuse a single headline

One pass rate is a poster. A metric set is a lab notebook. Define at least four numbers, and refuse to print any of them alone.

# proposed metric definitions — not executed claims
METRICS = {
    "n_attempted": "tasks the harness started",
    "n_scored": "attempted minus infra exclusions",
    "pass_strict": "scored tasks whose tests exited 0 with no retry",
    "timeout_rate": "scored tasks that hit the wall clock",
    "compile_fail_rate": "scored tasks that never reached tests",
    "tokens_per_pass": "sum(tokens) / max(pass_strict_count, 1)",
}

Notice what is missing: a blended “score.” You can compute one later for a appendix. You do not lead with it. Leading with a blend is how unequal tasks, unequal budgets, and retries get laundered into a rank.

Write the exclusion rule in English on the card:

Toolchain crash, runner OOM, and lost network: exclude from n_scored, count in n_infra.
Agent produced a patch that fails tests: scored failure. Not infra.
Agent asked for a retry after timeout: still one attempt unless the card pre-declares max_retries.

If you cannot state the denominator, you do not have a metric. You have a vibe.

Step 3 — Record controls as data, not as folklore

Controls are the difference between a result and a story about a result. Put them in the same JSONL row as the outcome.

# proposed row schema for results.jsonl
row = {
    "card_id": "coding-mini-2026-09-22",
    "card_hash": "…",
    "task_id": "001_parse_csv",
    "stratum": {"lang": "python", "kind": "wrong-output"},
    "endpoint": "unknown-until-run",
    "temperature": 0.0,
    "max_output_tokens": 4096,
    "timeout_s": 180,
    "max_retries": 0,
    "seed": 7,
    "outcome": "fail_tests",  # pass | fail_tests | fail_compile | timeout | infra
    "tokens_in": None,
    "tokens_out": None,
    "exit_code": 1,
}

Fill endpoint at run time. Do not hard-code a model name you have not actually called. If the free endpoint you used has no public model string, store the URL class and the date, not a nickname from memory.

Retry policy belongs here, not in a wrapper script you forgot. max_retries: 0 is a control. we reran the flaky ones is contamination.

Step 4 — Run the harness where the methods can live

You need a box that can clone the frozen tree, call a model, run tests, and write JSONL. A laptop works until the lid closes. A small always-on server works better because the card, the logs, and the run live together.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access and a free server option. Those two facts are the only product claims this protocol needs. If you are already paying for an endpoint, keep it. The card does not care who serves the tokens. It cares that the endpoint is written down.

A boring run looks like this:

python harness.py --card bench/CARD.yaml --out runs/2026-09-22.jsonl
python summarize.py --in runs/2026-09-22.jsonl --require-card-hash

The second command should refuse to print a pass rate if card_hash is missing or if n_scored is zero. That refusal is the whole point.

Artifact: a card and a runner that will not print a poster

Save this as bench/CARD.yaml. It is a template. Replace every unknown before you quote a number.

card_id: coding-mini-2026-09-22
dataset:
  name: coding-mini
  version: "0.1.0"
  manifest_sha256: unknown
  sampling: "manual strata; not a random sample of production tickets"
  n_tasks: 12
  license: unknown
task_contract:
  pass: "repository tests exit 0"
  timeout_s: 180
  max_retries: 0
metrics:
  - pass_strict
  - timeout_rate
  - compile_fail_rate
  - n_scored
  - n_infra
controls_required:
  - endpoint
  - temperature
  - max_output_tokens
  - seed
exclusions:
  infra: ["runner_oom", "clone_fail", "network_lost"]
publication_rule: "do not print pass_strict without n_scored and card_hash"

Then a small harness. This is proposed code. It does not report a measured leaderboard. It shows how a run becomes unprintable until the card is complete.

#!/usr/bin/env python3
"""Proposed coding-agent harness. Not a published score."""
from __future__ import annotations

import argparse, hashlib, json, subprocess, sys, time
from pathlib import Path

try:
    import yaml
except ImportError:
    yaml = None


def load_card(path: Path) -> dict:
    text = path.read_text()
    if yaml:
        card = yaml.safe_load(text)
    else:
        raise SystemExit("install pyyaml or convert CARD.yaml to json")
    required = ["card_id", "dataset", "task_contract", "metrics", "publication_rule"]
    missing = [k for k in required if k not in card]
    if missing:
        raise SystemExit(f"card missing fields: {missing}")
    if card["dataset"].get("manifest_sha256") in (None, "unknown"):
        raise SystemExit("refuse to run: freeze MANIFEST.sha256 first")
    return card


def card_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]


def run_tests(task_dir: Path, timeout_s: int) -> dict:
    tests = task_dir / "tests"
    if not tests.exists():
        return {"outcome": "infra", "reason": "missing_tests", "exit_code": None}
    try:
        proc = subprocess.run(
            ["pytest", "-q"],
            cwd=task_dir,
            timeout=timeout_s,
            capture_output=True,
        )
    except subprocess.TimeoutExpired:
        return {"outcome": "timeout", "reason": "wall_clock", "exit_code": None}
    if proc.returncode == 0:
        return {"outcome": "pass", "reason": "tests_ok", "exit_code": 0}
    return {"outcome": "fail_tests", "reason": "tests_nonzero", "exit_code": proc.returncode}


def summarize(rows: list[dict]) -> dict:
    scored = [r for r in rows if r["outcome"] != "infra"]
    n_scored = len(scored)
    def rate(name: str) -> float | None:
        if n_scored == 0:
            return None
        return round(sum(1 for r in scored if r["outcome"] == name) / n_scored, 4)
    return {
        "n_attempted": len(rows),
        "n_scored": n_scored,
        "n_infra": sum(1 for r in rows if r["outcome"] == "infra"),
        "pass_strict": rate("pass"),
        "timeout_rate": rate("timeout"),
        "compile_fail_rate": rate("fail_compile"),
    }


def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--card", type=Path, required=True)
    p.add_argument("--out", type=Path, required=True)
    p.add_argument("--endpoint", default="unknown")
    args = p.parse_args()

    card = load_card(args.card)
    digest = card_hash(args.card)
    timeout_s = int(card["task_contract"]["timeout_s"])
    rows = []
    for task_dir in sorted((args.card.parent / "tasks").iterdir()):
        if not task_dir.is_dir():
            continue
        result = run_tests(task_dir, timeout_s)
        rows.append({
            "card_id": card["card_id"],
            "card_hash": digest,
            "task_id": task_dir.name,
            "endpoint": args.endpoint,
            "timeout_s": timeout_s,
            "max_retries": card["task_contract"].get("max_retries", 0),
            **result,
            "ts": int(time.time()),
        })

    args.out.parent.mkdir(parents=True, exist_ok=True)
    with args.out.open("w") as f:
        for row in rows:
            f.write(json.dumps(row) + "\n")

    stats = summarize(rows)
    if stats["n_scored"] == 0 or stats["pass_strict"] is None:
        print("unprintable: n_scored is 0", file=sys.stderr)
        sys.exit(2)
    print(json.dumps({"card_hash": digest, **stats}, indent=2))


if __name__ == "__main__":
    main()

Run it dry against an empty tree and you should get a refusal, not a zero. That is the behavior you want in CI.

python harness.py --card bench/CARD.yaml --out runs/dry.jsonl; echo exit:$?

Why the numbers are not marketing

Marketing needs a rank, a round number, and a missing asterisk. This protocol attacks all three.

The dataset is versioned, so you cannot quietly add easy tasks. The metrics travel as a set, so “71%” has to drag n_scored and timeout_rate with it. The controls sit on every row, so a mid-run endpoint swap shows up as two experiments, not one lucky curve. The harness will not print if the card is incomplete.

You still can misuse the output. People will screenshot the pass rate. Your job is to make that screenshot look unfinished: no hash, no denominator, no strata. Unfinished numbers are easier to challenge in review than polished ones.

Do not average strata into a single trophy. Report python and TypeScript separately if both exist. A combined mean is how a six-task toy language subsidizes a hard systems bug.

Limitations, and who should not use this

This is a methods scaffold. It is not a substitute for a reviewed benchmark, and it does not produce a vendor ranking.

Skip it if you need statistically tight comparisons across labs. Twelve frozen tasks will not give you that, even with perfect logging. Skip it if your tests are non-deterministic or require licensed cloud APIs you cannot replay. Skip it if you want a public leaderboard more than you want a paper trail.

Free model access and a free server will not make a small card into a large study. They only keep the runner and the notebook in one place. Token quality, rate limits, and hardware class are still controls you must record. If those controls are unknown, the score stays unprintable.

The protocol also will not stop a reader who ignores the card. It only makes ignoring the card visible.

What you do on Monday

Pick twelve tasks you already own. Freeze the tree. Fill every unknown you can. Run the harness once with retries disabled. Paste the JSON summary into the PR, not into the slide.

If the summary refuses to print, that is success. You found a hole in the methods before the number found an audience.

When you need an overnight box so the card and the JSONL sit together, MonkeyCode’s free model access and free server option are a reasonable place to park the runner. Keep the headline off the homepage until card_hash and n_scored travel with it.

Four Fields That Make a Coding-Agent Score Citeable

Casey Zhang — Mon, 21 Sep 2026 12:42:32 +0000

You are in standup. A teammate says the new coding agent “clears 71 percent.” You ask 71 percent of what. Nobody has a file.

Someone pastes a screenshot from a vendor blog. The screenshot has a green bar, a model nickname, and no dataset hash. That number will leak into a roadmap slide by Friday. Treat it as marketing until a card travels with it.

A citeable coding-agent score is not a percentage. It is a percentage plus the dataset, the oracles, the metric family, and the resource envelope that produced it. If any of those four are missing, you do not have a measurement. You have a story that happens to contain digits.

What a marketing number looks like

You have seen the pattern. A public benchmark name. A single pass rate. No timeout. No token envelope. No note on whether the tasks already live in training corpora. The chart is clean because the protocol is invisible.

Public leaderboards still matter as conversation. They do not matter as procurement evidence. Named suites get trained against, coached against, and blog-optimized against. When the test is famous, the score is partly a memory test.

You need a local protocol you can rerun. The rest of this article is that protocol. It is a proposal you can execute on a laptop or a small server. It does not require you to trust a vendor chart.

The four objects on the card

Keep four objects. Refuse to quote a score if one is blank.

Dataset. A frozen task pack with content hashes, license notes, and difficulty strata.
Oracle. Commands that decide pass or fail without a human reading the agent’s essay.
Metric family. More than one number, and never a silent average across unequal envelopes.
Envelope. Wall-clock, step cap, and output cap, written down before the run starts.

Contamination posture sits beside those four. You will not prove a task is unseen. You can still record how you tried.

Step 1: Freeze a private task pack

Do not start from a leaderboard dump and call it private. Start from bugs you already shipped.

Pick 20 to 40 defects from your own tracker. Prefer ones with a failing test that already exists. If you must synthesize tasks, write the failing test first. Then freeze.

mkdir -p bench/tasks bench/oracles
git -C . rev-parse HEAD > bench/SOURCE_COMMIT
find bench/tasks -type f -print0 | sort -z | xargs -0 sha256sum > bench/DATASET.sha256
git add bench && git commit -m "freeze: coding-agent task pack"

Store one JSON line per task. Keep any gold patch out of the prompt the agent sees.

{"id":"INV-1843","lang":"py","strata":"medium","prompt_file":"tasks/INV-1843.md","oracle":"oracles/INV-1843.sh","timeout_s":120}

If you cannot hash it, you cannot cite it. A moving folder is not a dataset. It is a mood.

Step 2: Make the oracle a command, not a vibe

Each task needs a script that exits 0 on pass and non-zero on fail. The script must not read the agent’s chain of thought. It applies the patch, runs tests, and maybe a tiny linter you already own.

#!/usr/bin/env bash
set -euo pipefail
TASK_DIR="$1"
PATCH="$2"
cd "$TASK_DIR"
git apply --check "$PATCH"
git apply "$PATCH"
python -m pytest -q tests/test_invoice_tax.py

Label this as the contract. If the oracle needs a human, the metric is an annotation study. That is valid work. It is not a pass rate.

Step 3: Choose a metric family

One headline pass rate is how marketing happens. You want at least three printed numbers, each with a denominator.

oracle_pass: tasks where the oracle exited 0.
oracle_fail: tasks where the patch applied and tests failed.
infra_fail: tasks where git, the network, or the runner died. Do not fold these into oracle_fail.
strata_pass: oracle_pass split by easy / medium / hard.

You may add exact-match on a gold patch if you have one. Do not let that replace the oracle. Agents can fix the bug with a different diff.

The scorer below is a proposal, not a claim about any product. It refuses to emit a single average.

from collections import defaultdict

def summarize(rows):
    by_strata = defaultdict(lambda: {"n": 0, "pass": 0})
    infra = 0
    for r in rows:
        if r["status"] == "infra_fail":
            infra += 1
            continue
        bucket = by_strata[r["strata"]]
        bucket["n"] += 1
        bucket["pass"] += int(r["status"] == "oracle_pass")
    return {
        "infra_fail": infra,
        "strata": {
            name: {
                "n": vals["n"],
                "pass": vals["pass"],
                "rate": None if vals["n"] == 0 else round(vals["pass"] / vals["n"], 3),
            }
            for name, vals in sorted(by_strata.items())
        },
    }

Notice there is no mean of means. If someone later averages easy and hard, they are writing a new paper. They are not citing yours.

Step 4: Write the envelope before the first call

An agent with 40 steps and an agent with 8 steps are not the same method. Neither is an agent that may emit ten times the tokens. Put the cap in the card. Kill the run when it trips.

{
  "name": "invoice-tax-pack-2026-09",
  "dataset_sha256_file": "DATASET.sha256",
  "n_tasks": 24,
  "envelope": {
    "max_wall_s": 180,
    "max_steps": 12,
    "max_output_tokens": 8192
  },
  "metrics": ["oracle_pass", "infra_fail", "strata_pass"],
  "contamination": {
    "public_overlap_check": "manual filenames vs known suite lists",
    "training_guarantee": "none"
  },
  "print_policy": "refuse_headline_if_any_field_blank"
}

Save that as bench/card.json. If you raise the envelope after you see a disappointing rate, you started a new experiment. Rename the card. Do not overwrite the old one.

Step 5: Record contamination posture honestly

You will not get a cryptographic proof that a model never saw your bug. Say so.

Do three cheap checks anyway:

Search your prompt text against public benchmark names and famous function identifiers.
Keep proprietary identifiers in the fixtures so a memorized public snippet cannot pass.
Hold back a slice you never paste into a chat product, even “just to see.”

Write the outcome on the card: unchecked, filename_scan, or private_ids. Unchecked is allowed. Pretending you checked is not.

Step 6: Print the card, or print nothing

The script below refuses a headline if the card is incomplete. Run it after every batch. It uses only the standard library.

#!/usr/bin/env python3
"""Proposal: refuse a headline pass rate when the card is incomplete."""
from __future__ import annotations

import json
import sys
from collections import defaultdict
from pathlib import Path

REQUIRED = (
    "name",
    "dataset_sha256_file",
    "n_tasks",
    "envelope",
    "metrics",
    "contamination",
)

def summarize(rows):
    by_strata = defaultdict(lambda: {"n": 0, "pass": 0})
    infra = 0
    for r in rows:
        if r["status"] == "infra_fail":
            infra += 1
            continue
        bucket = by_strata[r["strata"]]
        bucket["n"] += 1
        bucket["pass"] += int(r["status"] == "oracle_pass")
    return {
        "infra_fail": infra,
        "strata": {
            name: {
                "n": vals["n"],
                "pass": vals["pass"],
                "rate": None if vals["n"] == 0 else round(vals["pass"] / vals["n"], 3),
            }
            for name, vals in sorted(by_strata.items())
        },
    }

def load_card(path: Path) -> dict:
    card = json.loads(path.read_text())
    missing = [k for k in REQUIRED if k not in card or card[k] in (None, "", [])]
    if missing:
        raise SystemExit(f"unprintable: missing {missing}")
    sha_file = path.parent / card["dataset_sha256_file"]
    if not sha_file.exists():
        raise SystemExit("unprintable: dataset hash file missing")
    env = card["envelope"]
    for key in ("max_wall_s", "max_steps", "max_output_tokens"):
        if key not in env:
            raise SystemExit(f"unprintable: envelope.{key} missing")
    return card

def main() -> None:
    root = Path(sys.argv[1] if len(sys.argv) > 1 else "bench")
    card = load_card(root / "card.json")
    result_path = root / "results.jsonl"
    if not result_path.exists():
        raise SystemExit("unprintable: results.jsonl missing")
    rows = [json.loads(line) for line in result_path.read_text().splitlines() if line.strip()]
    if len(rows) != card["n_tasks"]:
        raise SystemExit("unprintable: result count != n_tasks")
    summary = summarize(rows)
    print(json.dumps({"card": card["name"], "envelope": card["envelope"], **summary}, indent=2))

if __name__ == "__main__":
    main()

A tiny results.jsonl for a smoke test looks like this:

{"id":"INV-1843","strata":"medium","status":"oracle_pass"}
{"id":"INV-1844","strata":"hard","status":"oracle_fail"}
{"id":"INV-1845","strata":"easy","status":"infra_fail"}

Wire the scorer as a command, not a ritual:

chmod +x score_card.py
python3 score_card.py bench > bench/SCORE.json
test -s bench/SCORE.json

If the command exits non-zero, you have nothing to paste into Slack. That is the point.

Decision table: cite or do not cite

Situation	Citeable?	What you may say
Pass rate only, no card	No	“We do not have a measurement yet.”
Card complete, infra_fail folded into fail	No	“Rerun: infra is unlabeled.”
Card complete, strata hidden, only a mean	No	“Publish the strata or do not publish.”
Envelope raised after seeing results	No	“New experiment. New card name.”
Card complete, strata + infra + envelope present	Yes	Quote rates with n, envelope, and hash.
Public-benchmark name with no overlap note	No	“That is a conversation, not a citation.”

Tape this table next to the dashboard. The dashboard will try to grow a single tile. Do not let it.

Run the protocol where you can afford to rerun it

A protocol you cannot afford to rerun will rot into a one-off demo. That is how vendor charts get cited. You need a boring place to execute the oracle, keep the frozen pack, and call a model.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you do not already have an inference stack, MonkeyCode’s free model access and free server option are one way to exercise this card without first procuring GPUs. The card still lives in your repo. The oracles still belong to you. The product is only a runner you can actually repeat.

Do not treat a free runner as a published lab. Same envelope. Same hashes. Same refusal to print a naked percentage.

Limitations

This protocol does not estimate statistical power. Twenty-four tasks will not rank two agents. It will tell you whether your own card is honest.

It does not detect training-data leakage beyond cheap filename and identifier checks. Anyone who claims otherwise is selling comfort.

It does not score partial credit, human preference, or security impact. If those are your questions, design those oracles. Do not overload pytest until it lies.

The Python above is a proposal. It is not a result from a production eval run for this article. Copy it, then replace summarize with the metrics your oracles actually emit.

Who should not use this

Skip this if you are writing a launch tweet. A card slows you down. That is a feature for measurement and a bug for announcements.

Skip this if your tasks have no automatic oracle. You need an annotation protocol, not a bash exit code.

Skip this if you need a legally defensible comparison for procurement. Hire an eval team, pre-register the card, and size the task pack. This article is a hygiene kit, not a lab accreditation.

What you do Monday

Open the last agent score you quoted. Ask for the dataset hash, the oracle command, the strata table, and the envelope. If those four files do not exist, the number was marketing.

Write the card first. Then, and only then, quote a rate.

Score the Patch, Not the Story: A Coding-Agent Evaluation Schema

Casey Zhang — Sun, 20 Sep 2026 10:54:01 +0000

A teammate drops a number into Slack on a Friday: the agent hit 81% on “our coding suite.” You ask where the tasks came from. Silence. You ask whether the agent could read the unit tests. More silence. Two timeouts were deleted so the average would look cleaner. That 81% is a story. It is not a measurement.

You do not need a giant public leaderboard to stop this. You need a dataset schema, a handful of metrics that refuse to collapse into one float, and a rule for when a number may leave the lab. This article is a protocol you can run on a small box. Treat every code block as a working example, not a published result from a particular model.

What you are actually scoring

A coding agent is not a chatbot with a repo mounted. It is a patch factory. The only artifact that should enter a score is the diff, plus whether hidden tests and hidden constraints still hold after you apply that diff.

Visible tests are contaminated the moment they sit in the prompt, the tool output, or a CI log the agent can cat. If you average them with hidden tests, you are scoring reading comprehension of your own suite. Stop doing that.

Public GitHub issues are a second contamination path. If the task text, the failing test names, or the gold patch exist on the open web, a pretrained model may be recalling, not repairing. You still can use those tasks. You just cannot mix them with internal tickets and call the blend a single pass rate.

The four fields the dataset must carry

Every task file needs identity, isolation, provenance, and a locality bound. Miss one, and you are back to storytelling.

# tasks/PAY-1847.yaml  — example schema, not a live benchmark row
id: PAY-1847
origin: internal_ticket          # internal_ticket | synthetic | public_web
leakage_risk: low                # low | unknown | high
prompt: |
  Refunds posted after 17:00 UTC are dated the next business day.
  The batch job currently stamps local time. Fix the stamp, do not
  rewrite the ledger format.
allowed_files:
  - src/billing/batch_stamp.py
  - src/billing/business_day.py
max_files_touched: 2
max_diff_loc: 80
timeout_sec: 180
visible_tests: []                # empty on purpose
hidden_tests:
  - tests/test_batch_stamp_utc.py
hidden_constraints:
  - "Do not change CSV column order in ledger export"
  - "Stamp timezone must be UTC, not the server locale"

origin and leakage_risk are not metadata for a blog post. They are strata. You score each stratum alone. allowed_files and max_files_touched stop an agent from “winning” by rewriting the test harness. hidden_constraints catch the patch that goes green and still ships the wrong business rule.

If a task cannot fill those fields, it does not enter the set. You will have fewer tasks. That is the point.

Build the set in six numbered steps

Start from closed work, not from Twitter. Export 20–40 tickets your team already shipped. Prefer bugs with a failing test that landed in the same pull request as the fix. Synthetic tasks are allowed when you write both the bug and the hidden test yourself.
Strip the gold patch from disk. The runner image should contain the broken tree, the prompt, and the hidden tests. It should not contain expected.diff. If the agent can open the answer, you are measuring cat.
Split tests before the first run. Visible tests may teach. Hidden tests may only judge. Put hidden tests in a path the tools cannot list, or mount them after the agent exits. Label the mount in the run record.
Tag provenance on the same day you copy the text. A public issue cloned “just for coverage” is origin: public_web and leakage_risk: high even if you reword the title. Do not launder it into internal_ticket.
Freeze IDs. Once PAY-1847 is in the set, you do not edit its prompt because an agent failed it. You open PAY-1847.v2 as a new row. Otherwise your history is a moving target.
Write an exclusion log before you compute anything. Timeouts, checkout failures, missing interpreters, and network denials are not agent misses. They are rows in exclusions.jsonl. A protocol that drops them quietly is an ad.

Do the six steps on a sample of ten tasks before you scale. If you cannot fill the YAML, the suite is not ready. The agent is irrelevant until the suite is ready.

Metrics that refuse to become a headline

You report a row, not a percentage. Each completed run produces the object below. Average nothing across strata. Average nothing across unequal timeouts.

# score_patch.py — example scorer, unexecuted on any vendor model
from dataclasses import dataclass, asdict
from pathlib import Path
import json, subprocess, textwrap

@dataclass
class TaskScore:
    task_id: str
    origin: str
    leakage_risk: str
    hidden_pass: bool
    constraint_hits: int
    constraint_total: int
    files_touched: int
    files_allowed: int
    excess_files: int
    diff_loc: int
    max_diff_loc: int
    locality_ok: bool
    timeout: bool
    infra_failure: bool
    printable: bool

CONSTRAINT_HINTS = {
    "Do not change CSV column order in ledger export": "ledger",
    "Stamp timezone must be UTC, not the server locale": "UTC",
}

def git_diff_stat(repo: Path) -> tuple[int, list[str]]:
    diff = subprocess.check_output(["git", "-C", repo, "diff", "--numstat", "HEAD"], text=True)
    files, loc = [], 0
    for line in diff.splitlines():
        parts = line.split("\t")
        if len(parts) != 3:
            continue
        added, deleted, path = parts
        if added == "-" or deleted == "-":
            continue
        loc += int(added) + int(deleted)
        files.append(path)
    return loc, files

def constraints_hit(repo: Path, constraints: list[str]) -> int:
    blob = subprocess.check_output(["git", "-C", repo, "diff", "HEAD"], text=True)
    hits = 0
    for c in constraints:
        needle = CONSTRAINT_HINTS.get(c, "")
        # Toy check: real suites should use dedicated assertion files.
        if needle and needle in blob:
            hits += 1
    return hits

def score_run(task: dict, repo: Path, hidden_pass: bool, timeout: bool, infra: bool) -> TaskScore:
    loc, files = git_diff_stat(repo)
    allowed = set(task["allowed_files"])
    excess = [f for f in files if f not in allowed]
    hits = constraints_hit(repo, task["hidden_constraints"])
    locality_ok = (len(files) <= task["max_files_touched"]) and (not excess) and (loc <= task["max_diff_loc"])
    printable = (not timeout) and (not infra) and hidden_pass and locality_ok and hits == len(task["hidden_constraints"])
    return TaskScore(
        task_id=task["id"], origin=task["origin"], leakage_risk=task["leakage_risk"],
        hidden_pass=hidden_pass, constraint_hits=hits, constraint_total=len(task["hidden_constraints"]),
        files_touched=len(files), files_allowed=len(allowed), excess_files=len(excess),
        diff_loc=loc, max_diff_loc=task["max_diff_loc"], locality_ok=locality_ok,
        timeout=timeout, infra_failure=infra, printable=printable,
    )

Read the last field slowly. printable is true only when the hidden tests pass, the constraints hit, the diff stayed inside the fence, and the run was not an infra event. That is stricter than “pytest went green.” It is also the only boolean you should ever paste into a channel.

For a set, publish a table with one row per origin × leakage_risk cell. Counts, not rates, until every cell has a pre-declared minimum n. If a cell has three tasks, you write “3/3 printable” or “1/3 printable.” You do not write 33% as if it were a property of the agent.

Controls that keep the table from becoming marketing

Wall-clock, tool steps, and file fences belong in the run record. So does the judge version. If you change hidden tests, you bump the judge, not the agent name.

# run-record fields you store next to the diff
run_id: 2026-09-20T14-02Z-PAY-1847-a3
task_id: PAY-1847
judge_version: schema-1.2
agent_label: free-tier-runner    # never a model marketing name unless you logged it
timeout_sec: 180
tool_step_cap: 40
hidden_tests_mounted_after_exit: true
seed: 17
exclusion: null

Equalize what you compare. A 180-second cap on one agent and a 15-minute cap on another is a different exam. A tool cap of 40 versus unlimited bash is a different exam. You may still run both. You may not average them.

Seed the sampling. Free-tier decoding is noisy. One lucky pass on a three-task cell is not a ranking. Repeat the same frozen IDs. Report the repeats as repeats, not as extra tasks.

A decision table, not a leaderboard

Use this table before anyone quotes a number outside the team that built the suite.

Question	If no	What you may publish
Does every row have `origin` and `leakage_risk`?	Stop	Protocol draft only
Are hidden tests mounted after the agent exits?	Stop	Qualitative notes
Are infra failures in `exclusions.jsonl` instead of scored as misses?	Stop	Exclusion log
Do you report per-stratum counts, not one blend?	Stop	Per-task diffs
Is n per cell pre-declared and met?	Stop	“insufficient n”
Did both agents share timeout, tool cap, and judge version?	Stop	Separate run cards
Did the patch pass locality and hidden constraints?	Mark unprintable	Diff + failing field
Did all of the above hold?	—	Counts, with protocol hash

The last row is the only path to a number. Even then the number is a count attached to a protocol hash, not a brand claim. If you cannot name the hash, you cannot name the score.

Where a free runner actually helps

You can execute this protocol without a paid cluster. The dataset is YAML. The scorer is a Python file and git diff. The expensive part is the agent loop, and that is where a free-tier host is useful as a runner, not as a source of glory numbers.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already need a place to park the runner, MonkeyCode’s free model access and free server option can host the agent loop and the after-exit judge. Label those runs runtime: free-tier in the record above. Do not mix them with a different host, a different timeout, or a different tool cap and then sort the rows as if the runtime were noise.

The protocol does not care which vendor paid for the tokens. It cares that you wrote the runtime down. A free box that keeps the mount honest is worth more than a paid box that lets the agent read tests/.

Limitations

The constraint check in the sample scorer is a toy. String-matching “UTC” in a diff will both false-pass and false-fail. Production suites should encode each hidden constraint as an assertion file the judge runs after the agent is gone. Do not ship the sample CONSTRAINT_HINTS map into CI and call it science.

Locality bounds punish legitimate refactors. A three-line bug that truly requires a fourth file will look like a miss. That is acceptable for a repair benchmark. It is the wrong metric for a “modernize this module” benchmark. Split those into two sets. Do not reuse max_files_touched: 2 as a moral value.

Provenance tags are only as good as the person who set them. If you copy a well-known public issue and mark leakage_risk: low because you renamed a function, you have laundered the task. When in doubt, mark high and keep the cell separate.

This protocol says nothing about user experience, latency under load, or safety. A printable patch can still be rude, slow, or leak a secret in a log line you never asserted on. Add those oracles if they matter. Do not pretend hidden pytest covers them.

Who should not use this

Skip this if you need a press ranking this week. The schema shrinks your set and delays the first number. That is incompatible with a launch blog.

Skip this if your tasks have no hidden tests and no constraints worth encoding. A prompt-only coding quiz is a different instrument. Forcing the YAML onto it will not make it a patch benchmark.

Skip this if you cannot freeze IDs. Teams that rewrite failed tasks until the agent passes are debugging the suite, not the agent. There is a place for that work. It is not evaluation.

Skip this if you will not keep infra rows in the exclusion log. A free server that restarts mid-run will manufacture misses. If you cannot label those, you should not rank anything that ran there.

What you do on Monday

Pick ten closed tickets. Fill the YAML. Hide the tests. Run one agent twice on the same IDs with the same cap. Print TaskScore rows. If you cannot explain a single printable: false from the fields, the schema is doing its job and the Slack number was never real.

Keep the protocol hash next to the table. When someone asks “what did it score?”, you send the table, the exclusions, and the hash. You do not send 81%.

If you try the runner path on MonkeyCode’s free model access and free server option, use it to keep the mount and the record honest. The score still has to survive the decision table. The host does not get a vote.

Don't Average Pass Rates Across Unequal Token Budgets

Casey Zhang — Sat, 19 Sep 2026 09:10:54 +0000

You taped two terminal panes to a Friday review. The left pane showed Agent A with eight greens; the right pane showed Agent B with six. Someone asked which one to keep. You almost said A.

Then you scrolled the traces. Agent A retried until the cap cried. Agent B stopped after one compile and one test run. Same tasks. Different permission to spend. The ranking was a poster, not a measurement.

A pass rate answers one question: did the tests go green. It is silent on three others. How many tokens did the run burn before that green. Did a budget cap get scored as a model failure. Were the two agents even allowed the same tool-call envelope. If you cannot answer those, you do not have a benchmark. You have a vibe with a percentage sign.

This article is a protocol, not a leaderboard. You will pin a tiny task pack, declare a budget before the first call, write every attempt to a JSONL ledger, and publish three numbers together. The scripts below are a proposal you can copy. They are not a claim about any vendor eval, and they are not executed numbers from a production fleet.

Why the single percentage is usually marketing

Engineering reviews launder vibe-coding into a metric when the budget is a footnote. Two agents can share a task list and still live in different universes. One is allowed 40 tool calls and four repair loops. The other gets one shot and a 2k-token ceiling. Averaging their pass rates is like averaging two race times after giving one car a second tank of fuel.

You need the budget in the same table as the outcome. Not in a blog paragraph. In the file you commit.

The protocol below treats tokens, tool calls, and truncations as first-class results. Pass/fail is column four, not the headline.

What you will actually produce

Work in a single directory. Keep it boring.

A pinned task pack with SHA-256 hashes.
A budget envelope declared before any model call.
A JSONL ledger with one object per attempt, including the dull failures.
Three numbers you publish together, or you publish none of them: pass rate, truncation rate, and tokens-per-success.

You can run this on a laptop. You can also run it on a free server so the cost of being honest is not why the controls get skipped.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option matter here only as a lab: you can execute the ledger without folding a billed retry policy into the ranking. Strip the product name out and the method still stands. Free is still a cap. Write that cap in the same file as the tasks.

Step 1 — Pin a task pack you can hash

Do not point at "the latest suite on the internet." Snapshot files. Hash them. If a fixture changes, the run id changes. That is the whole point.

Proposed layout:

eval_pack/
  TASKS.md
  budget.yaml
  tasks/
    01_csv_summary/
      prompt.md
      src/app.py
      tests/test_app.py
    02_retry_header/
      prompt.md
      src/app.py
      tests/test_app.py
  hashes.sha256

Keep the pack small on purpose. Eight to twelve tasks is enough to debug the protocol. It is not enough to crown a winner. Label it that way in TASKS.md.

Each task needs three things: a prompt the agent may read, a hidden test file the agent must not edit, and a definition of done that is a command, not a feeling.

# tasks/01_csv_summary/prompt.md
Read src/app.py. Make summarize(path) return a dict with keys
rows, empty_rows, numeric_columns. Do not edit tests/.

Hash after you freeze the files:

cd eval_pack
find tasks -type f | sort | xargs sha256sum > hashes.sha256
sha256sum budget.yaml TASKS.md >> hashes.sha256
cat hashes.sha256

If hashes.sha256 drifts, you are not comparing agents. You are comparing days.

Step 2 — Declare the budget before the first call

Write the envelope first. Then refuse to start a run that does not load it. This is a control, not a comment.

# budget.yaml
envelope_id: "lab-2026-09-19-a"
max_input_tokens_per_task: 8000
max_output_tokens_per_task: 2000
max_tool_calls_per_task: 12
max_repair_loops: 2
wall_clock_seconds_per_task: 180
hidden_tests_are_read_only: true

Pick numbers that fit your lab. Do not copy these as if they were a standard. The standard is that both agents get the same envelope, and that truncation is recorded as its own outcome class.

Outcome classes you will allow:

pass — hidden tests exit 0 inside the envelope.
fail — tests exit non-zero, budget still remaining.
truncated — a cap fired before tests could pass.
invalid — the agent edited a forbidden path, or the runner crashed in a way you cannot attribute.

truncated is not fail. Mixing them is how a stingy envelope looks like a weak model.

Step 3 — Log every attempt, including the boring ones

JSONL beats a screenshot of a dashboard. One object per task attempt. No silent retries.

Proposed record (label: schema proposal):

{
  "run_id": "20260919T1400Z-lab-a",
  "envelope_id": "lab-2026-09-19-a",
  "task_id": "01_csv_summary",
  "agent_id": "agent-a",
  "pack_hash": "REPLACE_WITH_SHA256_OF_hashes.sha256",
  "input_tokens": 2410,
  "output_tokens": 880,
  "tool_calls": 5,
  "repair_loops": 1,
  "wall_clock_seconds": 47,
  "outcome": "pass",
  "test_exit_code": 0,
  "notes": ""
}

Append only. If a row is wrong, write a correction row. Do not silently edit history.

mkdir -p runs
touch runs/20260919T1400Z-lab-a.jsonl

Your runner should dump token counts from the provider response, not from a guess. Tokenizers differ. That is a limitation, not an excuse to skip the column.

Step 4 — Score three numbers, never one

Proposal, not a published score. After the JSONL exists, compute:

pass_rate = passes / tasks
truncation_rate = truncated / tasks
tokens_per_success = sum(input_tokens + output_tokens for passes) / max(passes, 1)

If passes is zero, tokens_per_success is undefined. Print NA. Do not print infinity and call it a ranking.

# score_ledger.py — proposal, unexecuted example
import json
import sys
from collections import defaultdict

REQUIRED = {
    "run_id", "envelope_id", "task_id", "agent_id", "pack_hash",
    "input_tokens", "output_tokens", "tool_calls", "repair_loops",
    "wall_clock_seconds", "outcome",
}

def load(path):
    rows = []
    with open(path, encoding="utf-8") as handle:
        for line_no, line in enumerate(handle, 1):
            line = line.strip()
            if not line:
                continue
            row = json.loads(line)
            missing = REQUIRED - row.keys()
            if missing:
                raise SystemExit(f"line {line_no} missing {sorted(missing)}")
            rows.append(row)
    return rows

def score(rows):
    grouped = defaultdict(list)
    for row in rows:
        grouped[(row["agent_id"], row["envelope_id"], row["pack_hash"])].append(row)
    reports = []
    for key, items in grouped.items():
        n = len(items)
        passes = [r for r in items if r["outcome"] == "pass"]
        trunc = [r for r in items if r["outcome"] == "truncated"]
        tokens = sum(r["input_tokens"] + r["output_tokens"] for r in passes)
        reports.append({
            "agent_id": key[0],
            "envelope_id": key[1],
            "pack_hash": key[2],
            "n": n,
            "pass_rate": round(len(passes) / n, 3) if n else None,
            "truncation_rate": round(len(trunc) / n, 3) if n else None,
            "tokens_per_success": round(tokens / len(passes), 1) if passes else "NA",
        })
    return reports

if __name__ == "__main__":
    for report in score(load(sys.argv[1])):
        print(json.dumps(report, indent=2))

Run it:

python3 score_ledger.py runs/20260919T1400Z-lab-a.jsonl

Compare agents only when envelope_id and pack_hash match. If they do not match, the script still prints two blocks. You do the adult thing and refuse the average.

A useful derived check: if truncation_rate differs by more than one task, you are mostly measuring the envelope, not the agent. Widen the budget and rerun both. Or keep the tight budget and say so in the first sentence of the writeup.

Step 5 — Decision table: measurement vs advertisement

Use this table before anyone pastes a percentage into Slack.

Situation	Quote a single pass rate?	Publish instead
Same `pack_hash`, same `envelope_id`, both agents logged	No	The three-number triple, plus n
Different token caps, "same" tasks	No	Two separate triples, never averaged
Hidden tests were edited by the agent	No	`invalid` count and a patch diff
n under 8 tasks	No	The ledger, labeled as a protocol debug
Truncation rate > 0.25 for one agent only	No	"envelope-bound" and the raw traces
Latency on a shared free server	No	Tokens and outcomes only; drop wall-clock from the headline
Homepage hero or vendor one-pager	No	Link the recipe, or do not cite the lab

If a cell says no, the percentage is marketing even when the arithmetic is correct.

Step 6 — Pairwise only inside the same envelope

When you must pick a winner, do not subtract two pass rates. Count paired outcomes on the same task_id.

Proposal rule:

Drop any task where either agent is invalid.
On the rest, score +1 for the agent that passed when the other did not.
If both passed, award +1 to the lower token total only when the gap is larger than 15 percent. Otherwise call it a tie.
If both truncated, call it a tie. You learned about the envelope, not the model.

That 15 percent band is a knob. Write the knob in budget.yaml. Do not tune it after you see which agent you like.

What this protocol refuses to claim

It does not detect training-set contamination. It does not prove an agent is safe to ship. It does not turn a free server into a latency benchmark. Shared CPUs lie. Tokenizers lie in smaller ways. Your hidden tests lie if they only cover the happy path.

It also does not make a 12-task pack into a scientific paper. Twelve tasks can falsify a sloppy ranking. They cannot support a superlative.

Who should not use this

Skip this approach if you need a homepage number this afternoon. Skip it if your "agent" is allowed network access the other agent is not. Skip it if you will not log truncated runs because they look ugly. Skip it if the real question is production SLA, not coding-agent comparison. And skip it if you planned to treat free-tier queue time as intelligence.

If you want a lab that already offers free model access and a free server option, MonkeyCode is one place to run the ledger. The protocol does not depend on it. The only number worth quoting is the triple you can rebuild from JSONL and a hash file.

Make a Coding-Agent Score Unprintable Until the Controls Pass

Casey Zhang — Fri, 18 Sep 2026 07:06:47 +0000

You are in standup with a screenshot. Someone says the new coding agent hit 91 percent. Nobody can name the tasks, the grader, the runtime pin, or the cases that were supposed to fail on purpose.

The number still travels. Slack copies it. A slide copies Slack. By Friday the method is gone and the percentage is the only artifact left.

That is not a benchmark. It is a demo with a percent sign. If you want a number you can defend, you need four files that can veto the headline. Until those files exist and the controls pass, the score should refuse to print.

This is a small, reproducible method. It is not a league table. It will not make a model look superhuman. It will make a weak evaluation obvious.

The problem you are actually measuring

Coding agents fail in clusters. A Python fixture leak, a flaky assertion, a wrong language version, or a grader that accepts a stub can move a score more than the model does.

You cannot see that in a single pass rate. A pass rate is a reduction. Reductions are useful after you publish the cuts you made. They are marketing when you hide the cuts.

So you stop leading with the percent. You lead with a dataset card, a grader contract, an environment lock, and a control set. The percent is a footnote that the report is allowed to emit only when the controls stay green.

Four files, then a number

Keep the working directory boring. Four JSON documents plus one runner is enough for a laptop-scale study.

dataset_card.json — what the tasks are, how they were chosen, and which strata they occupy.
grader_contract.json — what counts as pass, fail, skip, or invalid.
env.lock.json — language, test runner, dependency hash, timeout, seed.
controls.json — tasks the agent must pass and tasks the agent must fail, or the run is discarded.

If any file is missing, you do not have a score. You have a log.

Step 1: Write the dataset card first

Do not harvest random GitHub issues and call it a set. Write the card before the first agent call. The card is the claim. The agent is only a subject under that claim.

{
  "name": "tiny-fix-v1",
  "n_tasks": 24,
  "split": "frozen-dev",
  "source": "internal fixtures, not scraped leaderboards",
  "strata": [
    {"id": "py-assert", "n": 8, "language": "python", "skill": "failing-test-repair"},
    {"id": "js-types", "n": 8, "language": "javascript", "skill": "type-narrowing"},
    {"id": "cfg-leak", "n": 8, "language": "python", "skill": "config-isolation"}
  ],
  "exclusions": ["network required", "gpu required", "multi-hour jobs"],
  "contamination_note": "fixtures authored for this card; do not mix public interview puzzles"
}

The strata matter more than the total. Eight tasks in one skill and sixteen in another is not a balanced 24. If you later quote one number, the card must still show the mix. Readers can reject the mix. That is the point.

Label this card as a proposal until a human reviews every fixture. Do not pretend 24 tasks represent production engineering.

Step 2: Freeze a grader contract people can argue with

A hidden grader is a second model. You would not accept an unlabeled second model in the loop. Publish the contract.

{
  "pass": "all tests in tests/ exit 0 after the patch, with no extra files outside allowlist",
  "fail": "any test non-zero, timeout, or patch that edits tests/",
  "skip": "environment hash mismatch or missing fixture",
  "invalid": "control task outcome disagrees with expected_control",
  "partial_credit": false,
  "test_command": ["pytest", "-q", "--timeout=20"],
  "timeout_sec": 120,
  "max_patch_bytes": 20000
}

Notice what is absent. There is no “looks good to the author.” There is no hidden rubric. Partial credit is off on purpose. If you want partial credit later, add it as a second metric with its own contract. Do not smuggle it into the headline.

If two people cannot apply this contract to the same patch and get the same label, the contract is not frozen. Fix the contract before you rank anything.

Step 3: Lock the environment or admit the score is wet

You already know this from CI. Python 3.11 versus 3.12 can flip assertions. A floating pytest plugin can swallow failures. Pin it.

{
  "python": "3.11.9",
  "node": null,
  "pytest": "8.3.2",
  "seed": 501,
  "os": "linux",
  "fixture_hash": "sha256:replace-me-after-tar-cf",
  "network": false
}

Compute fixture_hash from a tarball of the task directories, not from a folder you keep editing. If the hash moves, the split is no longer frozen. Print SKIP for the whole run. Do not average old and new tasks in silence.

Step 4: Plant controls that can veto the run

Positive controls are tasks a stub should still fail and a known-good patch should pass. Negative controls are tasks that must remain failing if the grader is honest: a missing assertion, a tautology, a test file the agent is forbidden to edit.

{
  "must_pass_with_gold": ["py-assert/01", "js-types/03"],
  "must_fail_with_empty_patch": ["py-assert/01", "cfg-leak/02"],
  "must_fail_forever": ["controls/tautology-green", "controls/deleted-assert"]
}

If a gold patch fails must_pass_with_gold, your tests are broken. If an empty patch passes must_fail_with_empty_patch, your grader is broken. If must_fail_forever goes green, someone taught the agent to cheat the tests, or the tests never tested anything.

In all three cases the runner prints INVALID and no percent.

A runner that refuses to market itself

The artifact below is a complete, local harness. It does not call a vendor. It reads the four files, applies a patch directory, and writes a report that keeps the headline gated.

Save it as unprintable_score.py.

#!/usr/bin/env python3
"""Gate a coding-agent score behind dataset, grader, env, and controls."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent


def load(name: str) -> dict:
    return json.loads((ROOT / name).read_text())


def sha_tree(path: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(path.rglob("*")):
        if p.is_file():
            h.update(p.relative_to(path).as_posix().encode())
            h.update(p.read_bytes())
    return "sha256:" + h.hexdigest()


def run_tests(task_dir: Path, contract: dict) -> str:
    try:
        proc = subprocess.run(
            contract["test_command"],
            cwd=task_dir,
            capture_output=True,
            timeout=contract["timeout_sec"],
        )
    except subprocess.TimeoutExpired:
        return "fail"
    return "pass" if proc.returncode == 0 else "fail"


def main() -> int:
    card = load("dataset_card.json")
    contract = load("grader_contract.json")
    env = load("env.lock.json")
    controls = load("controls.json")

    fixture_root = ROOT / "tasks"
    current_hash = sha_tree(fixture_root)
    if current_hash != env["fixture_hash"]:
        json.dump({"status": "SKIP", "reason": "env hash mismatch",
                    "expected": env["fixture_hash"], "got": current_hash},
                   sys.stdout, indent=2)
        return 2

    rows = []
    for stratum in card["strata"]:
        stratum_dir = fixture_root / stratum["id"]
        for task_dir in sorted(p for p in stratum_dir.iterdir() if p.is_dir()):
            label = run_tests(task_dir, contract)
            rows.append({
                "task": f"{stratum['id']}/{task_dir.name}",
                "stratum": stratum["id"],
                "label": label,
            })

    by_task = {r["task"]: r["label"] for r in rows}

    def expect(names, label, bucket):
        bad = [n for n in names if by_task.get(n) != label]
        return bad

    # Gold and empty-patch checks are expected to be run as separate invocations
    # with different patch directories. This default run only enforces
    # must_fail_forever on the current tree.
    veto = expect(controls["must_fail_forever"], "fail", "forever")
    if veto:
        json.dump({"status": "INVALID", "veto": veto, "rows": rows},
                   sys.stdout, indent=2)
        return 3

    strata_rates = {}
    for stratum in card["strata"]:
        subset = [r for r in rows if r["stratum"] == stratum["id"]]
        n = len(subset) or 1
        strata_rates[stratum["id"]] = round(
            sum(r["label"] == "pass" for r in subset) / n, 4
        )

    overall = round(sum(r["label"] == "pass" for r in rows) / (len(rows) or 1), 4)
    report = {
        "status": "OK",
        "dataset": card["name"],
        "n": len(rows),
        "strata_pass_rate": strata_rates,
        "overall_pass_rate": overall,
        "note": "Do not quote overall without the strata table and status=OK.",
        "rows": rows,
    }
    json.dump(report, sys.stdout, indent=2)
    print()  # newline for shells
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it only after you replace fixture_hash with a real digest.

python3 unprintable_score.py | tee report.json
python3 -c 'import json,sys; r=json.load(open("report.json")); assert r["status"]=="OK"'

If the assert fires, you do not quote a number. You fix fixtures or the lockfile.

How to read the report so it cannot become a slide

Print strata first. Print overall last, or not at all.

Status	What you may say	What you may not say
`SKIP`	The split moved; rerun after freeze	Last week’s 91 percent still holds
`INVALID`	The grader or controls collapsed	The agent got worse / better
`OK` with one weak stratum	The agent passed A, failed B	“91 percent on coding”
`OK` balanced strata	The number plus the table	A rank against a public leaderboard

A useful sentence looks like this: “On frozen-dev tiny-fix-v1, 24 tasks, env hash H, controls green: py-assert 6/8, js-types 3/8, cfg-leak 5/8.” That sentence is longer than 91 percent. It is also harder to fake.

If you need a second agent in the comparison, replay the same frozen tree. Do not grow the set between the two runs and then rank them. Growing the set is a new experiment.

Where a free server actually helps

The method is local. You can run it on a laptop. The pain starts when you want overnight reruns, log retention, and a queue that survives a crash while you iterate on fixtures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already use MonkeyCode, its free model access and free server option are one way to host the runner and keep the four JSON files next to the job logs. That is infrastructure for the method, not a substitute for the method. Do not treat a hosted run as extra accuracy. It is the same grader contract with a machine that stays on.

You still pin the environment. You still refuse to print INVALID runs. You still publish the card. A free server does not wash a wet split.

Limits, on purpose

This harness does not estimate confidence intervals. Twenty-four tasks are a smoke study, not a population. It does not detect training-data contamination against the public web. It does not score patch quality beyond tests. It does not handle network tasks, GUI tasks, or multi-repo refactors.

It also will not stop a motivated reader from averaging strata in their head. You can only make the dishonest reduction harder by keeping status in the same JSON object as the percent.

Who should not use this approach: anyone who needs a single marketing number by Friday; anyone comparing agents across different fixture hashes; anyone grading with an LLM-as-judge while pretending the contract is deterministic; anyone who will not throw away a pretty run when a control flips.

If your job is to pick a tool for one repository, you may still use the four files. Shrink n_tasks to that repository’s real bugs. The card gets more honest when it gets more local.

Close the loop on the screenshot

Next time a 91 percent lands in standup, ask for the four files. If they do not exist, the score is unprintable. If they exist and status is not OK, the score is unprintable. If they exist and only one stratum carried the number, say that sentence instead.

The work is not ranking agents. The work is making the evaluation fail closed. When the controls pass, the percent is allowed to speak. Until then it should stay quiet.

If you want a place to leave the harness running while you review fixtures, MonkeyCode’s free server option is available to try. The four files still decide whether a number is fit to quote.

Freeze a Holdout Before You Quote a Coding-Agent Score

Casey Zhang — Thu, 17 Sep 2026 05:15:27 +0000

You are in a Friday ranking review. Someone pastes a table: Agent B hit 71 percent. Last week it was 58. The room relaxes. Then you ask whether the system prompt changed after the last failure dump. It did. Three times. The 71 is a fitted number. It is not a measurement.

That is how coding-agent scoreboards rot. You inspect failures, you edit the prompt, you rerun the same tasks, you publish the new pass rate. The worksheet became the exam. If you want a number you can defend, you freeze a holdout first and you never tune against it.

This article is a protocol, not a leaderboard. You will split a small task suite, lock the split, score two folders with two different commands, and refuse to publish anything that touched the tune set. Cheap inference and a spare server only matter at the end, as a place to rerun the frozen pass. They do not change the rules.

Why a fitted pass rate is marketing

A coding agent is a policy. Prompts, tools, retries, and stop conditions are all knobs. Every knob you turn on the same tasks moves the score toward the tasks. That is ordinary overfitting. It is not controversial in ML. It becomes controversial only when the slide says "the agent is better" instead of "the agent was fitted to this folder."

You need three things before a percentage is a measurement: a dataset with a frozen denominator, metrics that still mean something after a prompt edit, and controls that catch a broken grader. Miss any one of those and you are quoting a demo.

The protocol below is intentionally small. It fits on a laptop. It also fits on a free server if you want the lockfile to live somewhere you are not babysitting.

What you freeze, and what you do not

You freeze four objects, once:

The task identifiers in the holdout folder.
The hidden tests and the oracle patch for each of those tasks.
The metric formulas.
The agent command line you will actually quote.

You do not freeze the tune folder. You are allowed to wreck it. You may change prompts, swap tools, add retries, and read every failing diff. You may not copy a holdout task into tune because it "looks similar." You may not publish tune scores. You may not add holdout tasks after you have seen them fail.

If that feels strict, good. Strict is the point.

1. Build a tiny, honest dataset

Start with tasks you can grade in isolation. Each task is a directory, not a row in a spreadsheet. Spreadsheets hide missing oracles.

suite/
  tasks/
    T001_csv_header_drift/
      prompt.md
      repo/
      tests/
        test_hidden.py
      oracle.patch
      meta.json
    T002_retry_backoff_jitter/
      ...

meta.json stays boring on purpose:

{
  "id": "T001_csv_header_drift",
  "language": "python",
  "timeout_sec": 120,
  "oracle_must_pass": true,
  "tags": ["csv", "parser"]
}

Write every task against four rules. If a task breaks a rule, it is not in the suite yet.

The unfixed repo must fail the hidden tests. If it already passes, you are scoring a no-op.
The oracle patch must pass the same hidden tests. If it fails, your grader is wrong, not the agent.
Hidden tests are not in the prompt and not in the visible repo. If the agent can read them, you are scoring test-file editing.
Timeouts, checkout errors, and missing interpreters are not fails. They are exclusions. Exclusions leave the denominator.

Do not collect 400 tasks on day one. Twenty well-formed tasks beat two hundred demos. You can grow the suite later. You cannot un-see a holdout.

2. Split once. Write a manifest. Stop.

Use a seed, not your intuition. Intuition puts the gnarly tasks in tune so the holdout "looks fair." That is another form of fitting.

# split_suite.py
# Proposed local tool. Run it once, then commit the manifest.
from __future__ import annotations

import argparse, json, random
from pathlib import Path


def task_ids(root: Path) -> list[str]:
    tasks = sorted(p.name for p in (root / "tasks").iterdir() if p.is_dir())
    if len(tasks) < 12:
        raise SystemExit("Need at least 12 tasks before a holdout is meaningful.")
    return tasks


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--suite", type=Path, required=True)
    parser.add_argument("--seed", type=int, required=True)
    parser.add_argument("--holdout-frac", type=float, default=0.4)
    args = parser.parse_args()

    ids = task_ids(args.suite)
    rng = random.Random(args.seed)
    shuffled = ids[:]
    rng.shuffle(shuffled)
    cut = max(5, int(round(len(shuffled) * args.holdout_frac)))
    holdout = sorted(shuffled[:cut])
    tune = sorted(shuffled[cut:])
    overlap = set(tune) & set(holdout)
    if overlap:
        raise SystemExit(f"split leaked: {sorted(overlap)}")

    manifest = {
        "seed": args.seed,
        "holdout_frac": args.holdout_frac,
        "tune": tune,
        "holdout": holdout,
        "frozen": True,
        "note": "Do not edit lists after first agent run on holdout.",
    }
    out = args.suite / "split_manifest.json"
    if out.exists():
        raise SystemExit(f"{out} already exists; refusing to resplit")
    out.write_text(json.dumps(manifest, indent=2) + "\n")
    print(f"tune={len(tune)} holdout={len(holdout)} wrote {out}")


if __name__ == "__main__":
    main()

Run it like this and then treat the file as a contract:

python split_suite.py --suite ./suite --seed 20260917
git add suite/split_manifest.json suite/tasks
git commit -m "Freeze holdout split seed=20260917"

If the file already exists, the script exits. That is not a bug. Resplitting after you have seen holdout failures is how 71 percent happens.

3. Pick metrics that survive a prompt change

Pass rate alone is a trap. It moves when you drop hard tasks, when you swallow timeouts as failures, and when hidden tests are too thin. Publish a row, not a cell.

Metric	Formula	Publish?	Why
Tune pass rate	passes / attempted on tune	No	You fitted the prompt here
Holdout pass rate	passes / attempted on holdout	Yes, if frozen	This is the measurement
Exclusion rate	excluded / scheduled	Yes	Shows infra and timeout load
Oracle sanity	oracles that pass / tasks	Must be 1.0	If < 1.0, stop scoring agents
Weak-test flag	unfixed repo already passes	Must be 0	Task does not belong in the suite
Prompt fingerprint	hash of the quoted command	Yes	Stops silent knob changes

Attempted means the job produced a patch or an explicit empty diff. A missing interpreter is excluded, not failed. If you dump exclusions into failures, the agent that crashes less looks smarter. It is not.

You still need a decision table for the meeting, because someone will ask to "just add the three tasks it almost got."

Situation	Publish the holdout number?	Reason
Frozen split, frozen prompt, oracle sanity 1.0	Yes	Measurement
Prompt edited after reading holdout diffs	No	Holdout is now tune
Tasks added to holdout after first run	No	Denominator moved
Tune pass rate only	No	Fitted
Oracle patch fails hidden tests	No	Grader is broken
Timeout counted as fail	No	Infra mixed into skill

Print that table above the score. If people skip it, they are not reading a benchmark. They are reading a press line.

4. Two commands. Two folders. No third path.

The grader should not have a --all flag that publishes. Make the publish path refuse the tune folder.

# score_split.py
# Proposed grader wrapper. Wire run_task() to your existing harness.
from __future__ import annotations

import argparse, hashlib, json, subprocess, sys
from pathlib import Path


def fingerprint(argv: list[str]) -> str:
    blob = "\0".join(argv).encode()
    return hashlib.sha256(blob).hexdigest()[:16]


def load_manifest(suite: Path) -> dict:
    path = suite / "split_manifest.json"
    data = json.loads(path.read_text())
    if set(data["tune"]) & set(data["holdout"]):
        raise SystemExit("manifest overlap; split is invalid")
    return data


def oracle_ok(task_dir: Path) -> bool:
    repo = task_dir / "repo"
    patch = task_dir / "oracle.patch"
    tests = task_dir / "tests"
    r = subprocess.run(
        ["python", "-m", "pytest", str(tests), "-q"],
        cwd=apply_patch(repo, patch),  # you provide apply_patch()
        capture_output=True,
        text=True,
    )
    return r.returncode == 0


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--suite", type=Path, required=True)
    parser.add_argument("--split", choices=["tune", "holdout"], required=True)
    parser.add_argument("--publish", action="store_true")
    parser.add_argument("--agent-cmd", nargs=argparse.REMAINDER, required=True)
    args = parser.parse_args()

    if args.publish and args.split != "holdout":
        raise SystemExit("refusing to publish a non-holdout split")

    man = load_manifest(args.suite)
    ids = man[args.split]
    agent_fp = fingerprint(args.agent_cmd)
    rows = []
    for task_id in ids:
        task_dir = args.suite / "tasks" / task_id
        if not oracle_ok(task_dir):
            rows.append({"id": task_id, "status": "oracle_fail"})
            continue
        result = run_task(task_dir, args.agent_cmd)  # you provide run_task()
        rows.append(result)

    oracle_fails = [r for r in rows if r["status"] == "oracle_fail"]
    if oracle_fails:
        raise SystemExit(f"grader broken on {[r['id'] for r in oracle_fails]}")

    excluded = [r for r in rows if r["status"] == "excluded"]
    attempted = [r for r in rows if r["status"] in {"pass", "fail"}]
    passes = [r for r in attempted if r["status"] == "pass"]
    report = {
        "split": args.split,
        "publishable": bool(args.publish),
        "agent_fingerprint": agent_fp,
        "scheduled": len(ids),
        "attempted": len(attempted),
        "passed": len(passes),
        "excluded": len(excluded),
        "holdout_pass_rate": (len(passes) / len(attempted)) if attempted else None,
        "exclusion_rate": len(excluded) / len(ids) if ids else None,
        "rows": rows,
    }
    out = args.suite / f"report_{args.split}_{agent_fp}.json"
    out.write_text(json.dumps(report, indent=2) + "\n")
    print(out)
    if not args.publish:
        print("not publishable (tune or unpublished holdout)")


if __name__ == "__main__":
    main()

Daily loop on tune:

python score_split.py --suite ./suite --split tune --agent-cmd -- ./run_agent.sh

Quoted run, once, after you stop touching knobs:

python score_split.py --suite ./suite --split holdout --publish --agent-cmd -- ./run_agent.sh --prompt prompts/frozen_v3.md

Keep prompts/frozen_v3.md in the same commit as the report. If the file changes and the fingerprint does not, your wrapper is lying.

5. Where a free model and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The protocol is local. You can run it with any agent command. MonkeyCode is relevant only as cheap capacity: free model access for the tune loop, and a free server option for the overnight holdout pass so the lockfile and the report sit on a box you do not have to keep open. That is the whole product role here. No model names, no quota theater, no "superhuman" claim.

Use the free model access on --split tune while you are still reading diffs. You are not allowed to quote those scores, so burning paid capacity there is usually waste. When the prompt is frozen, send one holdout job to the free server option, write report_holdout_*.json next to split_manifest.json, and stop. If you keep iterating after that job, you no longer have a holdout. You have a second tune folder with extra latency.

If MonkeyCode disappeared tomorrow, the split, the oracle check, and the publish guard would still be the method. That is the test of whether this article is a workflow or an ad.

6. Read the report like a skeptic

A publishable report should look dull:

split: holdout
publishable: true
agent_fingerprint: 9c2e0b1a7d44f0c1
scheduled: 8
attempted: 7
passed: 3
excluded: 1
holdout_pass_rate: 0.429
exclusion_rate: 0.125

Three passes out of seven attempted is not a press release. It is a number with a denominator. The excluded task stays visible. If someone later "cleans" the suite by deleting that task, the next report is a different experiment. Say so, or do not compare the two numbers.

Watch for three quiet cheats.

First, prompt drift. The fingerprint must match the command you put in the slide. A retry flag is a different agent.

Second, task shopping. Moving one holdout failure into tune after the fact is not data cleaning. It is peeking.

Third, grader rot. If an OS update makes pytest skip a file, oracle sanity will save you. If you skip oracle sanity, you will rank agents on a suite that no longer tests what you think it tests.

Limitations

This protocol does not estimate human time. It does not produce confidence intervals. It does not prove that holdout tasks are drawn from your production distribution. A 40 percent holdout on 12 tasks is still a toy. It only answers one question: did you stop fitting before you quoted the number?

It also cannot catch contamination you cannot see. If the hidden tests were copied from a public kata, a model may have seen them in pretraining. A freeze does not fix leakage. It only fixes your own peeking.

Flaky tests will still lie. Run the oracle twice if your suite has time or network noise. If the oracle is flaky, the task is not ready.

Who should not use this

Do not use this split if you have fewer than twelve tasks. You cannot freeze a holdout you do not have.

Do not use it to certify a vendor, a hire, or a production SLA. A frozen folder of synthetic bugs is not your incident stream.

Do not use it if you need to change hidden tests during an incident. Fix production first. Rebuild the suite later. Mixing those jobs is how oracles rot.

Do not use a free shared server if the repos under test contain secrets, customer data, or private credentials. The protocol assumes throwaway tasks.

If you only want a demo that a coding agent can edit a file, skip the holdout. Say it is a demo. The damage starts when the demo grows a percentage.

What you can say after a frozen pass

You can say: on this frozen holdout, with this command line, the agent passed 3 of 7 attempted tasks, excluded 1 of 8 scheduled, and every oracle still passes. You cannot say the agent is better than most developers. You cannot say it is superhuman. You cannot say last week's 58 percent is comparable unless the manifest, the tests, and the fingerprint match.

That sentence is longer than 71 percent. It is also true. If you want a cheap box to run the frozen pass overnight, MonkeyCode's free server option is sufficient for this protocol. The number still comes from the holdout, not from the box.

Label Infra Failures Before You Rank Coding Agents

Casey Zhang — Wed, 16 Sep 2026 03:36:32 +0000

You finish a weekend bake-off on a cheap shared box. Agent A sits at 62 percent. Agent B sits at 81 percent. You are already drafting the post.

Then syslog loads. Four of A's misses are Broken pipe during git clone. One is disk-full on /tmp. Two are pytest importing a stale .pyc after a previous job died mid-write. You did not measure models. You measured weather on the box.

This piece is a method for labeling those misses. The percentage you quote should be about agent skill. Not about who drew the noisy hour.

The mixing problem

Public threads keep saying models already outcode most developers. That claim might hold on a clean bench. It does not hold for a CSV that treats every non-zero exit as "the agent failed."

Shared boxes make the mix worse. Preempted CPUs, noisy neighbors, rate limits, and flaky disks land in the same column as a wrong patch. Publish that column and you are doing marketing in a lab coat.

You need four buckets. Only one belongs in the skill score.

Four buckets, one rule

Give every unsuccessful or unfinished run exactly one label before you aggregate.

MODEL — the agent produced a patch or a refusal, the harness stayed healthy, and the tests still fail for a task reason.
HARNESS — your runner, prompt template, grader, or sandbox config is wrong. The agent never had a fair shot.
INFRA — SSH drops, disk full, host OOM, provider 5xx, rate limits, wall-clock preemption. The task was not scored.
FLAKY_TEST — the same task commit fails or passes when you rerun with the agent disabled.

Rule: skill score equals MODEL successes divided by MODEL successes plus MODEL failures. INFRA, HARNESS, and FLAKY_TEST are censored. You report them. You do not fold them into "the model is worse."

That denominator change is the method. It is boring. It is also how you stop lying to yourself.

A 16-task holdout, not a trophy case

Do not scrape a public leaderboard and call it a dataset. Build a small holdout you can freeze.

Defend these selection rules:

Each task has a failing test you can run with the agent turned off.
The gold patch, if you have one, is absent from the prompt, the README, and comments the agent will read.
At least four tasks are negative controls that are already green. An agent that "fixes" them by rewriting working code is a grader bug.
At least four tasks need a multi-file edit. Single-function toys inflate MODEL success.
Record license, commit SHA, and a one-line reason the task belongs.

Store one JSON object per task.

{"id": "holdout-04", "repo": "example/billing", "sha": "a1b2c3d", "test_cmd": "pytest -q tests/test_prorate.py", "expect_fail_without_patch": true, "negative_control": false, "multi_file": true, "license": "MIT"}

Sixteen tasks is enough to debug the method. It is not enough to crown a winner. Print that caveat in the same paragraph as any rate.

Metrics that survive a noisy box

You will want a single pass rate. Publish the split instead.

eligible_n: rows labeled MODEL
resolved_rate: MODEL successes / eligible_n
infra_censor_rate: INFRA / scheduled_n
harness_error_rate: HARNESS / scheduled_n
flaky_rate: FLAKY_TEST / scheduled_n
time_to_first_green_s: MODEL successes only
bytes_changed: git diff --stat, so silent no-ops cannot count as wins

If infra_censor_rate is high, you do not have a model comparison. You have a hosting diary. Stop. Fix the box or throw the batch out.

Controls you log on every run

Number them so a reviewer can audit you.

Harness commit SHA and container digest.
Task-suite SHA of the JSONL file.
Clock: start, end, timezone, NTP status.
Host fingerprint: uname -a, load average, free disk, memory.
Reachability of git and the model endpoint before the first prompt.
Retry policy: zero retries on the published number. Retries belong in an appendix.
Decoding settings as the API returned them, not as you hoped.
A probe that must fail: empty patch against a red test. If it "passes," your grader is broken.

Drop this probe into the job preamble.

#!/usr/bin/env bash
set -euo pipefail
{
  echo "ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "host=$(hostname)"
  uname -a
  uptime
  df -h / /tmp
  free -m || true
  git -C "$HARNESS_DIR" rev-parse HEAD
  sha256sum "$SUITE_JSONL"
  curl -sS -o /dev/null -w "endpoint_http=%{http_code}\n" --max-time 10 "$MODEL_HEALTH_URL" || echo "endpoint_http=down"
} | tee "$RUN_DIR/fingerprint.txt"

If free disk is already tight, label the whole batch INFRA. Do not start.

Artifact: classify the log

The script below is a proposal. Tune the patterns on your harness. Read the borderline rows yourself.

#!/usr/bin/env python3
"""Label one coding-agent run. Proposal — tune patterns on your harness."""
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

INFRA = [
    r"Broken pipe",
    r"Connection reset by peer",
    r"No space left on device",
    r"Killed",
    r"Out of memory",
    r"HTTP 429",
    r"HTTP 5\d\d",
    r"deadline exceeded",
    r"Could not resolve host",
]
HARNESS = [
    r"ModuleNotFoundError: No module named 'harness'",
    r"ERROR: file not found: pytest",
    r"unknown flag:",
    r"sandbox config",
]

def first_match(text: str, patterns: list[str]) -> str | None:
    for p in patterns:
        if re.search(p, text, re.I):
            return p
    return None

def classify(log: str, tests_without_agent_pass: bool | None, tests_after_pass: bool | None) -> dict:
    if tests_without_agent_pass is True and tests_after_pass is True:
        return {"label": "FLAKY_TEST", "reason": "suite already green with agent disabled"}
    if (m := first_match(log, INFRA)):
        return {"label": "INFRA", "reason": m}
    if (m := first_match(log, HARNESS)):
        return {"label": "HARNESS", "reason": m}
    if tests_after_pass is True:
        return {"label": "MODEL", "reason": "eligible success", "success": True}
    if tests_after_pass is False:
        return {"label": "MODEL", "reason": "eligible miss", "success": False}
    return {"label": "HARNESS", "reason": "grader produced no boolean"}

def main() -> None:
    run_dir = Path(sys.argv[1])
    log = (run_dir / "combined.log").read_text(errors="replace")
    meta = json.loads((run_dir / "meta.json").read_text())
    result = classify(
        log,
        meta.get("tests_without_agent_pass"),
        meta.get("tests_after_pass"),
    )
    result["run_id"] = meta.get("run_id")
    (run_dir / "label.json").write_text(json.dumps(result, indent=2) + "\n")
    print(result["label"], result["reason"])

if __name__ == "__main__":
    main()

Keep meta.json tiny:

{
  "run_id": "2026-09-16T04-11Z-holdout-04",
  "agent": "A",
  "task_id": "holdout-04",
  "tests_without_agent_pass": false,
  "tests_after_pass": false
}

Aggregate only after every row has a label.

# aggregate.py — proposal
import json
from collections import Counter
from pathlib import Path

rows = [json.loads(p.read_text()) for p in Path("runs").glob("*/label.json")]
c = Counter(r["label"] for r in rows)
model = [r for r in rows if r["label"] == "MODEL"]
wins = sum(1 for r in model if r.get("success"))
eligible = len(model)
print("scheduled", len(rows))
print("counts", dict(c))
print("resolved_rate", None if eligible == 0 else round(wins / eligible, 3))
print("infra_censor_rate", round(c["INFRA"] / len(rows), 3) if rows else None)

Read that printout before you open a charting tool.

Decision table

Symptom	First check	Label	In skill score?
SSH / clone / 5xx / 429 / disk / OOM	`fingerprint.txt`	INFRA	No, censor
Grader exception, missing pytest, bad cwd	harness SHA	HARNESS	No, fix then rerun
Agent disabled, tests already pass	negative control	FLAKY_TEST or bad task	No
Patch applied, harness healthy, tests red	diff + log	MODEL miss	Yes
Patch applied, tests green, negative control untouched	diff + log	MODEL hit	Yes
Agent rewrites a green negative control	diff	HARNESS or grader	No until explained

If two labels could apply, pick the earlier failure in wall-clock order. An OOM before the model returns is INFRA, not MODEL.

Where free model access and a free server fit

You do not need a fancy cluster to practice this. You need logs, a frozen suite, and a box that can finish a batch.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option. That pair helps when you are debugging the taxonomy: run the same 16 tasks, keep the fingerprint file, and see whether your INFRA rate is the method or the host. It is not a reason to skip labels. If the free server is noisy that day, the honest result is a high infra_censor_rate and no ranking.

Why these numbers are not marketing

Marketing wants one percentage and a winner. A lab report wants a denominator you can defend.

You do not get to say models outcode most developers from 16 tasks on a shared box. You do not get to say it when a third of the misses are SSH.

Publish this block, or do not publish a number:

suite SHA and task-selection rules
eligible_n and scheduled_n
resolved_rate on MODEL rows only
infra, harness, and flaky rates
retry policy (zero for the headline)
a sentence that the set is a method check, not a ranking of humanity

If a vendor quotes a score without those lines, treat it as copy. Not as measurement.

Limitations, and who should skip this

The regex classifier will mislabel. A model that prints Connection reset by peer inside a fixture can look like INFRA. Read those rows.

Four buckets are coarse. A wrong test pin is not the same as a confused planner. You can add sublabels later. Do not add them in the same week you still mix SSH into skill.

Free hosts can censor so many runs that eligible_n collapses. Then the method worked. It told you not to rank anyone.

Do not use this approach if you need production SLAs, if you cannot freeze the suite, if your tests hit the live network, or if your goal is a launch post. Do not compare agents that saw different prompts, timeouts, or days without saying so.

Skip it if you will not look at diffs. Labels without diffs are another costume.

A closing check

Before you quote a figure, open one failing log. Write the bucket on paper.

If you hesitate, the number is not ready.

Plant Bugs Your Agent Must Find: A Self-Test for Coding-Agent Benchmarks

Casey Zhang — Mon, 14 Sep 2026 21:26:01 +0000

The one question that invalidates a benchmark

Your suite says the agent fixed 91% of the bugs. Someone in review asks: has this harness ever returned a negative result on a task where you already knew the answer? You scroll the logs. It has never failed. That is not a clean bill of health. That is an untested instrument.

Coding-agent benchmarks get validated at the task level — is this bug realistic, is this repo representative — and almost never at the harness level. Can this rig detect a failure at all? If it cannot, your denominator is fiction, and the percentage above it is decoration.

Here is a canary self-test you can stand up in an afternoon. It uses planted bugs with known ground truth, two prompt phrasings, and four metrics that describe the rig instead of the agent.

Why a high fix rate can survive a broken harness

Two independent things have to hold: the tasks must resemble real work, and the runner must be able to observe failure. Teams audit the first and assume the second.

A concrete case of the second failing: the runner shells out to pytest in a directory where the new test file was never collected, the agent says everything passes, and the scoreboard records a success. Nothing in a standard report distinguishes that from a genuine fix.

The fix is not a better rubric. It is a set of tasks whose correct answer you control before the agent ever touches them.

Step 1: Write a canary manifest

Keep canaries small, boring, and fixed. Each entry names a fixture repo, a planted defect (or null for a clean control), and the single observable signal that proves the defect was detected.

# canary.yaml
version: 1
canaries:
  - id: c-offby-one
    repo: fixtures/range-utils
    planted: 'page_range() drops the final page'
    expected_signal: 'tests/test_range.py::test_last_page'
  - id: c-null-guard
    repo: fixtures/profile-api
    planted: 'None check removed before .strip()'
    expected_signal: 'AttributeError'
  - id: c-clean-a
    repo: fixtures/range-utils
    planted: null
    expected_signal: 'no failure reported'
  - id: c-clean-b
    repo: fixtures/profile-api
    planted: null
    expected_signal: 'no failure reported'

Two clean canaries matter as much as the planted ones. A rig that flags everything is as useless as one that flags nothing, and only the clean controls reveal that.

Step 2: Run canaries through the production adapter

Use the exact adapter your real suite uses. A canary that passes through a special code path proves nothing about production.

The runner below is a sketch, not a finished product — you will replace run_agent with a wrapper around your agent's CLI. Adapt the timeout and the reset step to your fixtures.

# canary_run.py
import json, pathlib, subprocess, time
import yaml

ADAPTER = './run_agent.sh'   # wrapper: <prompt> <repo>

def reset(repo):
    subprocess.run(['git', '-C', repo, 'checkout', '--', '.'], check=True)
    subprocess.run(['git', '-C', repo, 'clean', '-fd'], check=True)

def run_agent(prompt, repo, timeout=900):
    proc = subprocess.run([ADAPTER, prompt, repo],
                          capture_output=True, text=True, timeout=timeout)
    return {'rc': proc.returncode, 'stdout': proc.stdout, 'stderr': proc.stderr}

def main(manifest='canary.yaml', out='canary_runs.json'):
    spec = yaml.safe_load(pathlib.Path(manifest).read_text())
    rows = []
    for c in spec['canaries']:
        repo = c['repo']
        for phrasing in spec.get('phrasings', ['A', 'B']):
            reset(repo)
            prompt = build_prompt(c, phrasing)
            t0 = time.time()
            result = run_agent(prompt, repo)
            rows.append({'id': c['id'], 'planted': c['planted'],
                         'expected': c['expected_signal'], 'phrasing': phrasing,
                         'elapsed_s': round(time.time() - t0, 1), 'result': result})
    pathlib.Path(out).write_text(json.dumps(rows, indent=2))
    return rows

build_prompt is where paraphrase variance lives. Wording A is terse; wording B adds a sentence of context and changes the verb. Both ask for the same observation.

Reset between runs is not optional. If the agent leaves a fixture dirty, your next canary measures the previous run.

Step 3: Report harness metrics, not agent metrics

Now you can count things that mean something.

def summarize(rows):
    planted = [r for r in rows if r['planted']]
    clean   = [r for r in rows if not r['planted']]
    caught  = [r for r in planted
               if signal_in(r['expected'], r['result']['stdout'] + r['result']['stderr'])]
    quiet   = [r for r in clean if r['result']['rc'] == 0]
    cs = len(caught) / len(planted) if planted else float('nan')
    qr = len(quiet) / len(clean) if clean else float('nan')
    return {'canary_sensitivity': cs, 'quiet_rate': qr,
            'n_planted': len(planted), 'n_clean': len(clean)}

Define the four numbers once and keep the definitions fixed across every report you publish:

Canary sensitivity (CS) — share of planted defects the rig detected. This is a ceiling on every agent score you will ever publish from it.
Quiet rate (QR) — share of clean canaries correctly reported clean. A low QR means your harness leaks noise into results.
Paraphrase drift (PD) — canaries whose verdict flips between wording A and B. Drift is a property of the harness, not of the agent.
Cost per canary — wall-clock seconds and tokens per canary row. Multiply by your real suite size before you commit to a run.

One caveat: signal_in doing substring matching is a first cut. When it disagrees with what you see by eye, fix the matcher. Do not edit the log.

Step 4: Read the decision table before you quote anything

CS	QR	PD	What the numbers support
1.0	1.0	0.0	Harness passes its own check. Agent scores are interpretable within this rig.
< 1.0	any	any	Every agent score is capped by CS. Fix the rig first; the ranking is unreadable.
any	< 1.0	any	Clean canaries are failing. Agents may be penalized for harness noise.
any	any	> 0.0	Results depend on prompt wording. Freeze the wording and rerun before comparing runs.

The pessimistic row is the useful one. A harness that detects 4 of 6 planted bugs cannot be quoted at a resolution finer than two-thirds, no matter how many decimal places your report prints.

Where a free tier genuinely fits

Running canaries is repetitive and cheap per call, which is exactly the shape of work a free tier absorbs well. MonkeyCode's free model access and free server option are relevant here because the canary pass is throwaway infrastructure: you need a sandbox for the fixture repos and a lot of small model calls that you do not want on your production budget.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A working order that keeps the spend honest:

Run CS, QR, and PD on canaries using free model access. Iterate on the matcher and the prompt wording while the marginal cost is zero.
Confirm the rig passes its own check before touching the paid run. A harness that fails its canaries will happily burn budget producing uninterpretable numbers.
Move the real suite to paid capacity only after the self-test is green, then rerun a subset of canaries inside the paid environment to check that the environment itself did not change behavior.

Availability details are operator-supplied, not benchmark results. The operator states that the free tier includes access to models at no cost and a free server option, along with a free allowance of up to 10 million tokens. Terms like those change; check the project's current page before you wire it into a pipeline, and do not treat a quoted allowance as a guarantee of duration or throughput.

Limitations and who should not use this

A canary self-test validates the instrument, not the task set. It says nothing about whether your fixtures resemble production code.
Four canaries give you a coarse yes/no. Sensitivity estimates get meaningful with dozens, and each addition costs a run.
Substring signal matching produces false negatives against verbose agents that report a failure without reproducing the exact expected string.
Do not push proprietary or regulated source code through any third-party free tier unless your own review clears the terms. If you are air-gapped or need audited evaluation, this workflow is not for you; build the harness internally first.
If your goal is a statistically powered comparison between two agents, this is step zero, not the whole method. Sensitivity and quiet rate come before any interval estimation.

The habit worth keeping

Before any number leaves your terminal, ask what would have to be true for the harness to report a false success. Then plant that condition and watch whether the rig notices.

If you run this on your own suite, the interesting failure is paraphrase drift — two wordings, two verdicts. That is the one that quietly invalidates a quarter of comparison posts. If you find a pair that flips, share the wording difference in the comments; a reproducible harness bug is worth more than another leaderboard.

Match the Human Clock Before You Quote Superhuman Coding

Casey Zhang — Sun, 13 Sep 2026 19:37:17 +0000

You get the Slack ping at 9:14. A VP pasted a thread that says coding agents already outrun most software developers. Friday is the vendor meeting, and you are the person who has to turn that thread into a hire-or-buy decision.

That thread is not a measurement. It is a mood. If you quote it in the meeting, a staff engineer will ask one question you cannot answer: under what clock, on which tasks, with which tests visible, against which humans?

This article is a protocol for that question. You will seal a holdout, freeze the snapshot, time-box both sides, and refuse any superhuman sentence the table cannot support. The numbers you emit are local results. They are not a profession-wide ranking.

The claim that keeps leaking into standups

Public agent scores and "better than developers" headlines almost never share a protocol. The agent often sees the tests. The human does not get the same tools. Retries are unlimited on one side and calendar-bound on the other.

You cannot fix that with a bigger screenshot. You fix it by matching the human protocol and by keeping a holdout the internet has never ranked.

Here is the rule you will enforce. If a junior on your team gets 45 minutes, a laptop, a failing test command, and no extra browser tab, the agent gets the same 45 minutes, the same snapshot, the same command, and the same visibility. Anything else is a demo.

What you will produce

You will leave with four artifacts, not a leaderboard row.

A private task pack that you do not publish.
A leak check against public issue titles and URLs.
A time-boxed runner that records hidden-test pass, time-to-green, and steer count.
A decision table that tells you which sentences you may say out loud.

Label every number this protocol emits as a local result. It is not market share. It is not "most developers."

Step 1 — Plant bugs in a repo the internet does not own

Do not start with famous GitHub issues. Those issues are posters. They have comments, patches, blog write-ups, and eval forks. Your holdout should be a fixture repo you control.

Create a tiny service with real tests. Then plant defects a junior could reasonably fix in under an hour. Keep gold patches off the machine the agent can read.

holdout/
  repo/                 # the snapshot both sides receive
  tasks.json            # sealed; not committed to a public remote
  denylist.txt          # public issue URLs and titles you refuse
  hidden_tests/         # pytest files the agent does not see during the run
  visible_tests/        # the command both sides may execute

A task record should look like this.

{
  "task_id": "H-04",
  "title": "invoice totals ignore refunded line items",
  "visible_command": "pytest visible_tests/test_invoice.py -q",
  "hidden_command": "pytest hidden_tests/test_invoice_refunds.py -q",
  "time_box_sec": 2700,
  "language": "python",
  "notes": "defect planted 2026-09-14; never filed on a public tracker"
}

Twelve tasks is enough to catch a dishonest workflow. It is not enough to crown a profession. Write that sentence on the README before you run anything.

Step 2 — Run a leak check before the first agent call

If a task title, stack trace, or function name already lives on a public benchmark, you do not have a holdout. You have a rerun.

Keep a denylist of URLs and phrases you will not recycle. Scan your task pack against it before every session. The script below is a tripwire, not a contamination paper.

#!/usr/bin/env python3
"""holdout_clock.py — proposed local protocol. Not a published score."""
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

WORD = re.compile(r"[a-z0-9]+")


def tokens(text: str) -> set[str]:
    return {t for t in WORD.findall(text.lower()) if len(t) > 3}


def leak_hits(task: dict, denylist: list[str]) -> list[str]:
    hay = tokens(task["title"] + " " + task.get("notes", ""))
    hits = []
    for line in denylist:
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        needle = tokens(line)
        if not needle:
            continue
        overlap = hay & needle
        if len(overlap) >= 4 or line.lower() in task["title"].lower():
            hits.append(line)
    return hits


def check_leak(pack_path: Path, denylist_path: Path) -> int:
    tasks = json.loads(pack_path.read_text())
    denylist = denylist_path.read_text().splitlines()
    bad = 0
    for task in tasks:
        hits = leak_hits(task, denylist)
        if hits:
            bad += 1
            print(f"LEAK {task['task_id']}: {hits[:3]}")
    print(f"checked={len(tasks)} leak_suspects={bad}")
    return 1 if bad else 0

Run it like this:

python3 holdout_clock.py check-leak

If it prints LEAK, you rewrite the task. You do not "fix" the leak by paraphrasing a well-known public issue. Plant a new bug.

This check is coarse. Four-token overlap is a stop sign, not proof that a model trained on the text. Treat a hit as do-not-use.

Step 3 — Freeze the snapshot both sides will touch

You will hand the human and the agent the same tarball. Same commit. Same lockfile. Same toolchain.

If one side gets a newer pytest plugin, you threw away the comparison. Freeze first. Argue later.

git -C holdout/repo rev-parse HEAD > holdout/SNAPSHOT.sha
python3 -m pip freeze > holdout/requirements.lock
tar -C holdout -czf holdout-snapshot.tgz repo visible_tests SNAPSHOT.sha requirements.lock
sha256sum holdout-snapshot.tgz

Store the digest next to the results. If the digest moves, the table is void. You start over.

Hidden tests stay out of the tarball. That is the point. The visible command is the only oracle both sides may use while the clock runs. You score hidden tests after the time box, on a copy the worker cannot edit.

Step 4 — Match the human clock, including the boring parts

Write the human protocol down before anyone sits down. Then copy it onto the agent runner. Do not improvise mid-session.

Wall clock starts at clone and stops at 45 minutes or at the first green visible suite, whichever comes first.
Network is off except the model API you already approved. No docs spelunking the human did not also get.
The agent may run visible_command as often as it wants. It may not list hidden_tests/.
A steer is any human edit to the prompt, the plan, or the patch after the clock starts. Count it.
Timeouts are failures, not "inconclusive, please retry overnight."

You now have a fairer unit than "the model eventually produced a patch." You have a session.

The runner below is a proposed local helper. It does not call a model. It only enforces the clock and the two test commands.

import subprocess
import time
from pathlib import Path


def run_time_box(task: dict, role: str, repo: Path) -> dict:
    start = time.monotonic()
    limit = int(task["time_box_sec"])
    timed_out = False
    visible_pass = False
    hidden_pass = False
    elapsed = None

    try:
        visible = subprocess.run(
            task["visible_command"],
            cwd=repo,
            shell=True,
            capture_output=True,
            text=True,
            timeout=limit,
        )
        elapsed = time.monotonic() - start
        visible_pass = visible.returncode == 0
    except subprocess.TimeoutExpired:
        timed_out = True
        elapsed = limit

    if visible_pass and elapsed is not None:
        remaining = max(1, limit - elapsed)
        hidden = subprocess.run(
            task["hidden_command"],
            cwd=repo,
            shell=True,
            capture_output=True,
            text=True,
            timeout=remaining,
        )
        hidden_pass = hidden.returncode == 0

    return {
        "task_id": task["task_id"],
        "role": role,
        "visible_pass": visible_pass,
        "hidden_pass": hidden_pass,
        "time_to_green_sec": elapsed if visible_pass else None,
        "timed_out": timed_out,
        "steer_count": None,  # fill by hand; do not invent it
    }

Fill steer_count yourself. A script cannot see you rewriting the prompt. If you leave it null and then claim the agent was unsupervised, you forged the row.

Step 5 — Record metrics marketing decks skip

Pass rate on visible tests is the demo metric. You will still log it. You will not lead with it.

Use this result schema. Fill it per task, per role.

{
  "task_id": "H-04",
  "role": "agent",
  "visible_pass": true,
  "hidden_pass": false,
  "time_to_green_sec": 1412,
  "timed_out": false,
  "steer_count": 2,
  "notes": "visible suite green; hidden refund case still red"
}

Then compute four numbers on the sealed pack only:

Hidden pass rate. Did the patch survive tests the worker never saw?
Time-to-green. Seconds to a green visible suite, or null on timeout.
Steer rate. Steers per task. An unsupervised agent with a high hidden pass and zero steers is a different object than a chat window you babysat.
Public-minus-holdout delta. If you also run a famous public set, subtract. A fat positive delta means the public set is flattering you.

None of these numbers generalize to "most software developers." They generalize to this holdout, this clock, this junior protocol. That is still enough to reject a vendor.

Step 6 — Compare with a table, not a vibe

After both roles finish, print a comparison you can paste into the Friday doc. The function below is a worksheet. It uses a mean because the code is short. If you need a median, compute a median. Do not dress sum / len as a confidence interval.

def summarize(rows: list[dict]) -> dict:
    def take(role: str) -> list[dict]:
        return [r for r in rows if r["role"] == role]

    def hidden(role: str) -> float:
        group = take(role)
        return sum(1 for r in group if r["hidden_pass"]) / len(group)

    def mean_tto(role: str) -> float:
        times = [
            r["time_to_green_sec"]
            for r in take(role)
            if r["time_to_green_sec"] is not None
        ]
        return sum(times) / len(times) if times else float("nan")

    agent = take("agent")
    return {
        "n": len(agent),
        "agent_hidden_pass": hidden("agent"),
        "human_hidden_pass": hidden("human"),
        "agent_mean_tto_sec": mean_tto("agent"),
        "human_mean_tto_sec": mean_tto("human"),
        "agent_steers": sum((r["steer_count"] or 0) for r in agent),
        "agent_timeouts": sum(1 for r in agent if r["timed_out"]),
    }

Wire the commands so a skeptic can replay the worksheet:

python3 holdout_clock.py check-leak
python3 holdout_clock.py run --role human --minutes 45
python3 holdout_clock.py run --role agent --minutes 45
python3 holdout_clock.py compare results.json

If check-leak is red, you do not run compare. You do not average a contaminated pack into a friendlier number.

When you may quote a sentence

Use this table. If a cell says no, you do not get to tighten the wording. You drop the claim.

Sentence you want to publish	Allowed when	Never allowed when
"On this sealed pack, the agent matched our junior under a 45-minute clock."	Same snapshot, same visible tests, hidden tests scored, n and steers reported	Hidden tests were in the prompt
"The agent was faster to a green visible suite on these tasks."	Time-to-green logged for both roles, timeouts counted as failures	You discarded timeouts or reran only the agent
"Public-set accuracy was higher than holdout accuracy."	You ran both sets without editing tasks after seeing misses	You tuned prompts on the holdout
"AI is better at coding than most software developers."	—	Always, from this protocol

That last row is the point. A local junior is not most developers. Twelve planted bugs are not the industry. If a vendor needs that sentence, they need a different study, with humans you actually sampled.

The dataset is the sealed pack plus the snapshot digest. The metrics are hidden pass, time-to-green, steers, and timeouts. The controls are the matched clock, the hidden tests, the denylist, and the frozen tarball. Those four controls are why the numbers are not marketing. Remove any one of them and you are back to a demo.

Where a free model and a free server fit

You still need somewhere to run the agent side without turning the eval into a credit-card subplot.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. That pair is useful here because the protocol wants a stable box and a model endpoint you can call inside the time box, not a scavenger hunt across trial accounts. It does not replace the holdout. It does not turn a hidden-pass rate into a press release. If you try it, run the same holdout-snapshot.tgz you gave the human and judge the run with the table above.

Limitations, and who should not use this

This protocol will lie to you if you treat it as a championship.

It is a single language in the example. It plants bugs instead of mining production incidents. The leak check will miss paraphrases. A dozen tasks has almost no statistical power. Steer counting is only as honest as the human who logs it. A junior on your team is not a random developer, and a 45-minute clock is not a week-long feature.

Do not use this approach if you need a headline. Do not use it if you cannot recruit even one human baseline. Do not use it if your tasks already live on a public leaderboard. Do not use it to fire people. Do not use it to skip code review. A green hidden suite on a fixture repo is not a production incident closed.

What you take to Friday

You do not take a viral thread. You take a snapshot digest, a leak-check log, two role columns, and the sentences the table permits.

If the agent wins on hidden tests under the same clock, you have a local fact. If it only wins on visible tests, you have a memorizer. If it only wins after six steers, you have a pair programmer with extra latency. Name the thing you actually measured.

That is slower than quoting the internet. It is also the only version of the argument that will still make sense when the thread is gone.

Replay the Same 24 Tasks Twice Before You Rank Two Coding Agents

Casey Zhang — Sat, 12 Sep 2026 17:33:04 +0000

You drop two coding-agent scores into a Friday review. One slide says 71 percent. The other says 68 percent. Someone asks which stack to keep paying for, and the room goes quiet because nobody can name the unit behind those percentages.

Was a task one failing test, one repository, or one overnight session with hidden retries? If you cannot replay the same 24 tasks tomorrow and land inside a tiny drift band, you do not have a benchmark. You have a screenshot.

This piece is a methodology you can run yourself. It is not a leaderboard. It will not tell you which model is “best.” It will tell you whether a comparison is even legal.

Start from a scene, not a metric

Picture a checkout service with a flaky discount rule. You give Agent A and Agent B the same failing test, the same fixture files, and the same wall clock. Agent A edits three files and the test goes green. Agent B rewrites a helper, the test stays red, then a second attempt passes after it deletes an assertion.

If your scoreboard counts both as a win, you just mixed a repair with a vandalized oracle. Marketing loves that mix. You should not.

Name three objects before you write a single number.

The dataset: a frozen slice of tasks, each with a fixture directory and a hidden test command.
The oracle: the exact command that decides green, plus a checksum of the files it is allowed to read.
The unit of analysis: one attempt on one task, not a chat session with unbounded retries.

Until those three are pinned, pass rate is a mood.

Build a 24-task slice you can actually replay

You do not need 500 tasks to practice honesty. You need a slice small enough to re-run twice on one machine. Twenty-four is enough to feel variance and small enough to read every failure by hand.

Use a directory layout like this:

slice24/
  TASKS.md
  controls.yaml
  tasks/
    T01_discount_rule/
      prompt.md
      repo/
      oracle/
        test_discount.py
        sha256.txt
    T02_date_parse/
      ...
  runs/
    replay_a.jsonl
    replay_b.jsonl

TASKS.md is the human contract. Each row names the bug in one sentence, the language, and whether the oracle is allowed to see the production file the agent may edit. If the oracle reads a file the agent can rewrite, your “pass” is untrusted.

Keep the slice boring on purpose. Prefer bugs you could explain on a whiteboard: off-by-one, timezone parse, nil map, wrong status code, missing unique index. Skip anything that needs a live network. Skip anything whose gold patch is already in a public training dump you cannot audit.

A decision table helps you reject tasks before they contaminate the slice.

Candidate task	Keep?	Why
Failing unit test, local fixtures only	Yes	Oracle is a command you can pin
“Improve this repo” with no test	No	No oracle, only vibes
Needs GitHub auth or a paid API	No	Control plane leaks into the score
Agent may edit the test file	No	Oracle is no longer independent
Passes only after a second hidden retry	No	Unit of analysis drifted
Output depends on current date	No	Replay will move

If a row fails that table, it does not enter tasks/. It does not get a footnote. It is out.

Pin controls like they are part of the dataset

A dataset without controls is a prompt dump. Write controls.yaml first, then refuse to change it mid-bake-off.

# Proposal: example controls for a 24-task replay slice.
# Label this as a template until you fill every field on your machine.
slice_id: slice24-checkout-2026-09-13
unit_of_analysis: one_attempt_per_task
max_attempts: 1
wall_clock_sec: 180
cpus: 2
memory_mb: 2048
network: denied
writable_paths: ["repo/"]
oracle_cmd: "python -m pytest oracle/ -q"
oracle_sha256_file: oracle/sha256.txt
seed_policy: fixed
language_toolchain: "python3.11"
forbidden_edits: ["oracle/", "controls.yaml"]
replay_required: 2
max_replay_drift: 0

Read that file out loud. The important lines are max_attempts, network, and replay_required. One attempt means you are scoring the first patch, not the best patch after a human babysits the loop. Denied network means the agent cannot shop for answers. Two replays with zero allowed drift means the environment is a scientific instrument, not a laptop that happened to be free that afternoon.

If you later raise max_attempts from 1 to 3, you started a new study. Do not append those runs onto the old JSONL and call it an update.

Score paired deltas, not a headline percent

You will still compute a pass count. You will not lead with it.

For each task t, store four facts:

pass_a: 1 if Agent A’s attempt made the pinned oracle green
pass_b: 1 if Agent B did
oracle_intact: 1 if oracle/ still matches sha256.txt
replay_match: 1 if attempt 1 and attempt 2 agreed for that agent

Then report the paired picture:

Wins for A: tasks where A passed and B failed, oracle intact
Wins for B: the reverse
Ties: both passed or both failed
Void: oracle mutated, timeout disagreement, or replay mismatch

A void is not a loss. A void is a broken instrument. If more than two of 24 tasks void, stop ranking and fix the harness.

The number you may quote is narrow: “On slice24, with one attempt and a frozen oracle, A won 5, B won 3, 14 tied, 2 void.” That sentence is ugly. It is also hard to turn into an ad.

A harness you can run twice

The following is a labeled proposal. It does not claim production metrics. Swap run_agent with your real CLI. Keep the oracle check. If you skip the checksum, the rest of the file is theater.

# proposal_harness.py — replay gate for a 24-task slice
from __future__ import annotations

import hashlib, json, subprocess, time
from pathlib import Path

SLICE = Path("slice24")
WALL_CLOCK = 180

def sha256_tree(root: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(root.rglob("*")):
        if p.is_file():
            h.update(p.relative_to(root).as_posix().encode())
            h.update(p.read_bytes())
    return h.hexdigest()

def oracle_intact(task: Path) -> bool:
    expected = (task / "oracle" / "sha256.txt").read_text().strip()
    return sha256_tree(task / "oracle") == expected

def run_oracle(task: Path) -> bool:
    proc = subprocess.run(
        ["python", "-m", "pytest", str(task / "oracle"), "-q"],
        cwd=task / "repo",
        capture_output=True,
        timeout=WALL_CLOCK,
    )
    return proc.returncode == 0

def run_agent(agent_id: str, task: Path) -> None:
    prompt = (task / "prompt.md").read_text()
    # Replace this stub with your agent CLI. Keep the timeout.
    subprocess.run(
        ["your-agent", "--id", agent_id, "--prompt", prompt, "--cwd", str(task / "repo")],
        timeout=WALL_CLOCK,
        check=False,
    )

def one_attempt(agent_id: str, task: Path) -> dict:
    if not oracle_intact(task):
        return {"task": task.name, "void": "oracle_before"}
    t0 = time.monotonic()
    run_agent(agent_id, task)
    elapsed = round(time.monotonic() - t0, 3)
    if not oracle_intact(task):
        return {"task": task.name, "agent": agent_id, "void": "oracle_mutated", "sec": elapsed}
    passed = run_oracle(task)
    return {"task": task.name, "agent": agent_id, "pass": int(passed), "sec": elapsed, "void": None}

def replay_agent(agent_id: str) -> list[dict]:
    rows = []
    for task in sorted((SLICE / "tasks").iterdir()):
        first = one_attempt(agent_id, task)
        second = one_attempt(agent_id, task)
        if first.get("pass") != second.get("pass") or first.get("void") or second.get("void"):
            rows.append({"task": task.name, "agent": agent_id, "void": "replay_drift"})
        else:
            rows.append(first)
    return rows

def paired_report(a: list[dict], b: list[dict]) -> dict:
    by_a = {r["task"]: r for r in a}
    wins = {"A": 0, "B": 0, "tie": 0, "void": 0}
    for r in b:
        t = r["task"]
        left, right = by_a[t], r
        if left.get("void") or right.get("void"):
            wins["void"] += 1
        elif left.get("pass") and not right.get("pass"):
            wins["A"] += 1
        elif right.get("pass") and not left.get("pass"):
            wins["B"] += 1
        else:
            wins["tie"] += 1
    return wins

if __name__ == "__main__":
    a_rows = replay_agent("A")
    b_rows = replay_agent("B")
    Path("slice24/runs/replay_a.jsonl").write_text(
        "\n".join(json.dumps(r) for r in a_rows) + "\n"
    )
    Path("slice24/runs/replay_b.jsonl").write_text(
        "\n".join(json.dumps(r) for r in b_rows) + "n"
        if False else "\n".join(json.dumps(r) for r in b_rows) + "\n"
    )
    print(json.dumps(paired_report(a_rows, b_rows), indent=2))

Run it twice on purpose. The outer replay_agent already doubles each task. You still re-launch the process after a reboot. If the JSONL changes, your machine is part of the score. Fix the machine before you argue about models.

Useful commands while you debug the instrument:

find slice24/tasks -name sha256.txt -print
python -m pytest slice24/tasks/T01_discount_rule/oracle -q
sha256sum slice24/controls.yaml
python proposal_harness.py
diff -u slice24/runs/replay_a.jsonl slice24/runs/replay_a.jsonl.bak

If diff is not empty, you do not discuss Agent A versus Agent B. You discuss clocks, caches, and pytest plugins.

Why these numbers are not marketing

A vendor deck wants one percentage and a bold winner. This slice refuses that in four ways.

First, n equals 24 paired tasks, not 24 independent coin flips. Several tasks will share a language and a style of bug. You may still count wins. You may not pretend you have a population study.

Second, the unit is one attempt. Hidden retries, extra tool calls after timeout, and “I nudged it in chat” are protocol violations. They belong in a different paper.

Third, void tasks exist. Marketing absorbs voids into the denominator or drops them. You surface them. A method that cannot fail in public is not a method.

Fourth, replay drift is a stop condition, not a footnote. If the same agent on the same fixture flips pass/fail, the score is environmental noise. Publishing a mean over noise is how leaderboards rot.

You can add cost later. You can add failure taxonomy later. None of that is legal until replay is boring.

Where a free server actually helps

Methodology work burns tokens on scaffolding: broken prompts, oracle mistakes, path bugs. That traffic should not touch a production budget.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need model access and a machine without standing up your own GPU, MonkeyCode’s free model access and free server option are enough to host this 24-task slice while you harden the harness. Use that only for the replay loop. Move any claim you would show a finance team onto hardware you control and a slice you can checksum.

That is the whole product note.

Limitations, and who should not use this

This approach is wrong for several honest jobs.

Do not use a 24-task slice to pick a company-wide coding agent. The interval is wide, the tasks are local, and language coverage is thin. Do not use it for multi-hour refactors, GUI work, or anything that needs the public internet. Do not use it if your oracle is “a reviewer liked the patch.” Human taste is not a sha256.

Do not compare agents that do not share a tool surface. If Agent A can run pytest and Agent B can only emit a diff, you measured the toolchain. Say that, or stop.

Do not publish the pass count without the controls file, the slice id, the void count, and the replay rule. A number without those four is a caption.

If your goal is a press quote by Monday, this workflow will slow you down. That is the point. Rankings are cheap. Replay is the filter that keeps you from buying a percentage that cannot sit still.

Break Agent Ties With Cost Per Fix and a Bootstrap Interval

Casey Zhang — Fri, 11 Sep 2026 15:20:19 +0000

You walk into the same meeting that has been eating coding-agent reviews all year. Two harnesses ran on the same forty-task pack. Agent A prints 62.5 percent resolved. Agent B prints 57.5 percent. Someone already put A on the slide.

Then the invoice, or the GPU hours, or the queue delay shows up. A looped the test runner, rewrote one file three times, and still needed a human merge. B failed more tasks. When it passed, it passed in one shot.

If you published only the pass rate, you published a brochure. Rank agents by cost per resolved task, and refuse a winner when the confidence intervals overlap. The numbers below are a worked example, not a lab report.

The ranking that the slide hid

Pass rate answers one question: did the suite go green. It does not answer how much work the agent burned to get there. It does not tell you whether a three-point gap would survive a different seed.

Hot takes keep claiming coding agents already outrun most developers. Treat those claims as untrusted until the protocol shows the unit of cost and the uncertainty. You need both, or you are ranking vibes.

Freeze a price table if money matters. Freeze token counts and wall-clock even when the invoice is zero. Tokens and seconds are still scarce on a free endpoint.

What the ledger must record

Do not start with a dashboard. Start with one JSONL row per task attempt. If a field is missing, you do not get to quote a ranking.

Each row needs:

run_id, agent_id, task_id, seed
resolved as a boolean from a frozen checker, not from the model’s self-report
prompt_tokens, completion_tokens, tool_calls
wall_ms measured outside the vendor’s stream
price_table_id so a later reader can see which rates you applied
notes for human merges, skipped tests, or harness crashes

Keep monetary cost as a derived column. Multiply tokens by the frozen table. If the table is all zeros because you used a free model, you still rank on tokens and wall-clock. Zero dollars is not zero work.

A sample pack you can score without an API key

Label this data as illustrative. Eight rows, two agents, four tasks. Tiny on purpose. The point is the method, not a leaderboard.

{"run_id":"r1","agent_id":"a","task_id":"t1","seed":7,"resolved":true,"prompt_tokens":1800,"completion_tokens":900,"tool_calls":4,"wall_ms":14000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"a","task_id":"t2","seed":7,"resolved":true,"prompt_tokens":4200,"completion_tokens":3100,"tool_calls":11,"wall_ms":41000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"a","task_id":"t3","seed":7,"resolved":false,"prompt_tokens":5100,"completion_tokens":4400,"tool_calls":14,"wall_ms":56000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"a","task_id":"t4","seed":7,"resolved":true,"prompt_tokens":2600,"completion_tokens":1200,"tool_calls":5,"wall_ms":18000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"b","task_id":"t1","seed":7,"resolved":true,"prompt_tokens":900,"completion_tokens":400,"tool_calls":2,"wall_ms":8000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"b","task_id":"t2","seed":7,"resolved":false,"prompt_tokens":1600,"completion_tokens":800,"tool_calls":3,"wall_ms":12000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"b","task_id":"t3","seed":7,"resolved":true,"prompt_tokens":1100,"completion_tokens":500,"tool_calls":2,"wall_ms":9000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"b","task_id":"t4","seed":7,"resolved":false,"prompt_tokens":1500,"completion_tokens":700,"tool_calls":4,"wall_ms":11000,"price_table_id":"local-0"}

Save it as sample_ledger.jsonl. Agent A looks like the winner if you only count greens. Wait for the division.

Step 1: Derive the three columns that matter

Resolved rate is still useful. It is not sufficient. Compute three numbers per agent, then stop talking until the intervals exist.

Resolved rate: greens divided by tasks
Tokens per resolved (TPR): total tokens divided by greens. If greens is zero, the agent has no TPR. It failed the suite.
Seconds per resolved (SPR): total wall-clock divided by greens

Money per resolved is optional. Attach it only after you freeze price_table_id. Do not mix a March rate card with a September run.

Step 2: Score the ledger locally

This script is a worked example. It does not call a model. Point it at any JSONL that matches the schema.

# ledger_stats.py — worked example, not a published benchmark
from __future__ import annotations

import argparse
import json
import random
from collections import defaultdict
from pathlib import Path


def load_rows(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text().splitlines():
        line = line.strip()
        if line:
            rows.append(json.loads(line))
    return rows


def group(rows: list[dict]) -> dict[str, list[dict]]:
    by_agent = defaultdict(list)
    for row in rows:
        by_agent[row["agent_id"]].append(row)
    return by_agent


def point_estimates(rows: list[dict]) -> dict:
    n = len(rows)
    greens = [r for r in rows if r["resolved"]]
    tokens = sum(r["prompt_tokens"] + r["completion_tokens"] for r in rows)
    wall_s = sum(r["wall_ms"] for r in rows) / 1000.0
    resolved = len(greens)
    return {
        "n": n,
        "resolved": resolved,
        "resolved_rate": resolved / n if n else 0.0,
        "tpr": tokens / resolved if resolved else None,
        "spr": wall_s / resolved if resolved else None,
        "tokens": tokens,
    }


def bootstrap_rate(rows: list[dict], rounds: int, seed: int) -> tuple[float, float]:
    rng = random.Random(seed)
    n = len(rows)
    stats = []
    for _ in range(rounds):
        sample = [rows[rng.randrange(n)] for _ in range(n)]
        stats.append(point_estimates(sample)["resolved_rate"])
    stats.sort()
    lo = stats[int(0.025 * rounds)]
    hi = stats[int(0.975 * rounds)]
    return lo, hi


def intervals_overlap(a: tuple[float, float], b: tuple[float, float]) -> bool:
    return not (a[1] < b[0] or b[1] < a[0])


def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--path", type=Path, required=True)
    p.add_argument("--bootstrap", type=int, default=2000)
    p.add_argument("--seed", type=int, default=7)
    args = p.parse_args()

    by_agent = group(load_rows(args.path))
    summary = {}
    for agent, rows in sorted(by_agent.items()):
        est = point_estimates(rows)
        lo, hi = bootstrap_rate(rows, args.bootstrap, args.seed)
        est["ci95"] = (round(lo, 3), round(hi, 3))
        summary[agent] = est
        print(agent, est)

    ids = list(summary)
    if len(ids) == 2:
        a, b = ids
        overlap = intervals_overlap(summary[a]["ci95"], summary[b]["ci95"])
        print("ci_overlap", overlap)
        if overlap:
            tpr = {k: summary[k]["tpr"] for k in ids}
            print("tie_break_tpr", tpr)
            print("no_winner_on_rate")


if __name__ == "__main__":
    main()

Run it:

python ledger_stats.py --path sample_ledger.jsonl --bootstrap 2000 --seed 7

On this toy pack, A resolves 3/4 and B resolves 2/4. The intervals on four tasks are wide. They will overlap. That is the lesson, not a bug.

Step 3: Refuse a winner when the intervals overlap

A three-point gap on forty tasks can be noise. A fifteen-point gap on four tasks is still noise. Bootstrap the resolved bit, not the marketing headline.

Rule, in order:

If either agent has zero greens, do not compute TPR. Report failure, not a ranking.
If the 95 percent intervals on resolved rate overlap, you do not have a rate winner.
Only then compare TPR. Lower tokens per green wins the tie.
If TPR is within 10 percent, compare SPR. If both still tie, publish a draw.

The 10 percent band is a policy choice. Put it in the protocol. Do not tune it after you see the chart.

Step 4: Pin the rule with a test

If the tie-break lives only in a slide, it will drift. Lock it in a test you can run on a laptop.

# test_ledger_stats.py
from ledger_stats import intervals_overlap, point_estimates


def test_overlap_means_no_rate_winner():
    a = (0.40, 0.80)
    b = (0.35, 0.70)
    assert intervals_overlap(a, b)


def test_separated_intervals_are_not_a_tie():
    a = (0.60, 0.85)
    b = (0.20, 0.45)
    assert not intervals_overlap(a, b)


def test_tpr_uses_greens_not_attempts():
    rows = [
        {"resolved": True, "prompt_tokens": 100, "completion_tokens": 100, "wall_ms": 1000},
        {"resolved": False, "prompt_tokens": 900, "completion_tokens": 900, "wall_ms": 9000},
    ]
    est = point_estimates(rows)
    assert est["resolved_rate"] == 0.5
    assert est["tpr"] == 200  # failed tokens still sit in the numerator? change this if you prefer

Read that last assertion twice. The sample puts all tokens in the numerator, including failed attempts. That punishes thrash. If you only divide tokens from successful tasks, a noisy agent looks cheap. Pick one definition. Write it on the protocol. Do not switch after the ranking is ugly.

python -m pytest test_ledger_stats.py -q

Failed-attempt accounting is the whole fight. Include the waste, or you are ranking luck.

Decision table: what you may print

Situation	You may print	You may not print
Intervals overlap, TPR gap > 10%	Draw on rate; B cheaper per fix	"B is worse"
Intervals overlap, TPR gap ≤ 10%	Draw	Any crown
Intervals separated, higher rate also higher TPR	Rate winner, with the cost warning	"Strictly better"
Intervals separated, higher rate also lower TPR	Rate and cost winner	Human-level claims
N < 30	Exploratory notes	Vendor ranking
Price table not frozen	Tokens and seconds only	USD rankings
Checker not frozen	Nothing	Anything

Print the table next to the chart. If a cell says no, delete the sentence.

Where a free lane belongs

Run this harness off the production key. Eval traffic that shares a paid quota with users will contaminate both the bill and the latency column.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option are one way to keep a small ledger off paid production traffic. Point the agent at whatever OpenAI-compatible endpoint you already trust, write JSONL, and score it with the script above. That is the only product role here: a place to burn eval tokens without mixing them into customer traffic. No model names, no quota theater, no claimed speedup.

A free server is enough for this scale. Forty tasks, one process, a disk for JSONL. If you need a multi-GB sandbox per task, this lane is the wrong hardware story. Say so and stop.

Who should not use this

Do not use a four-task pack to rank vendors. Do not use bootstrap-on-tasks as a substitute for a held-out human study. Do not turn TPR into a hiring rubric. Do not compare a local 7-second checker against a cloud agent that waits on a queue and then claim a latency win.

Skip this method if you cannot freeze the checker. Skip it if tool schemas change between agents. Skip it if one agent may call the network and the other may not. Those are different tasks. They do not belong on one leaderboard.

Wall-clock on a free server will jitter. Treat SPR as a coarse filter, not a microbenchmark. If you need tail latency, provision dedicated hardware and say so in the protocol.

What you still may not claim

You may claim that, on this frozen pack, with this seed, with this checker, agent B spent fewer tokens per green while the rate intervals overlapped. You may not claim B is the better engineer. You may not claim the model is “better than most developers.” That sentence needs a human protocol you have not run.

Ship the ledger with the post. Ship the script. Ship the overlap rule. If a reader cannot replay the ranking from the JSONL, you do not have a number. You have a slide.

Hash the Suite. Lock the Controls. Then You May Quote a Number.

Casey Zhang — Wed, 09 Sep 2026 15:32:26 +0000

You get the ping at 16:40. "We're at 81% now. Can marketing have it?"

You ask which tasks, which commit, which temperature, which timeout, which tools, and what a pass even means. They send a screenshot. One green tile. No files.

That number is a press release. A benchmark is a replay kit: a frozen dataset, a metric spec, a control block, and one command that rebuilds the table from those artifacts. If you cannot rerun it, you cannot defend it.

This is a methodology, not a leaderboard. The scripts below are a template. Comments that look like scores are illustrative. They are not results from a private suite I ran for you.

What "not marketing" actually means

A marketing number is a point estimate with no denominator, no variance, and no way to reproduce the run. A benchmark number is boring on purpose. It names the dataset version, the metric formula, the controls locked before the first call, and the cost envelope that produced it.

You ship four files, not a percentage:

tasks.jsonl — the suite, then a hash of that file.
metrics.py — the only functions allowed to score a run.
controls.yaml — seed, temperature, timeout, retries, tool allowlist, model endpoint.
replay.sh — one command that rebuilds report.md from the three files above.

If any of those four is missing, you do not publish a score. You publish a demo.

The job this kit is built for

You are comparing two coding agents on a private suite of repository tasks. Each task is a git tree plus a hidden test command. You need the harness to sit overnight without a credit-card surprise, and you need the same files to replay on Monday morning.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access and a free server option. You can point the control block at that endpoint and leave the replay kit running unattended. Strip the product out and the four files still work against any OpenAI-compatible base URL. The host is plumbing. The methodology is the point.

Step 1 — Freeze the dataset, then hash it

Do not score against a living folder named tasks/. Living folders drift. Someone tweaks a fixture. Someone "cleans up" a prompt. Your 81% becomes a different experiment with the same label.

Export one JSONL file. One object per task. Then hash the bytes you will actually feed the harness.

# example layout — adjust paths; do not treat these counts as a published n
python3 - <<'PY'
import hashlib, json, pathlib
src = pathlib.Path("tasks.jsonl")
raw = src.read_bytes()
digest = hashlib.sha256(raw).hexdigest()
print(f"tasks.jsonl sha256={digest} bytes={len(raw)} lines={raw.count(10)}")
PY

Keep the schema tiny so you cannot hide extra hints in random fields:

{"id": "repo-017", "repo": "fixtures/repo-017.tar", "hidden_test": "pytest -q tests/test_hidden.py", "timeout_s": 120, "size_bucket": "sm", "tags": ["api", "refactor"]}

Rules you write down before the first call:

Hidden tests never enter the prompt. If the agent can read tests/test_hidden.py, the task is contaminated.
size_bucket is assigned by lines of production code, not by how you feel about the ticket.
You do not add, drop, or rewrite a task after controls.yaml is locked. You cut a new suite version instead.

If you cannot explain how a task entered the suite, it does not enter the suite.

Step 2 — Write the metric spec as code, not as a slide

Pass rate is a ratio. Ratios hide the denominator, the variance, and the cost of being right. Your metric module should refuse to print a single percentage without those three.

# metrics.py — template, not a published result
from __future__ import annotations

import math
from dataclasses import dataclass

@dataclass(frozen=True)
class Trial:
    task_id: str
    passed: bool
    timeout: bool
    tool_violation: bool
    prompt_tokens: int
    completion_tokens: int
    wall_s: float
    retries_used: int

def wilson_interval(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n <= 0:
        raise ValueError("n must be > 0")
    p = k / n
    z2 = z * z
    denom = 1.0 + z2 / n
    center = (p + z2 / (2 * n)) / denom
    half = (z * math.sqrt((p * (1 - p) + z2 / (4 * n)) / n)) / denom
    return (max(0.0, center - half), min(1.0, center + half))

def summarize(trials: list[Trial]) -> dict:
    n = len(trials)
    k = sum(1 for t in trials if t.passed)
    lo, hi = wilson_interval(k, n)
    tokens = sum(t.prompt_tokens + t.completion_tokens for t in trials)
    walls = sum(t.wall_s for t in trials)
    return {
        "n": n,
        "passes": k,
        "pass_rate": k / n,
        "wilson95": [lo, hi],
        "timeout_rate": sum(t.timeout for t in trials) / n,
        "tool_violation_rate": sum(t.tool_violation for t in trials) / n,
        "tokens_per_pass": (tokens / k) if k else None,
        "wall_s_per_task": walls / n,
        "retries_mean": sum(t.retries_used for t in trials) / n,
    }

Notice what this refuses to do. It will not let you quote pass_rate without n and a Wilson interval. It will not let a timeout look like a wrong answer. It will not let a tool violation look like a model miss. Cost sits next to correctness. That is the whole point.

A number is marketing when it is a point estimate. A number is a measurement when it carries a sample size, an interval, and a cost.

Step 3 — Lock the control block before the first token

Controls are the difference between an eval and a vibe. You write them down. Then you stop touching them.

# controls.yaml
suite_file: tasks.jsonl
suite_sha256: REPLACE_WITH_HASH
model_base_url: http://127.0.0.1:8080/v1
model_name: FROM_ENDPOINT_NOT_FROM_A_BLOG
temperature: 0.0
seed: 7
max_retries: 0
per_task_timeout_s: 180
max_output_tokens: 2048
tool_allowlist:
  - read_file
  - apply_patch
  - run_hidden_test
forbidden_paths:
  - tests/test_hidden.py
  - .git/
stop_on_control_mismatch: true

Number the lock procedure. Do not skip a line.

Fill suite_sha256 from Step 1. If the file bytes change, the run aborts.
Set max_retries to 0 for the headline table. Retries are a different product. Mix them in and you are scoring a loop, not an agent.
Pin temperature and seed. Creative sampling is a demo setting.
Name every allowed tool. An unbounded shell is not a coding-agent benchmark. It is a sysadmin benchmark.
Record the endpoint URL, not a nickname. Nicknames drift when someone "upgrades" the proxy.

If a later run cannot load this file and reproduce the same process table, the later run is a new experiment. Give it a new name.

Step 4 — Make replay the only way to print a table

Humans edit dashboards. Scripts edit report.md. You want the second one.

#!/usr/bin/env bash
# replay.sh — rebuild report.md from frozen inputs only
set -euo pipefail
python3 - <<'PY'
import hashlib, json, pathlib, sys, yaml
ctrl = yaml.safe_load(pathlib.Path("controls.yaml").read_text())
raw = pathlib.Path(ctrl["suite_file"]).read_bytes()
got = hashlib.sha256(raw).hexdigest()
if got != ctrl["suite_sha256"]:
    sys.exit(f"suite hash mismatch: got {got}")
print("controls ok; suite hash ok")
PY
python3 harness.py --controls controls.yaml --out runs.jsonl
python3 report.py --runs runs.jsonl --controls controls.yaml --out report.md

harness.py is your agent loop. Keep it dull. For each task it must write one JSONL row with the Trial fields from metrics.py. No extra narrative. No "mostly correct" flag. Binary pass, timeout, tool violation, tokens, wall clock, retries used.

report.py should render a table you would actually paste into a review:

| field | value |
|---|---|
| suite_sha256 | abc… |
| n | 40 |
| passes | 17 |
| pass_rate | 0.425 |
| wilson95 | 0.285 – 0.578 |
| timeout_rate | 0.100 |
| tool_violation_rate | 0.050 |
| tokens_per_pass | 12840 |
| wall_s_per_task | 74.2 |

Those interior figures are format examples. They are not a claim about any vendor, model, or night of running. If you publish a table, it must come from replay.sh on your suite, with your hash in the first row.

Why this table is harder to use as marketing

Look at the interval, not the point. On n=40, a 0.425 pass rate has a wide Wilson band. Anyone quoting "43%" without that band is rounding a small sample into a slogan.

Look at cost per pass, not cost per call. Cheap failures are not efficiency. They are a model that gives up early.

Look at timeout rate and tool-violation rate as first-class rows. If your "smarter" agent climbs pass rate by blowing the timeout budget or by reading hidden tests, you did not get smarter. You changed the rules.

That is the control story in one sentence: you freeze the rules so the number cannot be rescued by a quiet rule change.

A decision table you can keep next to the PR

Use this before anyone pastes a percentage into a README.

Question	If the answer is no	What you publish instead
Can you hash `tasks.jsonl` and match `controls.yaml`?	The suite moved.	"demo, suite unpinned"
Is pass defined by a hidden test the agent cannot read?	You scored the prompt.	"prompt eval, not agent eval"
Are retries zero in the headline run?	You scored a loop.	a separate retry study
Do you report n and a 95% interval?	You scored a point.	no public number
Do you report tokens per pass and wall time?	You hid the bill.	correctness plus envelope
Can a stranger rerun `replay.sh`?	You scored a screenshot.	internal note only

If two rows fail, you do not negotiate. You withhold the number.

Limitations

This kit does not make a small suite large. Wilson intervals get honest, not narrow. Private tasks leak the moment an engineer pastes one into a chat window. Free servers preempt. Free model endpoints can change behavior without sending you a changelog. Pin what you can pin. Do not pretend a pin is a contract.

The harness also cannot see contamination you failed to encode. If hidden tests live in the tarball under another name, forbidden_paths will not save you. If the prompt already contains the patch, you are scoring a copy task.

Do not treat OpenAI-compatible as "same model." Base URL, model name, and decoding knobs are part of the control block. Change one, and you start a new run id.

Who should not use this

Do not use this kit as a procurement score if you cannot freeze tools and tests. Buyers who need a certified figure need a lab protocol and a contract, not a JSONL file on a free server.

Do not use it if your n is a handful of golden tickets you already memorized. You will get a tight-looking rate and a worthless interval.

Do not use it to compare agents that were allowed different tools, different retry budgets, or different hidden tests. That comparison is a category error. Split the studies.

If you only need a live demo for a hallway, skip the kit. Run the demo. Do not call it a benchmark.

Run it overnight, then read the files

Copy the four files onto a quiet machine. Fill the hash. Keep retries at zero. Start replay.sh. In the morning you read report.md, not a chat transcript.

If you want that quiet machine to be a free server with free model access, MonkeyCode can host the same scripts. The replay kit stays the source of truth either way.