Balance the Outcome Ledger: A Publication Gate for Coding-Agent Benchmarks

Last week a teammate pasted a number into a planning doc: 78% pass rate on a 40-task agent suite. Nobody argued. The number was clean, the slide was clean, and the meeting moved on.

Then you opened the run logs. Three tasks never produced an outcome at all. The harness timed out twice, and one grader crashed on a parse error. Those three were dropped from the export, so the denominator quietly became 37. The suite was still called "40 tasks" in the title.

That is not a lie. It is worse than a lie. It is bookkeeping so sloppy that nobody can tell whether the number is true.

The front page this week was full of AI takes, including a good thread on plain AI fatigue. My honest read is that fatigue is not cured by fewer numbers. It is cured by numbers that are cheap to check. So here is a ledger rule and a small gate you can run before any agent score leaves your machine.

The rule: every task that enters must exit exactly once

A benchmark claim is a financial statement. It has entries, exits, and a reconciliation. If the ledger does not balance, the headline is not a measurement — it is a story with digits attached.

Every task attempt must land in exactly one bucket:

1. pass — the grader's success condition was satisfied.
2. fail — the task was attempted and the condition was not satisfied.
3. unscoreable — the harness itself could not produce a verdict, with a named reason class.

The conservation law is one line: pass + fail + unscoreable == tasks_started. And tasks_started must equal the declared dataset size, or you must list the tasks that never started by ID.

Step 1 — Write the manifest (JSON, no dependencies)

Keep the manifest in JSON rather than YAML so the validator runs on a stock Python install with zero packages.

{
  "claim_id": "agent-x-mini-2026-10-16",
  "dataset": {
    "name": "local-40",
    "task_count": 40,
    "task_ids_sha256": "replace-with-digest-from-gate",
    "strata": { "repo_edit": 24, "test_write": 10, "refactor": 6 }
  },
  "metrics": [
    { "name": "pass_rate", "numerator": "outcome == 'pass'", "denominator": "dataset.task_count", "unit": "tasks" },
    { "name": "unscoreable_rate", "numerator": "outcome == 'unscoreable'", "denominator": "dataset.task_count", "unit": "tasks" }
  ],
  "controls": [
    { "task_id": "neg-01", "expected": "fail", "role": "negative" },
    { "task_id": "neg-02", "expected": "fail", "role": "negative" },
    { "task_id": "pos-01", "expected": "pass", "role": "positive" }
  ],
  "envelope": {
    "runner": "monkeycode-free-server",
    "network": "off",
    "wall_clock_s_per_task": 900,
    "max_steps": 40,
    "toolchain_sha": "replace-with-toolchain-digest"
  },
  "rerun_recipe": "python gate.py --manifest bench.manifest.json --runs runs.jsonl"
}

Note two deliberate choices. Every metric is written as an explicit fraction, not as a word like score. And the negative controls carry an expected outcome, so the gate can fail when the harness is broken rather than when the model is.

Step 2 — Emit one JSONL record per attempt

One line per attempt. No aggregation at write time, no filtering at export time. Filtering is where denominators die.

{"run_id":"r1","task_id":"neg-01","attempt":1,"outcome":"fail","reason":null,"wall_clock_s":41}
{"run_id":"r1","task_id":"t-013","attempt":1,"outcome":"unscoreable","reason":"harness_timeout","wall_clock_s":900}
{"run_id":"r1","task_id":"t-027","attempt":1,"outcome":"unscoreable","reason":"grader_error","wall_clock_s":12}
{"run_id":"r1","task_id":"pos-01","attempt":1,"outcome":"pass","reason":null,"wall_clock_s":88}

Keep the allowed reason classes short and fixed: harness_timeout, grader_error, env_flake, missing_artifact. A free-text reason field will be filled with "weird" and "idk" within a week, and then you cannot count anything.

Step 3 — The gate itself

This is the whole validator. Standard library only.

#!/usr/bin/env python3
"""Publication gate for coding-agent benchmark claims. Stdlib only."""
import hashlib, json, math, pathlib, sys

BUCKETS = ("pass", "fail", "unscoreable")
REASONS = {"harness_timeout", "grader_error", "env_flake", "missing_artifact"}


def wilson(k, n, z=1.96):
    if n == 0:
        return (0.0, 0.0)
    p = k / n
    d = 1 + z * z / n
    center = (p + z * z / (2 * n)) / d
    half = (z / d) * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (max(0.0, center - half), min(1.0, center + half))


def read_jsonl(path):
    text = pathlib.Path(path).read_text()
    return [json.loads(line) for line in text.splitlines() if line.strip()]


def main(manifest_path, runs_path):
    manifest = json.loads(pathlib.Path(manifest_path).read_text())
    runs = read_jsonl(runs_path)
    errors, declared = [], manifest["dataset"]["task_count"]

    ids = sorted({r["task_id"] for r in runs})
    digest = hashlib.sha256("\n".join(ids).encode()).hexdigest()
    if digest != manifest["dataset"]["task_ids_sha256"]:
        errors.append("dataset_digest_mismatch: you ran a different suite than you published")
    if len(ids) != declared:
        errors.append(f"dataset_size_mismatch: declared {declared}, observed {len(ids)}")

    by_task = {}
    for r in runs:
        by_task.setdefault(r["task_id"], []).append(r)

    for tid, attempts in sorted(by_task.items()):
        if len(attempts) != 1:
            errors.append(f"multiple_attempts:{tid}={len(attempts)} (retries are attempts, not replacements)")
        last = attempts[-1]
        if last.get("outcome") not in BUCKETS:
            errors.append(f"unknown_outcome:{tid}={last.get('outcome')}")
        if last.get("outcome") == "unscoreable" and last.get("reason") not in REASONS:
            errors.append(f"uncoded_unscoreable:{tid}")

    counts = {b: sum(1 for r in runs if r.get("outcome") == b) for b in BUCKETS}
    if sum(counts.values()) != declared:
        errors.append(f"ledger_does_not_balance: {sum(counts.values())} exits vs {declared} entries")

    for c in manifest.get("controls", []):
        row = by_task.get(c["task_id"], [None])[-1]
        if row is None:
            errors.append(f"control_missing:{c['task_id']}")
        elif row.get("outcome") == "unscoreable":
            errors.append(f"control_inconclusive:{c['task_id']} ({row.get('reason')})")
        elif row.get("outcome") != c["expected"]:
            errors.append(f"control_violated:{c['task_id']} expected {c['expected']} got {row['outcome']}")

    k, lo, hi = counts["pass"], *wilson(counts["pass"], declared)
    scoreable = declared - counts["unscoreable"]
    k2 = counts["pass"]
    lo2, hi2 = wilson(k2, scoreable) if scoreable else (0.0, 0.0)

    print(f"claim       : {manifest['claim_id']}")
    print(f"dataset     : {manifest['dataset']['name']} sha256 {digest[:12]}")
    print(f"entries     : {declared}")
    print(f"exits       : {sum(counts.values())}")
    for b in BUCKETS:
        extra = ""
        if b == "unscoreable":
            detail = sorted((r["reason"] for r in runs if r.get("outcome") == b))
            extra = f" ({', '.join(detail)})"
        print(f"  {b:<12}: {counts[b]}{extra}")
    print(f"pass_rate   : {k}/{declared} = {k/declared:.3f}  95% CI [{lo:.3f}, {hi:.3f}]  <- headline")
    print(f"            : {k2}/{scoreable} = {k2/scoreable:.3f}  95% CI [{lo2:.3f}, {hi2:.3f}]  <- labeled secondary")
    print(f"verdict     : {'BLOCKED' if errors else 'PUBLISHABLE'}")
    for e in errors:
        print(f"  ! {e}")
    return 1 if errors else 0


if __name__ == "__main__":
    a = sys.argv
    sys.exit(main(a[a.index("--manifest") + 1], a[a.index("--runs") + 1]))

Step 4 — Run it and read the report card

python gate.py --manifest bench.manifest.json --runs runs.jsonl

claim       : agent-x-mini-2026-10-16
dataset     : local-40 sha256 3f9c1a7b0e44
entries     : 40
exits       : 40
  pass        : 29
  fail        : 8
  unscoreable : 3 (grader_error, harness_timeout, harness_timeout)
pass_rate   : 29/40 = 0.725  95% CI [0.570, 0.842]  <- headline
            : 29/37 = 0.784  95% CI [0.624, 0.889]  <- labeled secondary
verdict     : PUBLISHABLE

Both lines are real. The headline keeps the full denominator, because an unscoreable task is an outcome of your harness, not an absence from the suite. The secondary line is the number people actually want, and it is fine to show it as long as the label travels with it.

Notice how wide the interval is. On 40 tasks, 29/40 and 30/40 are not different results. If your release notes say "improved from 72% to 75%," the gate just told you that you are reading noise.

Why this makes the number a measurement and not marketing

Five properties separate the two, and the gate enforces all five.

1. A named numerator and denominator. Not "score," not "performance index." Two counts and a division.
2. A stated interval. A point estimate without uncertainty is a claim about your confidence, not about the system.
3. A first-class unscoreable bucket. Attrition is printed next to the result, with reason classes, never dropped at export.
4. Controls with expected outcomes in the manifest. If a control violates its expectation, the gate blocks the publication, because the instrument is broken.
5. A re-run recipe a stranger can execute. If someone else cannot reproduce the ledger from your manifest and logs, you published a story.

That last point is the one that costs you an afternoon and saves you a quarter.

Where free compute fits in this loop

The expensive part of a gate like this is not the arithmetic. It is the control slice and the re-run smoke test — the runs that exist purely to prove the instrument works, which feel wasteful when you are paying per call.

This is where a free tier changes the economics of doing it honestly. MonkeyCode provides free model access and a free server option, so the control slice and the smoke test can run without a spend decision blocking them. Check the current terms for the applicable allowance and availability before you depend on it, since free-tier limits and hardware change over time. Incidentally, this article was prepared as part of MonkeyCode's product outreach.

One caution that matters more than the free part. A different runner is a different envelope. Record the runner, network policy, wall clock, and toolchain digest in the manifest, and never compare a pass rate from one envelope to a pass rate from another. A ledger balances inside a single envelope. Across envelopes, you are comparing two different instruments.

If you want to try this on your own suite, run the control slice on the free server and get a PUBLISHABLE verdict from gate.py before you quote anything.

Limitations, and who should skip this

The gate validates bookkeeping, not task quality. Forty weak tasks with a balanced ledger is still a weak benchmark, and a balanced ledger will not fix a suite that leaked into training data. Wilson intervals also assume independent, identically distributed tasks, which agent benchmarks only approximate — treat the interval as a floor on your uncertainty, not a precise bound. And the reason classes above are a starting taxonomy; yours should reflect the failures your harness actually produces.

Skip this if you are doing a vibe check on a personal project — the overhead is real for ten tasks. Skip it too if you need a leaderboard number: a publication gate is necessary and nowhere near sufficient, because it says nothing about a hidden split, seed sensitivity, or task contamination. Fix those separately.

Use it if you are about to put a percentage in a doc that other people will make decisions from. Then do the boring thing first: count the entries, count the exits, and make them match.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.