Make a Coding-Agent Score Unprintable Until the Controls Pass

You are in standup with a screenshot. Someone says the new coding agent hit 91 percent. Nobody can name the tasks, the grader, the runtime pin, or the cases that were supposed to fail on purpose.

The number still travels. Slack copies it. A slide copies Slack. By Friday the method is gone and the percentage is the only artifact left.

That is not a benchmark. It is a demo with a percent sign. If you want a number you can defend, you need four files that can veto the headline. Until those files exist and the controls pass, the score should refuse to print.

This is a small, reproducible method. It is not a league table. It will not make a model look superhuman. It will make a weak evaluation obvious.

The problem you are actually measuring

Coding agents fail in clusters. A Python fixture leak, a flaky assertion, a wrong language version, or a grader that accepts a stub can move a score more than the model does.

You cannot see that in a single pass rate. A pass rate is a reduction. Reductions are useful after you publish the cuts you made. They are marketing when you hide the cuts.

So you stop leading with the percent. You lead with a dataset card, a grader contract, an environment lock, and a control set. The percent is a footnote that the report is allowed to emit only when the controls stay green.

Four files, then a number

Keep the working directory boring. Four JSON documents plus one runner is enough for a laptop-scale study.

1. dataset_card.json — what the tasks are, how they were chosen, and which strata they occupy.
2. grader_contract.json — what counts as pass, fail, skip, or invalid.
3. env.lock.json — language, test runner, dependency hash, timeout, seed.
4. controls.json — tasks the agent must pass and tasks the agent must fail, or the run is discarded.

If any file is missing, you do not have a score. You have a log.

Step 1: Write the dataset card first

Do not harvest random GitHub issues and call it a set. Write the card before the first agent call. The card is the claim. The agent is only a subject under that claim.

{
  "name": "tiny-fix-v1",
  "n_tasks": 24,
  "split": "frozen-dev",
  "source": "internal fixtures, not scraped leaderboards",
  "strata": [
    {"id": "py-assert", "n": 8, "language": "python", "skill": "failing-test-repair"},
    {"id": "js-types", "n": 8, "language": "javascript", "skill": "type-narrowing"},
    {"id": "cfg-leak", "n": 8, "language": "python", "skill": "config-isolation"}
  ],
  "exclusions": ["network required", "gpu required", "multi-hour jobs"],
  "contamination_note": "fixtures authored for this card; do not mix public interview puzzles"
}

The strata matter more than the total. Eight tasks in one skill and sixteen in another is not a balanced 24. If you later quote one number, the card must still show the mix. Readers can reject the mix. That is the point.

Label this card as a proposal until a human reviews every fixture. Do not pretend 24 tasks represent production engineering.

Step 2: Freeze a grader contract people can argue with

A hidden grader is a second model. You would not accept an unlabeled second model in the loop. Publish the contract.

{
  "pass": "all tests in tests/ exit 0 after the patch, with no extra files outside allowlist",
  "fail": "any test non-zero, timeout, or patch that edits tests/",
  "skip": "environment hash mismatch or missing fixture",
  "invalid": "control task outcome disagrees with expected_control",
  "partial_credit": false,
  "test_command": ["pytest", "-q", "--timeout=20"],
  "timeout_sec": 120,
  "max_patch_bytes": 20000
}

Notice what is absent. There is no “looks good to the author.” There is no hidden rubric. Partial credit is off on purpose. If you want partial credit later, add it as a second metric with its own contract. Do not smuggle it into the headline.

If two people cannot apply this contract to the same patch and get the same label, the contract is not frozen. Fix the contract before you rank anything.

Step 3: Lock the environment or admit the score is wet

You already know this from CI. Python 3.11 versus 3.12 can flip assertions. A floating pytest plugin can swallow failures. Pin it.

{
  "python": "3.11.9",
  "node": null,
  "pytest": "8.3.2",
  "seed": 501,
  "os": "linux",
  "fixture_hash": "sha256:replace-me-after-tar-cf",
  "network": false
}

Compute fixture_hash from a tarball of the task directories, not from a folder you keep editing. If the hash moves, the split is no longer frozen. Print SKIP for the whole run. Do not average old and new tasks in silence.

Step 4: Plant controls that can veto the run

Positive controls are tasks a stub should still fail and a known-good patch should pass. Negative controls are tasks that must remain failing if the grader is honest: a missing assertion, a tautology, a test file the agent is forbidden to edit.

{
  "must_pass_with_gold": ["py-assert/01", "js-types/03"],
  "must_fail_with_empty_patch": ["py-assert/01", "cfg-leak/02"],
  "must_fail_forever": ["controls/tautology-green", "controls/deleted-assert"]
}

If a gold patch fails must_pass_with_gold, your tests are broken. If an empty patch passes must_fail_with_empty_patch, your grader is broken. If must_fail_forever goes green, someone taught the agent to cheat the tests, or the tests never tested anything.

In all three cases the runner prints INVALID and no percent.

A runner that refuses to market itself

The artifact below is a complete, local harness. It does not call a vendor. It reads the four files, applies a patch directory, and writes a report that keeps the headline gated.

Save it as unprintable_score.py.

#!/usr/bin/env python3
"""Gate a coding-agent score behind dataset, grader, env, and controls."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent


def load(name: str) -> dict:
    return json.loads((ROOT / name).read_text())


def sha_tree(path: Path) -> str:
    h = hashlib.sha256()
    for p in sorted(path.rglob("*")):
        if p.is_file():
            h.update(p.relative_to(path).as_posix().encode())
            h.update(p.read_bytes())
    return "sha256:" + h.hexdigest()


def run_tests(task_dir: Path, contract: dict) -> str:
    try:
        proc = subprocess.run(
            contract["test_command"],
            cwd=task_dir,
            capture_output=True,
            timeout=contract["timeout_sec"],
        )
    except subprocess.TimeoutExpired:
        return "fail"
    return "pass" if proc.returncode == 0 else "fail"


def main() -> int:
    card = load("dataset_card.json")
    contract = load("grader_contract.json")
    env = load("env.lock.json")
    controls = load("controls.json")

    fixture_root = ROOT / "tasks"
    current_hash = sha_tree(fixture_root)
    if current_hash != env["fixture_hash"]:
        json.dump({"status": "SKIP", "reason": "env hash mismatch",
                    "expected": env["fixture_hash"], "got": current_hash},
                   sys.stdout, indent=2)
        return 2

    rows = []
    for stratum in card["strata"]:
        stratum_dir = fixture_root / stratum["id"]
        for task_dir in sorted(p for p in stratum_dir.iterdir() if p.is_dir()):
            label = run_tests(task_dir, contract)
            rows.append({
                "task": f"{stratum['id']}/{task_dir.name}",
                "stratum": stratum["id"],
                "label": label,
            })

    by_task = {r["task"]: r["label"] for r in rows}

    def expect(names, label, bucket):
        bad = [n for n in names if by_task.get(n) != label]
        return bad

    # Gold and empty-patch checks are expected to be run as separate invocations
    # with different patch directories. This default run only enforces
    # must_fail_forever on the current tree.
    veto = expect(controls["must_fail_forever"], "fail", "forever")
    if veto:
        json.dump({"status": "INVALID", "veto": veto, "rows": rows},
                   sys.stdout, indent=2)
        return 3

    strata_rates = {}
    for stratum in card["strata"]:
        subset = [r for r in rows if r["stratum"] == stratum["id"]]
        n = len(subset) or 1
        strata_rates[stratum["id"]] = round(
            sum(r["label"] == "pass" for r in subset) / n, 4
        )

    overall = round(sum(r["label"] == "pass" for r in rows) / (len(rows) or 1), 4)
    report = {
        "status": "OK",
        "dataset": card["name"],
        "n": len(rows),
        "strata_pass_rate": strata_rates,
        "overall_pass_rate": overall,
        "note": "Do not quote overall without the strata table and status=OK.",
        "rows": rows,
    }
    json.dump(report, sys.stdout, indent=2)
    print()  # newline for shells
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it only after you replace fixture_hash with a real digest.

python3 unprintable_score.py | tee report.json
python3 -c 'import json,sys; r=json.load(open("report.json")); assert r["status"]=="OK"'

If the assert fires, you do not quote a number. You fix fixtures or the lockfile.

How to read the report so it cannot become a slide

Print strata first. Print overall last, or not at all.

Status	What you may say	What you may not say
SKIP	The split moved; rerun after freeze	Last week’s 91 percent still holds
INVALID	The grader or controls collapsed	The agent got worse / better
OK with one weak stratum	The agent passed A, failed B	“91 percent on coding”
OK balanced strata	The number plus the table	A rank against a public leaderboard

A useful sentence looks like this: “On frozen-dev tiny-fix-v1, 24 tasks, env hash H, controls green: py-assert 6/8, js-types 3/8, cfg-leak 5/8.” That sentence is longer than 91 percent. It is also harder to fake.

If you need a second agent in the comparison, replay the same frozen tree. Do not grow the set between the two runs and then rank them. Growing the set is a new experiment.

Where a free server actually helps

The method is local. You can run it on a laptop. The pain starts when you want overnight reruns, log retention, and a queue that survives a crash while you iterate on fixtures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already use MonkeyCode, its free model access and free server option are one way to host the runner and keep the four JSON files next to the job logs. That is infrastructure for the method, not a substitute for the method. Do not treat a hosted run as extra accuracy. It is the same grader contract with a machine that stays on.

You still pin the environment. You still refuse to print INVALID runs. You still publish the card. A free server does not wash a wet split.

Limits, on purpose

This harness does not estimate confidence intervals. Twenty-four tasks are a smoke study, not a population. It does not detect training-data contamination against the public web. It does not score patch quality beyond tests. It does not handle network tasks, GUI tasks, or multi-repo refactors.

It also will not stop a motivated reader from averaging strata in their head. You can only make the dishonest reduction harder by keeping status in the same JSON object as the percent.

Who should not use this approach: anyone who needs a single marketing number by Friday; anyone comparing agents across different fixture hashes; anyone grading with an LLM-as-judge while pretending the contract is deterministic; anyone who will not throw away a pretty run when a control flips.

If your job is to pick a tool for one repository, you may still use the four files. Shrink n_tasks to that repository’s real bugs. The card gets more honest when it gets more local.

Close the loop on the screenshot

Next time a 91 percent lands in standup, ask for the four files. If they do not exist, the score is unprintable. If they exist and status is not OK, the score is unprintable. If they exist and only one stratum carried the number, say that sentence instead.

The work is not ranking agents. The work is making the evaluation fail closed. When the controls pass, the percent is allowed to speak. Until then it should stay quiet.

If you want a place to leave the harness running while you review fixtures, MonkeyCode’s free server option is available to try. The four files still decide whether a number is fit to quote.