Freeze a Holdout Before You Quote a Coding-Agent Score

You are in a Friday ranking review. Someone pastes a table: Agent B hit 71 percent. Last week it was 58. The room relaxes. Then you ask whether the system prompt changed after the last failure dump. It did. Three times. The 71 is a fitted number. It is not a measurement.

That is how coding-agent scoreboards rot. You inspect failures, you edit the prompt, you rerun the same tasks, you publish the new pass rate. The worksheet became the exam. If you want a number you can defend, you freeze a holdout first and you never tune against it.

This article is a protocol, not a leaderboard. You will split a small task suite, lock the split, score two folders with two different commands, and refuse to publish anything that touched the tune set. Cheap inference and a spare server only matter at the end, as a place to rerun the frozen pass. They do not change the rules.

Why a fitted pass rate is marketing

A coding agent is a policy. Prompts, tools, retries, and stop conditions are all knobs. Every knob you turn on the same tasks moves the score toward the tasks. That is ordinary overfitting. It is not controversial in ML. It becomes controversial only when the slide says "the agent is better" instead of "the agent was fitted to this folder."

You need three things before a percentage is a measurement: a dataset with a frozen denominator, metrics that still mean something after a prompt edit, and controls that catch a broken grader. Miss any one of those and you are quoting a demo.

The protocol below is intentionally small. It fits on a laptop. It also fits on a free server if you want the lockfile to live somewhere you are not babysitting.

What you freeze, and what you do not

You freeze four objects, once:

1. The task identifiers in the holdout folder.
2. The hidden tests and the oracle patch for each of those tasks.
3. The metric formulas.
4. The agent command line you will actually quote.

You do not freeze the tune folder. You are allowed to wreck it. You may change prompts, swap tools, add retries, and read every failing diff. You may not copy a holdout task into tune because it "looks similar." You may not publish tune scores. You may not add holdout tasks after you have seen them fail.

If that feels strict, good. Strict is the point.

1. Build a tiny, honest dataset

Start with tasks you can grade in isolation. Each task is a directory, not a row in a spreadsheet. Spreadsheets hide missing oracles.

suite/
  tasks/
    T001_csv_header_drift/
      prompt.md
      repo/
      tests/
        test_hidden.py
      oracle.patch
      meta.json
    T002_retry_backoff_jitter/
      ...

meta.json stays boring on purpose:

{
  "id": "T001_csv_header_drift",
  "language": "python",
  "timeout_sec": 120,
  "oracle_must_pass": true,
  "tags": ["csv", "parser"]
}

Write every task against four rules. If a task breaks a rule, it is not in the suite yet.

1. The unfixed repo must fail the hidden tests. If it already passes, you are scoring a no-op.
2. The oracle patch must pass the same hidden tests. If it fails, your grader is wrong, not the agent.
3. Hidden tests are not in the prompt and not in the visible repo. If the agent can read them, you are scoring test-file editing.
4. Timeouts, checkout errors, and missing interpreters are not fails. They are exclusions. Exclusions leave the denominator.

Do not collect 400 tasks on day one. Twenty well-formed tasks beat two hundred demos. You can grow the suite later. You cannot un-see a holdout.

2. Split once. Write a manifest. Stop.

Use a seed, not your intuition. Intuition puts the gnarly tasks in tune so the holdout "looks fair." That is another form of fitting.

# split_suite.py
# Proposed local tool. Run it once, then commit the manifest.
from __future__ import annotations

import argparse, json, random
from pathlib import Path


def task_ids(root: Path) -> list[str]:
    tasks = sorted(p.name for p in (root / "tasks").iterdir() if p.is_dir())
    if len(tasks) < 12:
        raise SystemExit("Need at least 12 tasks before a holdout is meaningful.")
    return tasks


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--suite", type=Path, required=True)
    parser.add_argument("--seed", type=int, required=True)
    parser.add_argument("--holdout-frac", type=float, default=0.4)
    args = parser.parse_args()

    ids = task_ids(args.suite)
    rng = random.Random(args.seed)
    shuffled = ids[:]
    rng.shuffle(shuffled)
    cut = max(5, int(round(len(shuffled) * args.holdout_frac)))
    holdout = sorted(shuffled[:cut])
    tune = sorted(shuffled[cut:])
    overlap = set(tune) & set(holdout)
    if overlap:
        raise SystemExit(f"split leaked: {sorted(overlap)}")

    manifest = {
        "seed": args.seed,
        "holdout_frac": args.holdout_frac,
        "tune": tune,
        "holdout": holdout,
        "frozen": True,
        "note": "Do not edit lists after first agent run on holdout.",
    }
    out = args.suite / "split_manifest.json"
    if out.exists():
        raise SystemExit(f"{out} already exists; refusing to resplit")
    out.write_text(json.dumps(manifest, indent=2) + "\n")
    print(f"tune={len(tune)} holdout={len(holdout)} wrote {out}")


if __name__ == "__main__":
    main()

Run it like this and then treat the file as a contract:

python split_suite.py --suite ./suite --seed 20260917
git add suite/split_manifest.json suite/tasks
git commit -m "Freeze holdout split seed=20260917"

If the file already exists, the script exits. That is not a bug. Resplitting after you have seen holdout failures is how 71 percent happens.

3. Pick metrics that survive a prompt change

Pass rate alone is a trap. It moves when you drop hard tasks, when you swallow timeouts as failures, and when hidden tests are too thin. Publish a row, not a cell.

Metric	Formula	Publish?	Why
Tune pass rate	passes / attempted on tune	No	You fitted the prompt here
Holdout pass rate	passes / attempted on holdout	Yes, if frozen	This is the measurement
Exclusion rate	excluded / scheduled	Yes	Shows infra and timeout load
Oracle sanity	oracles that pass / tasks	Must be 1.0	If < 1.0, stop scoring agents
Weak-test flag	unfixed repo already passes	Must be 0	Task does not belong in the suite
Prompt fingerprint	hash of the quoted command	Yes	Stops silent knob changes

Attempted means the job produced a patch or an explicit empty diff. A missing interpreter is excluded, not failed. If you dump exclusions into failures, the agent that crashes less looks smarter. It is not.

You still need a decision table for the meeting, because someone will ask to "just add the three tasks it almost got."

Situation	Publish the holdout number?	Reason
Frozen split, frozen prompt, oracle sanity 1.0	Yes	Measurement
Prompt edited after reading holdout diffs	No	Holdout is now tune
Tasks added to holdout after first run	No	Denominator moved
Tune pass rate only	No	Fitted
Oracle patch fails hidden tests	No	Grader is broken
Timeout counted as fail	No	Infra mixed into skill

Print that table above the score. If people skip it, they are not reading a benchmark. They are reading a press line.

4. Two commands. Two folders. No third path.

The grader should not have a --all flag that publishes. Make the publish path refuse the tune folder.

# score_split.py
# Proposed grader wrapper. Wire run_task() to your existing harness.
from __future__ import annotations

import argparse, hashlib, json, subprocess, sys
from pathlib import Path


def fingerprint(argv: list[str]) -> str:
    blob = "\0".join(argv).encode()
    return hashlib.sha256(blob).hexdigest()[:16]


def load_manifest(suite: Path) -> dict:
    path = suite / "split_manifest.json"
    data = json.loads(path.read_text())
    if set(data["tune"]) & set(data["holdout"]):
        raise SystemExit("manifest overlap; split is invalid")
    return data


def oracle_ok(task_dir: Path) -> bool:
    repo = task_dir / "repo"
    patch = task_dir / "oracle.patch"
    tests = task_dir / "tests"
    r = subprocess.run(
        ["python", "-m", "pytest", str(tests), "-q"],
        cwd=apply_patch(repo, patch),  # you provide apply_patch()
        capture_output=True,
        text=True,
    )
    return r.returncode == 0


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--suite", type=Path, required=True)
    parser.add_argument("--split", choices=["tune", "holdout"], required=True)
    parser.add_argument("--publish", action="store_true")
    parser.add_argument("--agent-cmd", nargs=argparse.REMAINDER, required=True)
    args = parser.parse_args()

    if args.publish and args.split != "holdout":
        raise SystemExit("refusing to publish a non-holdout split")

    man = load_manifest(args.suite)
    ids = man[args.split]
    agent_fp = fingerprint(args.agent_cmd)
    rows = []
    for task_id in ids:
        task_dir = args.suite / "tasks" / task_id
        if not oracle_ok(task_dir):
            rows.append({"id": task_id, "status": "oracle_fail"})
            continue
        result = run_task(task_dir, args.agent_cmd)  # you provide run_task()
        rows.append(result)

    oracle_fails = [r for r in rows if r["status"] == "oracle_fail"]
    if oracle_fails:
        raise SystemExit(f"grader broken on {[r['id'] for r in oracle_fails]}")

    excluded = [r for r in rows if r["status"] == "excluded"]
    attempted = [r for r in rows if r["status"] in {"pass", "fail"}]
    passes = [r for r in attempted if r["status"] == "pass"]
    report = {
        "split": args.split,
        "publishable": bool(args.publish),
        "agent_fingerprint": agent_fp,
        "scheduled": len(ids),
        "attempted": len(attempted),
        "passed": len(passes),
        "excluded": len(excluded),
        "holdout_pass_rate": (len(passes) / len(attempted)) if attempted else None,
        "exclusion_rate": len(excluded) / len(ids) if ids else None,
        "rows": rows,
    }
    out = args.suite / f"report_{args.split}_{agent_fp}.json"
    out.write_text(json.dumps(report, indent=2) + "\n")
    print(out)
    if not args.publish:
        print("not publishable (tune or unpublished holdout)")


if __name__ == "__main__":
    main()

Daily loop on tune:

python score_split.py --suite ./suite --split tune --agent-cmd -- ./run_agent.sh

Quoted run, once, after you stop touching knobs:

python score_split.py --suite ./suite --split holdout --publish --agent-cmd -- ./run_agent.sh --prompt prompts/frozen_v3.md

Keep prompts/frozen_v3.md in the same commit as the report. If the file changes and the fingerprint does not, your wrapper is lying.

5. Where a free model and a free server actually help

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The protocol is local. You can run it with any agent command. MonkeyCode is relevant only as cheap capacity: free model access for the tune loop, and a free server option for the overnight holdout pass so the lockfile and the report sit on a box you do not have to keep open. That is the whole product role here. No model names, no quota theater, no "superhuman" claim.

Use the free model access on --split tune while you are still reading diffs. You are not allowed to quote those scores, so burning paid capacity there is usually waste. When the prompt is frozen, send one holdout job to the free server option, write report_holdout_*.json next to split_manifest.json, and stop. If you keep iterating after that job, you no longer have a holdout. You have a second tune folder with extra latency.

If MonkeyCode disappeared tomorrow, the split, the oracle check, and the publish guard would still be the method. That is the test of whether this article is a workflow or an ad.

6. Read the report like a skeptic

A publishable report should look dull:

split: holdout
publishable: true
agent_fingerprint: 9c2e0b1a7d44f0c1
scheduled: 8
attempted: 7
passed: 3
excluded: 1
holdout_pass_rate: 0.429
exclusion_rate: 0.125

Three passes out of seven attempted is not a press release. It is a number with a denominator. The excluded task stays visible. If someone later "cleans" the suite by deleting that task, the next report is a different experiment. Say so, or do not compare the two numbers.

Watch for three quiet cheats.

First, prompt drift. The fingerprint must match the command you put in the slide. A retry flag is a different agent.

Second, task shopping. Moving one holdout failure into tune after the fact is not data cleaning. It is peeking.

Third, grader rot. If an OS update makes pytest skip a file, oracle sanity will save you. If you skip oracle sanity, you will rank agents on a suite that no longer tests what you think it tests.

Limitations

This protocol does not estimate human time. It does not produce confidence intervals. It does not prove that holdout tasks are drawn from your production distribution. A 40 percent holdout on 12 tasks is still a toy. It only answers one question: did you stop fitting before you quoted the number?

It also cannot catch contamination you cannot see. If the hidden tests were copied from a public kata, a model may have seen them in pretraining. A freeze does not fix leakage. It only fixes your own peeking.

Flaky tests will still lie. Run the oracle twice if your suite has time or network noise. If the oracle is flaky, the task is not ready.

Who should not use this

Do not use this split if you have fewer than twelve tasks. You cannot freeze a holdout you do not have.

Do not use it to certify a vendor, a hire, or a production SLA. A frozen folder of synthetic bugs is not your incident stream.

Do not use it if you need to change hidden tests during an incident. Fix production first. Rebuild the suite later. Mixing those jobs is how oracles rot.

Do not use a free shared server if the repos under test contain secrets, customer data, or private credentials. The protocol assumes throwaway tasks.

If you only want a demo that a coding agent can edit a file, skip the holdout. Say it is a demo. The damage starts when the demo grows a percentage.

What you can say after a frozen pass

You can say: on this frozen holdout, with this command line, the agent passed 3 of 7 attempted tasks, excluded 1 of 8 scheduled, and every oracle still passes. You cannot say the agent is better than most developers. You cannot say it is superhuman. You cannot say last week's 58 percent is comparable unless the manifest, the tests, and the fingerprint match.

That sentence is longer than 71 percent. It is also true. If you want a cheap box to run the frozen pass overnight, MonkeyCode's free server option is sufficient for this protocol. The number still comes from the holdout, not from the box.