Freeze the Manifest Before the Agent Leaderboard

A published agent score is trustworthy only after the dataset, metric functions, and control runs are frozen. Percentages that still depend on a moving task pack, a fuzzy grader, or an unstated baseline behave like advertisements rather than measurements. Teams that skip those three locks can still learn from a trial, yet they should not ship a ranking as a result. The protocol below treats a coding-agent comparison as a small experiment with pinned inputs, not as a demo reel.

Current developer debate often treats a coding agent as already better than a typical engineer, which collapses many jobs into one percentage. That collapse is convenient for a launch post and almost useless for a lab that must reproduce the number next week. A frozen pack behaves more like a sealed envelope in an exam room than like a buffet that chefs may restock during judging. Once the envelope can be reopened, the score describes the latest prompt more than it describes the system under test.

The first lock is the dataset, and it must be hashed before any agent is allowed to see a task. A practical pack is a directory of small programming jobs with fixtures, hidden tests, and a declared time budget per item. Hidden tests stay off the prompt on purpose, because an agent that reads the grader is no longer solving the stated problem. The proposed harness below pins every file with SHA-256 so later edits cannot silently inflate a published mean.

# Proposed, unexecuted example: pin_pack.py
from __future__ import annotations

import argparse, hashlib, json, os
from pathlib import Path

SKIP_DIR = {".git", "__pycache__", ".venv", "node_modules"}

def digest_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def pin_pack(root: Path) -> dict:
    files = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in sorted(dirnames) if d not in SKIP_DIR]
        for name in sorted(filenames):
            path = Path(dirpath) / name
            rel = path.relative_to(root).as_posix()
            files.append({"path": rel, "sha256": digest_file(path), "bytes": path.stat().st_size})
    blob = json.dumps({"root": str(root.resolve()), "files": files}, separators=(",", ":")).encode()
    return {
        "pack_id": hashlib.sha256(blob).hexdigest(),
        "file_count": len(files),
        "files": files,
        "metric_version": "strict_conjunction_v1",
    }

def main() -> None:
    p = argparse.ArgumentParser(description="Freeze a coding-agent task pack")
    p.add_argument("--root", type=Path, required=True)
    p.add_argument("--out", type=Path, required=True)
    args = p.parse_args()
    manifest = pin_pack(args.root)
    args.out.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
    print(manifest["pack_id"])

if __name__ == "__main__":
    main()

The manifest is the scientific object, not the blog sentence that later quotes a rounded percentage from a dashboard. If a task is too easy, too flaky, or too English-heavy, the repair happens by issuing a new pack identifier, never by quiet edits. Quiet edits are how a leaderboard becomes a brochure, because the old number and the new number stop sharing a denominator. Reviewers should refuse a comparison that cites two means without proving both means were computed on the same manifest digest.

The second lock is the metric contract, which must be a function of artifacts rather than a vibe about code quality. Pass rate alone is a weak contract because it hides compile failures, timeouts, skipped tests, and patches that delete the assertion. An honest report records several binary events per task and then publishes the joint table, not a single blended hero number. The functions below are labeled as unexecuted examples, and a lab should replace the stub runner with its real sandbox.

# Proposed, unexecuted example: metrics.py
from __future__ import annotations

from dataclasses import asdict, dataclass

METRIC_VERSION = "strict_conjunction_v1"

@dataclass(frozen=True)
class TaskOutcome:
    task_id: str
    compiled: bool
    tests_passed: bool
    lint_clean: bool
    timed_out: bool
    deleted_assertions: bool

    @property
    def strict_pass(self) -> bool:
        return (
            self.compiled
            and self.tests_passed
            and self.lint_clean
            and not self.timed_out
            and not self.deleted_assertions
        )

def summarize(outcomes: list[TaskOutcome]) -> dict:
    n = len(outcomes) or 1
    keys = ("compiled", "tests_passed", "lint_clean", "timed_out", "deleted_assertions", "strict_pass")
    rates = {}
    for key in keys:
        rates[key] = sum(getattr(row, key) if key != "strict_pass" else row.strict_pass for row in outcomes) / n
    return {
        "metric_version": METRIC_VERSION,
        "n_tasks": len(outcomes),
        "rates": rates,
        "rows": [asdict(row) | {"strict_pass": row.strict_pass} for row in outcomes],
    }

Joint tables are less pretty than a single percentage, which is exactly why they resist conversion into a slogan. A lab that needs one headline figure can still compute a strict score that requires compile, tests, and lint to succeed together. That conjunction is harsher than an average of partial credits, and the harshness is a feature when the claim is production readiness. Partial credit belongs in an appendix for debugging, not in the sentence that claims one agent beat another.

The third lock is a control battery that must fail in expected ways before any candidate agent is ranked. An empty patch, a shuffled-test oracle, and a template that copies the prompt into a file are cheap negative controls. If the empty patch scores above zero, the hidden tests are not hidden, or the grader awards points for an untouched fixture. If shuffled tests still pass, the suite is asserting noise, and the candidate ranking should be withheld until the suite is repaired.

# Proposed, unexecuted example: blanks.py
from __future__ import annotations

import random
from pathlib import Path

from metrics import TaskOutcome, summarize

def empty_patch(_task_dir: Path) -> bytes:
    return b""

def echo_prompt(task_dir: Path) -> bytes:
    prompt = (task_dir / "prompt.md").read_bytes()
    return b"# echoed prompt\n" + prompt

def shuffle_hidden_tests(task_dir: Path, seed: int = 7) -> None:
    tests = (task_dir / "hidden_tests.py").read_text(encoding="utf-8").splitlines()
    rng = random.Random(seed)
    rng.shuffle(tests)
    (task_dir / "hidden_tests.shuffled.py").write_text("\n".join(tests) + "\n", encoding="utf-8")

def blanks_must_fail(outcomes: dict[str, list[TaskOutcome]]) -> None:
    empty_rate = summarize(outcomes["empty_patch"])["rates"]["strict_pass"]
    shuffled_rate = summarize(outcomes["shuffled_tests"])["rates"]["strict_pass"]
    echo_rate = summarize(outcomes["echo_prompt"])["rates"]["strict_pass"]
    if empty_rate != 0.0 or shuffled_rate != 0.0:
        raise SystemExit("publish_gate: blanks scored above zero; withhold the leaderboard")
    if echo_rate > 0.0:
        raise SystemExit("publish_gate: prompt echo passed hidden tests; withhold the leaderboard")

Control failures are not an inconvenience to hide in a footnote after the marketing round has already shipped. They are the reason a biology lab runs blanks, and software evaluation deserves the same unromantic habit. A ranking without blanks is a tasting menu described as a randomized trial, which confuses appetite with evidence. The report schema below forces the blanks to travel with the candidate scores so a reader can reject the comparison.

# Proposed, unexecuted example: report.py
from __future__ import annotations

import json
from pathlib import Path

def build_report(pack_id: str, metric_version: str, blanks: dict, candidate: dict) -> dict:
    return {
        "pack_id": pack_id,
        "metric_version": metric_version,
        "controls": {
            "empty_patch_strict": blanks["empty_patch"]["rates"]["strict_pass"],
            "shuffled_tests_strict": blanks["shuffled_tests"]["rates"]["strict_pass"],
            "echo_prompt_strict": blanks["echo_prompt"]["rates"]["strict_pass"],
        },
        "candidate": {
            "model_id": candidate["model_id"],
            "strict_rate": candidate["rates"]["strict_pass"],
            "compile_rate": candidate["rates"]["compiled"],
            "test_rate": candidate["rates"]["tests_passed"],
            "lint_rate": candidate["rates"]["lint_clean"],
            "timeout_rate": candidate["rates"]["timed_out"],
        },
        "publishable": (
            blanks["empty_patch"]["rates"]["strict_pass"] == 0.0
            and blanks["shuffled_tests"]["rates"]["strict_pass"] == 0.0
            and candidate.get("n_tasks", 0) > 0
        ),
    }

def write_or_refuse(report: dict, path: Path) -> None:
    path.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
    if not report["publishable"]:
        raise SystemExit("publish_gate: report written, leaderboard printing is forbidden")
    print(report["candidate"]["strict_rate"])

The publish gate can be read as a short contract: every row must hold, or the leaderboard row is simply omitted. Omitting a row is not a failed communications strategy; it is the correct output of an experiment that did not qualify. Marketing copy that cannot survive this omission was never a measurement, and the protocol should not be bent to rescue it.

Gate	Required value before any leaderboard cell is printed
pack_id	SHA-256 of the committed file list, not a folder nickname
metric_version	Exact scorer identity, here strict_conjunction_v1
empty-patch strict rate	0.0
shuffled-test strict rate	0.0
candidate headline	Conjunction of compile, tests, and lint, never an average of leftovers

Running the same frozen pack more than once is where cost and access start to distort the protocol itself. Teams that can afford only a single paid vendor run often freeze the vendor instead of freezing the pack, which inverts the experiment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that currently offers free model access and a free server option for teams that need extra runs. Those two availability claims matter here only as a way to repeat the frozen pack against another endpoint without rewriting the metric contract. The harness stores an opaque model identifier, so the same digest, scorer, and blanks can execute on a free remote worker. A lab can keep the pack and the grader on its own disk and treat the remote worker as an untrusted code author.

The free server option is useful when local machines cannot isolate builds, not as proof that a particular model is stronger. Strength is a property of the report after controls pass, never a property of the brochure that announces free capacity. Anyone repeating the method should log the pack digest, the metric version, and the control outcomes beside the candidate means. The commands below sketch a local workflow that writes the manifest, runs blanks, and refuses a leaderboard when blanks fail.

# Proposed, unexecuted local workflow. Replace MODEL_ID with an opaque endpoint name.
python3 pin_pack.py --root ./pack --out manifest.json
PACK_ID=$(python3 -c 'import json; print(json.load(open("manifest.json"))["pack_id"])')
printf 'frozen pack_id=%s\n' "$PACK_ID"

python3 run_blanks.py --manifest manifest.json --out blanks.json
python3 -c 'import json,sys; b=json.load(open("blanks.json"));
assert b["empty_patch"]["rates"]["strict_pass"]==0.0
assert b["shuffled_tests"]["rates"]["strict_pass"]==0.0'

MODEL_ID="opaque-endpoint"  # keep the pack and grader local; treat the worker as untrusted
python3 run_candidate.py --manifest manifest.json --model "$MODEL_ID" --out candidate.json
python3 report.py --manifest manifest.json --blanks blanks.json --candidate candidate.json --out report.json

Printing a leaderboard after a failed blank is the software equivalent of toasting a thermometer that never left the box. The exit code in the sketch is the publish gate: a non-zero status means the number is not allowed to leave the lab notebook. Editors and internal wikis should treat that gate as a review checklist rather than as optional polish after design review.

This protocol does not measure taste, architecture, or long-horizon refactors, and it will punish agents that write correct code in an unexpected layout. Hidden unit tests are a poor oracle for user-interface work, data migration, and incident response, where the fixture cannot encode the real loss. Teams that lack a sandbox should not execute model-authored patches on a shared workstation, even when the surrounding service is free. Security reviewers, regulated domains, and classrooms that grade process rather than patches should pick a different evaluation, because this method scores artifacts only.

Vendors that need a single uplifting percentage for a launch should not use this method, because the joint table will refuse to flatten. Researchers who already possess a public, versioned benchmark with hidden tests and documented controls already have the lock this article reconstructs. The remaining audience is a small engineering group that wants to compare coding agents on private jobs without producing an advertisement.

A number earns the right to travel when the pack hash, the metric code, and the blank scores can travel with it. Without those companions, the percentage is a caption, and captions belong under screenshots rather than under methods. Groups that want another worker for extra blank runs can try MonkeyCode's free model access and free server option after the pack is pinned.