DEV Community: Avery Wang

Sign the Metric Before You Publish an Agent Score

Avery Wang — Sun, 20 Sep 2026 11:50:55 +0000

A headline agent score is not a measurement until the metric function is frozen, hashed, and tested against golden fixtures. Teams still paste a single percentage into a README and treat that figure as evidence. A one-line change in the scorer can move the number more than any model swap. This article treats the metric as an instrument that must be signed before any ranking is quoted.

Measuring agents without a signed metric is like weighing samples on a scale whose firmware updates overnight. The reading looks precise and the units look familiar, yet the instrument is a different object every morning. Freezing task files helps, but it does not bind the function that turns traces into points. When that function drifts, yesterday’s ranking and today’s ranking are not comparable, even with identical task bytes.

A usable dataset for this protocol is a directory of task cards with stable identifiers and content hashes, not a living spreadsheet. Each card should name inputs, oracles, and a timeout that is recorded rather than silently extended after a slow run. The split between probe items and evaluation items belongs in the same digest, because a moved row is a new experiment. The commands below hash that tree so a published number can point at one object and no other.

# content-address the evaluation pack; do not edit files after this digest is quoted
find eval_pack -type f -print0 | sort -z | xargs -0 sha256sum > eval_pack.sums
sha256sum eval_pack.sums > eval_pack.tree.sha256
cat eval_pack.tree.sha256

Metrics need a contract: a versioned function, a fixture suite, and an interval estimate instead of a naked mean. The contract states which fields of a trace count and how missing patches, crashed shells, or truncated logs are scored. Golden fixtures are tiny traces with known scores, and they fail the build if a refactor of the scorer changes them. Bootstrap intervals then show whether a claimed gap is larger than sampling noise on the same frozen pack.

# metric_contract.py — labeled sketch for a frozen scorer, not a vendor result
from __future__ import annotations

import hashlib, json, random
from pathlib import Path
from typing import Callable

CONTRACT_ID = "agent-metric/v3-patch-apply"
PASS, FAIL, INVALID = 1.0, 0.0, float("nan")

def digest_source(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def score_trace(trace: dict) -> float:
    if trace.get("schema") != "trace.v1":
        return INVALID
    if trace.get("exit_code") not in (0, 1):
        return INVALID
    if not trace.get("patch"):
        return FAIL
    oracle = set(trace.get("oracle_files") or [])
    touched = set(trace.get("touched_files") or [])
    tests = trace.get("tests_passed")
    if oracle and not oracle.issubset(touched):
        return FAIL
    return PASS if tests is True else FAIL

FIXTURES = [
    ({"schema": "trace.v1", "exit_code": 0, "patch": "diff", "oracle_files": ["a.py"],
      "touched_files": ["a.py"], "tests_passed": True}, PASS),
    ({"schema": "trace.v1", "exit_code": 0, "patch": "", "oracle_files": ["a.py"],
      "touched_files": [], "tests_passed": False}, FAIL),
    ({"schema": "trace.v2", "exit_code": 0, "patch": "diff", "oracle_files": [],
      "touched_files": [], "tests_passed": True}, INVALID),
]

def assert_fixtures(fn: Callable[[dict], float]) -> None:
    for i, (trace, expected) in enumerate(FIXTURES):
        got = fn(trace)
        if got != expected and not (got != got and expected != expected):
            raise AssertionError(f"fixture {i}: expected {expected!r}, got {got!r}")

def valid_scores(rows: list[float]) -> list[float]:
    return [x for x in rows if x == x]

def bootstrap_mean_ci(rows: list[float], n: int = 2000, alpha: float = 0.05, seed: int = 7) -> dict:
    rng = random.Random(seed)
    xs = valid_scores(rows)
    if len(xs) < 8:
        return {"n_valid": len(xs), "mean": None, "ci95": None, "note": "too few valid runs"}
    means = []
    for _ in range(n):
        sample = [xs[rng.randrange(len(xs))] for _ in xs]
        means.append(sum(sample) / len(sample))
    means.sort()
    lo = means[int(alpha / 2 * n)]
    hi = means[int((1 - alpha / 2) * n)]
    return {
        "n_valid": len(xs),
        "n_invalid": len(rows) - len(xs),
        "mean": sum(xs) / len(xs),
        "ci95": [lo, hi],
        "seed": seed,
        "resamples": n,
    }

if __name__ == "__main__":
    assert_fixtures(score_trace)
    print(json.dumps({
        "contract_id": CONTRACT_ID,
        "metric_sha256": digest_source(Path(__file__)),
        "fixture_count": len(FIXTURES),
    }, indent=2))

Controls keep the instrument honest without turning the write-up into a brochure. A stub agent that returns empty patches should sit near the floor if the metric actually penalizes missing work. A shuffled-oracle pass should collapse toward chance if filenames or ordering leak the answer into the scorer. Seed locks and environment fingerprints sit beside the score so a later reader can see whether the runtime or the trace schema changed. None of these controls prove a product is better; they only show that the number was produced under a named procedure.

# controls.py — labeled sketch: stub agent and shuffled oracles as scorer sanity checks
import json, os, platform, random, subprocess, sys
from pathlib import Path
from metric_contract import score_trace, bootstrap_mean_ci, CONTRACT_ID, digest_source

def fingerprint() -> dict:
    py = subprocess.check_output([sys.executable, "-c", "import sys; print(sys.version)"], text=True).strip()
    return {
        "python": py,
        "platform": platform.platform(),
        "cwd": str(Path.cwd()),
        "metric_sha256": digest_source(Path("metric_contract.py")),
        "tree_sha256": Path("eval_pack.tree.sha256").read_text().split()[0] if Path("eval_pack.tree.sha256").exists() else None,
    }

def stub_agent(task: dict) -> dict:
    return {
        "schema": "trace.v1",
        "exit_code": 1,
        "patch": "",
        "oracle_files": task.get("oracle_files", []),
        "touched_files": [],
        "tests_passed": False,
        "agent": "stub.empty_patch",
    }

def shuffle_oracles(tasks: list[dict], seed: int = 13) -> list[dict]:
    rng = random.Random(seed)
    files = [t.get("oracle_files", []) for t in tasks]
    rng.shuffle(files)
    out = []
    for task, oracle in zip(tasks, files):
        item = dict(task)
        item["oracle_files"] = oracle
        out.append(item)
    return out

def run_pack(tasks: list[dict], agent) -> list[float]:
    return [score_trace(agent(t)) for t in tasks]

if __name__ == "__main__":
    tasks = json.loads(Path("eval_pack/tasks.json").read_text())
    stub_rows = run_pack(tasks, stub_agent)
    shuffled = shuffle_oracles(tasks)
    report = {
        "contract_id": CONTRACT_ID,
        "env": fingerprint(),
        "stub_empty_patch": bootstrap_mean_ci(stub_rows),
        "shuffled_oracles_on_stub": bootstrap_mean_ci(run_pack(shuffled, stub_agent)),
        "seed_lock": {"pythonhashseed": os.environ.get("PYTHONHASHSEED")},
    }
    Path("score_report.json").write_text(json.dumps(report, indent=2, default=str))
    print(json.dumps(report, indent=2, default=str))

export PYTHONHASHSEED=0
python metric_contract.py
python controls.py
# quote score_report.json as a whole; do not lift only the mean field into a title

The report is the artifact, and the mean is only one field inside it. A serious write-up quotes the contract identifier, the metric digest, the tree digest, the valid-run count, the invalid-run count, and the interval. If two agents overlap inside that interval on the same pack, the honest sentence is that the pack cannot separate them yet. Stretching a 0.04 gap into a crown graphic is how a measurement turns into marketing without any change in the underlying traces.

Reproducing that report is cheaper when the harness can run in a disposable workspace that already exposes models and a server, rather than on a laptop that silently differs from last week. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option can host the same frozen pack and the same signed scorer so a reviewer reruns metric_contract.py and controls.py without rebuilding the instrument by hand. The product is not the metric, and a convenience host does not enlarge a confidence interval or repair a leaking oracle.

Limitations follow from the statistics, not from taste. Bootstrap intervals treat tasks as roughly exchangeable, while agent packs are often clustered by repository, language, or bug class, which narrows the real uncertainty. Golden fixtures catch scorer regressions, but they cannot certify that partial credit matches a human maintainer’s judgment on messy diffs. Invalid runs must stay visible; dropping crashes until the mean looks tidy is another quiet rescoring. Wall-clock, token use, and retry counts belong in the report as context, yet they are not a substitute for the signed success function.

Teams that need regulatory evidence, customer acceptance, or safety certification should not treat this sketch as a qualification method. Groups that cannot freeze the pack, or that let prompt text and hidden tools change between runs, will hash a moving target and then argue about ghosts. Writers who only want a single percentage for a launch post will find the extra fields inconvenient, which is the point of putting them in the file. A ranking that cannot name its instrument is not a ranking that later engineers can refute.

The core conclusion stays small on purpose. Publish the contract, the digests, the fixtures, and the interval, then let the mean sit in the middle of that paragraph. Readers who rerun the stub and the shuffle already know whether the scorer still behaves like yesterday’s instrument. That habit is slower than a leaderboard screenshot, and it is the difference between a number that can be cited and a number that can only be advertised. If the signed report is reproduced in a shared workspace, the useful next step is to keep the pack and the scorer in lockstep rather than to chase a prettier headline.

Calibrate the Judge Before the Agent Score

Avery Wang — Sat, 19 Sep 2026 09:57:23 +0000

An agent percentage is not a measurement until the scoring harness itself can be replayed against frozen golden traces. Leaderboards still publish a single success rate as if the grader, the timeouts, and the parse rules were as stable as the task files. Those hidden pieces move whenever a prompt template, a JSON schema, or a retry policy changes underneath the same dataset name. This article treats harness calibration as the first published result, and the candidate agent as a later comparison that inherits that protocol.

Public conversation keeps returning to a familiar worry that evaluation suites age faster than the systems they claim to rank. That worry is reasonable, because a test that once discriminated among tools can become a puzzle that many large models memorize. Expanding the suite without locking the grader simply multiplies an unstable instrument across more task files. A more honest step is to freeze a small golden-trace pack and refuse ranking until the harness scores those traces identically twice.

A golden trace is not a unit test for the agent; it is a unit test for the judge that assigns credit. Each trace records the task identifier, the exact stdout, the exit status, and the wall time from the original run. The protocol card then states how those fields map onto pass, fail, skip, and parse_error, including the timeout ceiling. If two independent checkouts of the harness disagree on a golden trace, the candidate ranking is invalid rather than merely late.

The dataset side of the card is a digest, not a friendly folder name on someone's laptop. Task files, fixtures, and the golden traces themselves are hashed together so a renamed repository cannot quietly swap a prompt. The metric contract sits beside that digest and names the aggregation rule, because micro-averages and family macros are not interchangeable summaries. Controls in this design are golden replay, an order shuffle, and an idempotent rescore, which catch graders that depend on global state.

Protocol card

{
  "protocol_id": "harness.v3.golden",
  "dataset_glob": "tasks/*.json",
  "dataset_sha256": "replace-after-hashing",
  "golden_glob": "golden/*.json",
  "metrics": {
    "pass_rule": "exit_code==0 and schema_ok and wall_ms<=timeout_ms",
    "timeout_ms": 15000,
    "aggregate": "macro_by_task_family"
  },
  "controls": ["golden_replay", "idempotent_rescore", "order_shuffle"]
}

That object is intentionally boring, because a timeout and a hash travel better than a rounded marketing percentage. The card also refuses to mention a candidate agent, since the harness must stand on its own before any comparison is fair. Teams that skip this file usually discover the omission later, when a one-line JSON parsing change moves every historical score. A leaderboard percentage without a calibrated judge resembles a stopwatch that was never zeroed at the starting line.

Replay kit

The following Python module implements the three controls against a directory of traces and prints a calibration report. It is labeled as a local replay proposal, not as evidence from a production contest or a vendor bake-off. Readers should replace the digest placeholders after the first real run, rather than copying a fictional hash into a blog table.

# harness_calibrate.py
from __future__ import annotations

import hashlib
import json
import random
from pathlib import Path
from typing import Any, Callable, Dict, List, Tuple

Trace = Dict[str, Any]
Score = Dict[str, Any]


def sha256_tree(root: Path, pattern: str) -> str:
    digest = hashlib.sha256()
    paths = sorted(root.glob(pattern))
    if not paths:
        raise FileNotFoundError(f"no files for {pattern}")
    for path in paths:
        digest.update(path.name.encode("utf-8"))
        digest.update(path.read_bytes())
    return digest.hexdigest()


def load_json_dir(root: Path, pattern: str) -> List[Trace]:
    items: List[Trace] = []
    for path in sorted(root.glob(pattern)):
        payload = json.loads(path.read_text())
        payload["_source"] = path.name
        items.append(payload)
    return items


def score_trace(trace: Trace, timeout_ms: int) -> Score:
    schema_ok = isinstance(trace.get("stdout"), str)
    if schema_ok and trace.get("expect_json"):
        try:
            json.loads(trace["stdout"])
        except json.JSONDecodeError:
            schema_ok = False
    elif trace.get("expect_json"):
        schema_ok = False
    wall = int(trace.get("wall_ms", timeout_ms + 1))
    passed = (
        int(trace.get("exit_code", 1)) == 0
        and schema_ok
        and wall <= timeout_ms
    )
    return {
        "id": trace.get("id", trace.get("_source")),
        "family": trace.get("family", "ungrouped"),
        "pass": passed,
        "parse_error": not schema_ok,
        "timeout": wall > timeout_ms,
    }


def macro_by_family(scores: List[Score]) -> Dict[str, float]:
    buckets: Dict[str, List[int]] = {}
    for row in scores:
        buckets.setdefault(row["family"], []).append(int(row["pass"]))
    return {
        name: (sum(vals) / len(vals) if vals else 0.0)
        for name, vals in sorted(buckets.items())
    }


def idempotent_rescore(traces: List[Trace], scorer: Callable[[Trace], Score]) -> bool:
    first = [scorer(item) for item in traces]
    second = [scorer(item) for item in traces]
    return first == second


def order_shuffle_stable(
    traces: List[Trace],
    scorer: Callable[[Trace], Score],
    seeds: Tuple[int, ...] = (0, 1, 7, 13),
) -> bool:
    baseline = macro_by_family([scorer(item) for item in traces])
    for seed in seeds:
        shuffled = list(traces)
        random.Random(seed).shuffle(shuffled)
        again = macro_by_family([scorer(item) for item in shuffled])
        if again != baseline:
            return False
    return True


def calibrate(root: Path, timeout_ms: int = 15000) -> Dict[str, Any]:
    traces = load_json_dir(root, "golden/*.json")
    scorer = lambda item: score_trace(item, timeout_ms)
    scored = [scorer(item) for item in traces]
    expected = {item["id"]: bool(item["must_pass"]) for item in traces}
    mismatches = [row["id"] for row in scored if row["pass"] != expected.get(row["id"])]
    report = {
        "dataset_sha256": sha256_tree(root, "tasks/*.json"),
        "golden_sha256": sha256_tree(root, "golden/*.json"),
        "golden_n": len(traces),
        "mismatches": mismatches,
        "idempotent": idempotent_rescore(traces, scorer),
        "order_stable": order_shuffle_stable(traces, scorer),
        "macro_by_family": macro_by_family(scored),
    }
    report["ready_for_candidates"] = (
        not mismatches and report["idempotent"] and report["order_stable"]
    )
    return report


if __name__ == "__main__":
    print(json.dumps(calibrate(Path(".")), indent=2))

A matching golden file makes the contract visible to anyone who clones the repository and runs the module. The fixture below is synthetic on purpose, so it should not be cited as a result from a live agent evaluation. Known-good traces prove the happy path still parses; they do not prove that the candidate model understood the task.

{
  "id": "json.extract.001",
  "family": "extract",
  "exit_code": 0,
  "wall_ms": 820,
  "expect_json": true,
  "stdout": "{\"ok\": true, \"value\": 42}",
  "must_pass": true
}

A second fixture should fail on purpose, because a harness that only sees successes cannot detect a grader that always returns true. Known-bad traces act like a laboratory blank, proving the instrument can still reject malformed output. Without that blank, a broken parser that treats every blob as valid JSON will inflate later agent scores by one hidden constant. Publishing the blank is what separates a measurement from a brochure, even when the candidate percentage looks impressive.

{
  "id": "json.extract.002",
  "family": "extract",
  "exit_code": 0,
  "wall_ms": 640,
  "expect_json": true,
  "stdout": "almost json but not",
  "must_pass": false
}

Commands for a stranger replaying the kit stay small and should live in continuous integration beside the percentage job. The hash step belongs in that pipeline so a silent fixture edit cannot land next to an unchanged published score. A failing calibration job is the desired outcome when someone improves the parser without updating golden expectations.

python -m pip install pytest
python harness_calibrate.py
python - <<'PY'
from pathlib import Path
from harness_calibrate import calibrate
report = calibrate(Path("."))
assert report["ready_for_candidates"], report
print("harness calibrated:", report["dataset_sha256"][:12])
PY

A short pytest file turns those assertions into a gate that ranking jobs must pass before any candidate is scored. The test does not claim that any hosted model is strong; it claims that the local judge is boringly consistent. Timeout handling is part of the metric contract, so a slow but pretty answer is still a failure under this protocol.

# test_harness_calibrate.py
from pathlib import Path
from harness_calibrate import calibrate, score_trace


def test_golden_pack_locks_the_judge():
    report = calibrate(Path("."))
    assert report["mismatches"] == []
    assert report["idempotent"] is True
    assert report["order_stable"] is True
    assert report["ready_for_candidates"] is True


def test_timeout_is_part_of_the_metric():
    slow = {
        "id": "slow.001",
        "family": "extract",
        "exit_code": 0,
        "wall_ms": 20000,
        "stdout": "{}",
        "expect_json": True,
    }
    row = score_trace(slow, timeout_ms=15000)
    assert row["pass"] is False
    assert row["timeout"] is True

Only after that gate is green does a candidate agent deserve a percentage stored beside the protocol hashes. People reading a blog table rarely notice a missing digest, which is why marketing scores travel farther than protocol cards. A fair write-up lists the candidate score as a comparison that inherits the frozen harness, not as an isolated trophy. Family-level rates should remain visible, because a two-item family must not be summarized as if it contained a hundred tasks.

When a team wants a second machine to rerun the candidate arm after calibration, a free remote shell can be enough. Disclosure: This article was prepared as part of MonkeyCode's product outreach, and the product notes below are availability claims rather than benchmark results. MonkeyCode currently offers free model access and a free server option that can host the same replay commands after local calibration. Those notes are not a quota, a hardware spec, or a claim that any particular model won the synthetic tasks above.

What the numbers still cannot mean

The numbers stay non-marketing for a dull reason: every published figure names the digest it depends on. A reader who cannot obtain the golden traces cannot audit the judge, and a reader who cannot hash the tasks cannot detect silent edits. Golden traces still freeze yesterday's failures, so they will not detect a novel cheating strategy the original agents never tried. A purely syntactic JSON check will miss semantic wrongness, and an unpinned model judge will reintroduce the drift this protocol tried to remove.

The shuffle test only proves order independence of the scorer, not independence from the host clock or a polluted virtualenv. Teams that should skip this workflow include daily prompt shootouts and classrooms that want a live demo more than a ledger. Vendors that refuse to publish failing traces cannot use this kit without turning the blank into another marketing surface. Safety-critical evaluations still need human review on top of schema checks, because a well-formed object can still be a harmful action.

Stale tests will keep aging, and public models will keep saturating yesterday's puzzles without asking anyone's permission. Ranking can still be honest if the first published artifact is the harness report, with hashes, mismatches, and a ready_for_candidates bit. Candidate percentages then become a second paragraph instead of the only sentence a reader remembers from the table.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Agent Scores Without a Null Pack Are Marketing

Avery Wang — Fri, 18 Sep 2026 08:17:33 +0000

An agent percentage published without a versioned dataset, a locked metric function, and a passing null pack is marketing copy. Weekend evaluations often move because the judge prompt, the task mix, or the timeout changed, not because the agent improved. A measurement protocol therefore refuses to print a success rate until those three pieces are sealed and the negative controls behave. The small harness below treats that refusal as a hard gate rather than a polite style recommendation.

Public coding-agent numbers have started to resemble product landing pages, even when the underlying tasks are thin. Saturated unit tests and loosely written judges both inflate pass rates without showing whether the agent can miss on purpose. When a suite cannot produce a clean miss, the headline percentage is no longer a measurement; it is a selected screenshot. That failure mode matches a load test that never opens a connection to the real database.

A defensible agent benchmark is a sealed protocol, not a folder of prompts edited after the first chart appeared. The dataset hash, the metric implementation, the timeout budget, and the null pack travel together as one version. Changing any of those fields creates a new protocol identity, so older percentages cannot be compared as if they were the same exam. The useful analogy is pinning a compiler and a test runner before quoting a runtime score.

The null pack is the piece most leaderboards skip because it can kill a flattering number. It is a small set of tasks whose outcomes are known in advance: items that must pass, items that must fail, and items that must be refused rather than guessed. If a supposedly strong agent sails through poisoned files, or fails tasks a stub solver should finish, the run is invalid and no percentage is emitted. The pack is not extra credit; it is the instrument check that comes before the experiment.

The following protocol is a proposed, unexecuted example. It does not claim production results, vendor rankings, or model-specific scores. Teams can copy the files, seal them, and replace the solver command with whatever agent they already run locally.

# protocol.yaml — identity for one exam, not a leaderboard
schema: agent-bench-protocol/v1
protocol_id: coding-repair.nullpack.2026-09-18
dataset:
  path: ./tasks/repair_v3.jsonl
  sha256: "PENDING_SEAL"
metric:
  module: ./metric_exact_diff.py
  sha256: "PENDING_SEAL"
budget:
  timeout_sec: 90
  max_files_touched: 8
null_pack:
  path: ./tasks/null_pack_v1.jsonl
  sha256: "PENDING_SEAL"
  rules:
    must_pass: ["NP-STUB-01", "NP-STUB-02"]
    must_fail: ["NP-POISON-01", "NP-LEAK-02"]
    must_refuse: ["NP-SECRET-01"]
publish:
  min_n: 40
  require_ci: true
  forbidden_fields: ["single_percentage_only"]

Sealing replaces every PENDING_SEAL value with a digest and then refuses further edits without a new protocol_id. The command below is intentionally boring, because a benchmark that needs a clever seal is already drifting toward theater.

#!/usr/bin/env bash
# seal_protocol.sh — proposed example, run from the protocol directory
set -euo pipefail
sha256sum tasks/repair_v3.jsonl tasks/null_pack_v1.jsonl metric_exact_diff.py \
  | tee protocol.sums
python3 - <<'PY'
import hashlib, pathlib, re, sys
text = pathlib.Path("protocol.yaml").read_text()
sums = {}
for line in pathlib.Path("protocol.sums").read_text().splitlines():
    digest, name = line.split()
    sums[name] = digest
mapping = {
    "./tasks/repair_v3.jsonl": sums["tasks/repair_v3.jsonl"],
    "./metric_exact_diff.py": sums["metric_exact_diff.py"],
    "./tasks/null_pack_v1.jsonl": sums["tasks/null_pack_v1.jsonl"],
}
for path, digest in mapping.items():
    text = text.replace('sha256: "PENDING_SEAL"', f'sha256: "{digest}"', 1)
if "PENDING_SEAL" in text:
    sys.exit("seal incomplete")
pathlib.Path("protocol.sealed.yaml").write_text(text)
print("sealed protocol written")
PY

The metric module must be a function with its own digest, not a chat prompt that a reviewer rephrases between runs. Exact-diff scoring is deliberately unimpressive: it compares a normalized unified diff against an oracle patch and returns a boolean. Teams that want a language-model judge can still use one, but that judge becomes part of the sealed metric file, not a hidden sidebar comment. If the judge changes, the protocol identity changes, and yesterday's percentage is retired.

# metric_exact_diff.py — proposed example
from pathlib import Path

def normalize(diff: str) -> str:
    lines = []
    for line in diff.splitlines():
        if line.startswith(("index ", "diff --git", "---", "+++")):
            continue
        lines.append(line.rstrip())
    return "\n".join(lines)

def score(result_diff: str, oracle_path: str) -> dict:
    oracle = normalize(Path(oracle_path).read_text())
    got = normalize(result_diff)
    return {"pass": got == oracle, "metric": "exact_diff_v1"}

Null-pack records look like ordinary tasks, except each row carries an expected class instead of an open-ended success flag. must_pass rows are trivial repairs a stub can finish, which detects a broken runner. must_fail rows include poisoned instructions and leaked answers that a careful agent should reject. must_refuse rows mention secrets or out-of-scope files; a completion that writes those files fails the instrument check even if the rest of the suite looks green.

# run_protocol.py — proposed example; prints no headline until gates pass
import json, random, statistics, subprocess, sys, time, yaml
from pathlib import Path
from metric_exact_diff import score

def load_jsonl(path):
    rows = []
    with open(path) as handle:
        for line in handle:
            rows.append(json.loads(line))
    return rows

def run_solver(task, timeout):
    started = time.time()
    proc = subprocess.run(
        ["bash", "./solve.sh", task["id"]],
        capture_output=True, text=True, timeout=timeout,
    )
    return proc.stdout, time.time() - started, proc.returncode

def bootstrap_ci(flags, rounds=1000):
    means = []
    n = len(flags)
    for _ in range(rounds):
        sample = [flags[random.randrange(n)] for _ in range(n)]
        means.append(sum(sample) / n)
    means.sort()
    lo = means[int(0.025 * rounds)]
    hi = means[int(0.975 * rounds)]
    return lo, hi, statistics.mean(flags)

def main(protocol_path):
    proto = yaml.safe_load(Path(protocol_path).read_text())
    if "PENDING_SEAL" in Path(protocol_path).read_text():
        sys.exit("refuse: protocol is not sealed")
    null_rows = load_jsonl(proto["null_pack"]["path"])
    for row in null_rows:
        diff, _, _ = run_solver(row, proto["budget"]["timeout_sec"])
        passed = score(diff, row["oracle"])["pass"]
        expected = row["expected_class"]
        if expected == "must_pass" and not passed:
            sys.exit(f"null pack failed open: {row['id']}")
        if expected in {"must_fail", "must_refuse"} and passed:
            sys.exit(f"null pack failed closed: {row['id']}")
    tasks = load_jsonl(proto["dataset"]["path"])
    if len(tasks) < proto["publish"]["min_n"]:
        sys.exit("refuse: sample smaller than publish.min_n")
    flags = []
    for task in tasks:
        diff, elapsed, code = run_solver(task, proto["budget"]["timeout_sec"])
        flags.append(1.0 if score(diff, task["oracle"])["pass"] else 0.0)
    lo, hi, mean = bootstrap_ci(flags)
    report = {
        "protocol_id": proto["protocol_id"],
        "n": len(flags),
        "mean": round(mean, 4),
        "ci95": [round(lo, 4), round(hi, 4)],
        "null_pack": "pass",
    }
    Path("report.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report))

if __name__ == "__main__":
    main(sys.argv[1])

A team that wants a number for slides will dislike this runner, because the happy path is easy to block. That is the point. Marketing copy needs a single percentage with no denominator, no interval, and no story about the tasks that should have failed. A measurement needs the opposite: a protocol identity, a sample size, a confidence interval, and a null pack that actually bites. If the interval is wide, the honest publication is the interval, not a rounded headline.

Variance is not a footnote for later. Two agents can share a mean and still disagree on which files they break, which is why a protocol that only stores a percentage cannot be replayed. The report above keeps n and ci95 in the same object as protocol_id, so a later reader can see whether the exam was large enough to quote. Teams that need a ranking can still rank, but they rank sealed protocols against each other rather than mixing leftover prompts from different weeks.

Running the gate locally stays small on purpose. The solver is a shell entrypoint, the metric is a file, and the dataset is content-addressed. That shape fits a short-lived server as well as a laptop, which is where free model access can participate without becoming the subject of the exam. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host the sealed runner when a laptop is a poor place to keep timeouts honest; they do not replace the protocol, the null pack, or the interval. The measurement remains valid if those product pieces are removed.

Several limitations follow from the same design, and they are not cosmetic. Exact-diff metrics under-count valid alternate patches, so this harness is a conservative instrument, not a creativity contest. Bootstrap intervals assume the task rows are exchangeable; they are not a substitute for stratified sampling when the dataset is a pile of unrelated bugs. The null pack only detects the failures someone thought to encode, which means a clever leak outside those rows can still flatter a solver. Timeouts, network flakes, and non-deterministic tools can also fail the pack for reasons that have nothing to do with agent quality.

This approach is a poor fit for sales demos, for regulatory claims, and for any score that must be produced before the protocol can be sealed. It is also the wrong tool when the sample is smaller than the publish floor, when the oracle patches are themselves unreviewed, or when a team needs a qualitative read of design taste rather than a boolean metric. Those jobs want notebooks, design reviews, or hold-the-line tests of a different kind. They do not want a percentage that pretends to be an exam.

The core conclusion does not change after the code is copied. An agent score is a measurement only when the dataset, the metric, and the null pack share a version and the run can still fail closed. Numbers that cannot fail are advertisements with extra decimal places. Seal the protocol, watch the null pack, and publish the interval; the ranking, if it is still needed, can wait until those pieces exist.

Freeze the Manifest Before the Agent Leaderboard

Avery Wang — Thu, 17 Sep 2026 06:34:48 +0000

A published agent score is trustworthy only after the dataset, metric functions, and control runs are frozen. Percentages that still depend on a moving task pack, a fuzzy grader, or an unstated baseline behave like advertisements rather than measurements. Teams that skip those three locks can still learn from a trial, yet they should not ship a ranking as a result. The protocol below treats a coding-agent comparison as a small experiment with pinned inputs, not as a demo reel.

Current developer debate often treats a coding agent as already better than a typical engineer, which collapses many jobs into one percentage. That collapse is convenient for a launch post and almost useless for a lab that must reproduce the number next week. A frozen pack behaves more like a sealed envelope in an exam room than like a buffet that chefs may restock during judging. Once the envelope can be reopened, the score describes the latest prompt more than it describes the system under test.

The first lock is the dataset, and it must be hashed before any agent is allowed to see a task. A practical pack is a directory of small programming jobs with fixtures, hidden tests, and a declared time budget per item. Hidden tests stay off the prompt on purpose, because an agent that reads the grader is no longer solving the stated problem. The proposed harness below pins every file with SHA-256 so later edits cannot silently inflate a published mean.

# Proposed, unexecuted example: pin_pack.py
from __future__ import annotations

import argparse, hashlib, json, os
from pathlib import Path

SKIP_DIR = {".git", "__pycache__", ".venv", "node_modules"}

def digest_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def pin_pack(root: Path) -> dict:
    files = []
    for dirpath, dirnames, filenames in os.walk(root):
        dirnames[:] = [d for d in sorted(dirnames) if d not in SKIP_DIR]
        for name in sorted(filenames):
            path = Path(dirpath) / name
            rel = path.relative_to(root).as_posix()
            files.append({"path": rel, "sha256": digest_file(path), "bytes": path.stat().st_size})
    blob = json.dumps({"root": str(root.resolve()), "files": files}, separators=(",", ":")).encode()
    return {
        "pack_id": hashlib.sha256(blob).hexdigest(),
        "file_count": len(files),
        "files": files,
        "metric_version": "strict_conjunction_v1",
    }

def main() -> None:
    p = argparse.ArgumentParser(description="Freeze a coding-agent task pack")
    p.add_argument("--root", type=Path, required=True)
    p.add_argument("--out", type=Path, required=True)
    args = p.parse_args()
    manifest = pin_pack(args.root)
    args.out.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
    print(manifest["pack_id"])

if __name__ == "__main__":
    main()

The manifest is the scientific object, not the blog sentence that later quotes a rounded percentage from a dashboard. If a task is too easy, too flaky, or too English-heavy, the repair happens by issuing a new pack identifier, never by quiet edits. Quiet edits are how a leaderboard becomes a brochure, because the old number and the new number stop sharing a denominator. Reviewers should refuse a comparison that cites two means without proving both means were computed on the same manifest digest.

The second lock is the metric contract, which must be a function of artifacts rather than a vibe about code quality. Pass rate alone is a weak contract because it hides compile failures, timeouts, skipped tests, and patches that delete the assertion. An honest report records several binary events per task and then publishes the joint table, not a single blended hero number. The functions below are labeled as unexecuted examples, and a lab should replace the stub runner with its real sandbox.

# Proposed, unexecuted example: metrics.py
from __future__ import annotations

from dataclasses import asdict, dataclass

METRIC_VERSION = "strict_conjunction_v1"

@dataclass(frozen=True)
class TaskOutcome:
    task_id: str
    compiled: bool
    tests_passed: bool
    lint_clean: bool
    timed_out: bool
    deleted_assertions: bool

    @property
    def strict_pass(self) -> bool:
        return (
            self.compiled
            and self.tests_passed
            and self.lint_clean
            and not self.timed_out
            and not self.deleted_assertions
        )

def summarize(outcomes: list[TaskOutcome]) -> dict:
    n = len(outcomes) or 1
    keys = ("compiled", "tests_passed", "lint_clean", "timed_out", "deleted_assertions", "strict_pass")
    rates = {}
    for key in keys:
        rates[key] = sum(getattr(row, key) if key != "strict_pass" else row.strict_pass for row in outcomes) / n
    return {
        "metric_version": METRIC_VERSION,
        "n_tasks": len(outcomes),
        "rates": rates,
        "rows": [asdict(row) | {"strict_pass": row.strict_pass} for row in outcomes],
    }

Joint tables are less pretty than a single percentage, which is exactly why they resist conversion into a slogan. A lab that needs one headline figure can still compute a strict score that requires compile, tests, and lint to succeed together. That conjunction is harsher than an average of partial credits, and the harshness is a feature when the claim is production readiness. Partial credit belongs in an appendix for debugging, not in the sentence that claims one agent beat another.

The third lock is a control battery that must fail in expected ways before any candidate agent is ranked. An empty patch, a shuffled-test oracle, and a template that copies the prompt into a file are cheap negative controls. If the empty patch scores above zero, the hidden tests are not hidden, or the grader awards points for an untouched fixture. If shuffled tests still pass, the suite is asserting noise, and the candidate ranking should be withheld until the suite is repaired.

# Proposed, unexecuted example: blanks.py
from __future__ import annotations

import random
from pathlib import Path

from metrics import TaskOutcome, summarize

def empty_patch(_task_dir: Path) -> bytes:
    return b""

def echo_prompt(task_dir: Path) -> bytes:
    prompt = (task_dir / "prompt.md").read_bytes()
    return b"# echoed prompt\n" + prompt

def shuffle_hidden_tests(task_dir: Path, seed: int = 7) -> None:
    tests = (task_dir / "hidden_tests.py").read_text(encoding="utf-8").splitlines()
    rng = random.Random(seed)
    rng.shuffle(tests)
    (task_dir / "hidden_tests.shuffled.py").write_text("\n".join(tests) + "\n", encoding="utf-8")

def blanks_must_fail(outcomes: dict[str, list[TaskOutcome]]) -> None:
    empty_rate = summarize(outcomes["empty_patch"])["rates"]["strict_pass"]
    shuffled_rate = summarize(outcomes["shuffled_tests"])["rates"]["strict_pass"]
    echo_rate = summarize(outcomes["echo_prompt"])["rates"]["strict_pass"]
    if empty_rate != 0.0 or shuffled_rate != 0.0:
        raise SystemExit("publish_gate: blanks scored above zero; withhold the leaderboard")
    if echo_rate > 0.0:
        raise SystemExit("publish_gate: prompt echo passed hidden tests; withhold the leaderboard")

Control failures are not an inconvenience to hide in a footnote after the marketing round has already shipped. They are the reason a biology lab runs blanks, and software evaluation deserves the same unromantic habit. A ranking without blanks is a tasting menu described as a randomized trial, which confuses appetite with evidence. The report schema below forces the blanks to travel with the candidate scores so a reader can reject the comparison.

# Proposed, unexecuted example: report.py
from __future__ import annotations

import json
from pathlib import Path

def build_report(pack_id: str, metric_version: str, blanks: dict, candidate: dict) -> dict:
    return {
        "pack_id": pack_id,
        "metric_version": metric_version,
        "controls": {
            "empty_patch_strict": blanks["empty_patch"]["rates"]["strict_pass"],
            "shuffled_tests_strict": blanks["shuffled_tests"]["rates"]["strict_pass"],
            "echo_prompt_strict": blanks["echo_prompt"]["rates"]["strict_pass"],
        },
        "candidate": {
            "model_id": candidate["model_id"],
            "strict_rate": candidate["rates"]["strict_pass"],
            "compile_rate": candidate["rates"]["compiled"],
            "test_rate": candidate["rates"]["tests_passed"],
            "lint_rate": candidate["rates"]["lint_clean"],
            "timeout_rate": candidate["rates"]["timed_out"],
        },
        "publishable": (
            blanks["empty_patch"]["rates"]["strict_pass"] == 0.0
            and blanks["shuffled_tests"]["rates"]["strict_pass"] == 0.0
            and candidate.get("n_tasks", 0) > 0
        ),
    }

def write_or_refuse(report: dict, path: Path) -> None:
    path.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
    if not report["publishable"]:
        raise SystemExit("publish_gate: report written, leaderboard printing is forbidden")
    print(report["candidate"]["strict_rate"])

The publish gate can be read as a short contract: every row must hold, or the leaderboard row is simply omitted. Omitting a row is not a failed communications strategy; it is the correct output of an experiment that did not qualify. Marketing copy that cannot survive this omission was never a measurement, and the protocol should not be bent to rescue it.

Gate	Required value before any leaderboard cell is printed
`pack_id`	SHA-256 of the committed file list, not a folder nickname
`metric_version`	Exact scorer identity, here `strict_conjunction_v1`
empty-patch strict rate	`0.0`
shuffled-test strict rate	`0.0`
candidate headline	Conjunction of compile, tests, and lint, never an average of leftovers

Running the same frozen pack more than once is where cost and access start to distort the protocol itself. Teams that can afford only a single paid vendor run often freeze the vendor instead of freezing the pack, which inverts the experiment.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that currently offers free model access and a free server option for teams that need extra runs. Those two availability claims matter here only as a way to repeat the frozen pack against another endpoint without rewriting the metric contract. The harness stores an opaque model identifier, so the same digest, scorer, and blanks can execute on a free remote worker. A lab can keep the pack and the grader on its own disk and treat the remote worker as an untrusted code author.

The free server option is useful when local machines cannot isolate builds, not as proof that a particular model is stronger. Strength is a property of the report after controls pass, never a property of the brochure that announces free capacity. Anyone repeating the method should log the pack digest, the metric version, and the control outcomes beside the candidate means. The commands below sketch a local workflow that writes the manifest, runs blanks, and refuses a leaderboard when blanks fail.

# Proposed, unexecuted local workflow. Replace MODEL_ID with an opaque endpoint name.
python3 pin_pack.py --root ./pack --out manifest.json
PACK_ID=$(python3 -c 'import json; print(json.load(open("manifest.json"))["pack_id"])')
printf 'frozen pack_id=%s\n' "$PACK_ID"

python3 run_blanks.py --manifest manifest.json --out blanks.json
python3 -c 'import json,sys; b=json.load(open("blanks.json"));
assert b["empty_patch"]["rates"]["strict_pass"]==0.0
assert b["shuffled_tests"]["rates"]["strict_pass"]==0.0'

MODEL_ID="opaque-endpoint"  # keep the pack and grader local; treat the worker as untrusted
python3 run_candidate.py --manifest manifest.json --model "$MODEL_ID" --out candidate.json
python3 report.py --manifest manifest.json --blanks blanks.json --candidate candidate.json --out report.json

Printing a leaderboard after a failed blank is the software equivalent of toasting a thermometer that never left the box. The exit code in the sketch is the publish gate: a non-zero status means the number is not allowed to leave the lab notebook. Editors and internal wikis should treat that gate as a review checklist rather than as optional polish after design review.

This protocol does not measure taste, architecture, or long-horizon refactors, and it will punish agents that write correct code in an unexpected layout. Hidden unit tests are a poor oracle for user-interface work, data migration, and incident response, where the fixture cannot encode the real loss. Teams that lack a sandbox should not execute model-authored patches on a shared workstation, even when the surrounding service is free. Security reviewers, regulated domains, and classrooms that grade process rather than patches should pick a different evaluation, because this method scores artifacts only.

Vendors that need a single uplifting percentage for a launch should not use this method, because the joint table will refuse to flatten. Researchers who already possess a public, versioned benchmark with hidden tests and documented controls already have the lock this article reconstructs. The remaining audience is a small engineering group that wants to compare coding agents on private jobs without producing an advertisement.

A number earns the right to travel when the pack hash, the metric code, and the blank scores can travel with it. Without those companions, the percentage is a caption, and captions belong under screenshots rather than under methods. Groups that want another worker for extra blank runs can try MonkeyCode's free model access and free server option after the pack is pinned.

Stratify the Task Pack Before Averaging Agent Scores

Avery Wang — Wed, 16 Sep 2026 04:43:25 +0000

A single pass rate across mixed coding tasks remains a marketing artifact rather than an engineering measurement. Honest agent comparison begins with a frozen stratified task pack, a paired control run, and metrics that refuse to collapse unlike work. Teams that publish one percentage after mixing refactors, bug fixes, and greenfield scaffolds are averaging bicycles with freight trucks. The method below treats dataset structure as the first control, not a footnote after the model call.

Recent discussion about coding agents often jumps from a demo transcript to a headline number as if those two objects were the same kind of evidence. A demo is a path through one prompt, one repository, and one reviewer mood, while a benchmark is a locked population of tasks with named families and a refusal to average them too early. When the population is undefined, the percentage is a slogan wearing a decimal point. The workflow here is a replay harness, not a leaderboard costume.

A useful analogy is vehicle testing on a mixed road that never labels the surface. Highway mileage, city mileage, and mountain mileage can each be honest inside their own family, yet a blended figure hides which surface produced the win. Coding-agent tasks behave the same way once refactors, test repairs, documentation edits, and scaffold generation share one mean. The candidate that looks strongest on the blend may be weakest on the family the team actually ships.

The dataset is therefore a pack, not a folder of leftover tickets. Each item carries a stable task identifier, a family label, a frozen fixture hash, a timeout, and an oracle that does not move during the comparison window. Families stay coarse enough to fill, such as refactor, bugfix, tests, and scaffold, because twelve tiny families with two items each recreate the original averaging problem under a fancier name. Operators who cannot name the family before the first agent call are not ready to report a score.

Metrics follow the pack instead of leading it. Per-family pass rate, median time to a green oracle, timeout rate, and a paired delta against a frozen control agent are the columns that survive review. A global mean is allowed only after a homogeneity check shows the families are not arguing with each other. When they argue, the table stays split, and the narrative stays split, even if that makes the blog post less dramatic.

Controls sit beside the candidate rather than in a later appendix. The same pack hash, the same seed, the same wall-clock budget, and the same oracle binary run twice: once for the control configuration and once for the candidate. Order is randomized inside a family so the second run does not inherit a warm cache as if it were skill. Any number that cannot be regenerated from the pack hash and the run manifest is treated as commentary, not as a result.

The artifact below is a proposed local harness, labeled as an unexecuted example rather than a claimed production study. It loads a JSON pack, refuses mixed-family averages by default, hashes the pack, and prints family rows that a later agent runner can fill. Teams can swap the stub run_agent function for a real runner without changing the reporting contract.

# bench_pack.py — proposed replay harness, not a published study
from __future__ import annotations

import hashlib
import json
import random
import statistics
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Callable, Dict, List


@dataclass(frozen=True)
class Task:
    task_id: str
    family: str
    fixture_sha256: str
    timeout_s: float


@dataclass
class Trial:
    task_id: str
    family: str
    passed: bool
    elapsed_s: float
    timed_out: bool
    agent_label: str


def load_pack(path: Path) -> List[Task]:
    raw = json.loads(path.read_text())
    tasks = [Task(**row) for row in raw["tasks"]]
    families = {t.family for t in tasks}
    if len(families) < 2:
        raise ValueError("pack must contain at least two families")
    counts = {f: sum(1 for t in tasks if t.family == f) for f in families}
    if min(counts.values()) < 3:
        raise ValueError("each family needs at least three tasks")
    return tasks


def pack_hash(tasks: List[Task]) -> str:
    blob = json.dumps([t.__dict__ for t in tasks], sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()


def stub_agent(task: Task, label: str, rng: random.Random) -> Trial:
    # Deterministic stub so the harness is testable without a network.
    start = time.perf_counter()
    bias = 0.72 if label == "candidate" else 0.64
    family_shift = {"refactor": 0.08, "bugfix": 0.00, "tests": -0.04, "scaffold": -0.10}
    p = bias + family_shift.get(task.family, 0.0)
    elapsed = rng.uniform(0.4, min(task.timeout_s, 8.0))
    timed_out = elapsed >= task.timeout_s
    passed = (not timed_out) and (rng.random() < p)
    return Trial(task.task_id, task.family, passed, elapsed, timed_out, label)


def run_pack(
    tasks: List[Task],
    runner: Callable[[Task, str, random.Random], Trial],
    seed: int,
) -> List[Trial]:
    rng = random.Random(seed)
    labeled = [(t, lab) for t in tasks for lab in ("control", "candidate")]
    rng.shuffle(labeled)
    return [runner(task, lab, rng) for task, lab in labeled]


def family_report(trials: List[Trial]) -> Dict[str, dict]:
    out: Dict[str, dict] = {}
    families = sorted({t.family for t in trials})
    for family in families:
        rows = [t for t in trials if t.family == family]
        def slice_label(label: str) -> List[Trial]:
            return [t for t in rows if t.agent_label == label]
        control, cand = slice_label("control"), slice_label("candidate")
        def pass_rate(xs: List[Trial]) -> float:
            return sum(t.passed for t in xs) / len(xs)
        out[family] = {
            "n": len(control),
            "control_pass": round(pass_rate(control), 3),
            "candidate_pass": round(pass_rate(cand), 3),
            "delta_pass": round(pass_rate(cand) - pass_rate(control), 3),
            "candidate_median_s": round(statistics.median(t.elapsed_s for t in cand), 3),
            "candidate_timeout_rate": round(sum(t.timed_out for t in cand) / len(cand), 3),
        }
    return out


def maybe_global_mean(report: Dict[str, dict]) -> str:
    deltas = [row["delta_pass"] for row in report.values()]
    # Crude homogeneity gate: mixed signs mean the mean is a costume.
    if any(d < 0 for d in deltas) and any(d > 0 for d in deltas):
        return "REFUSE_GLOBAL_MEAN: families disagree on sign"
    return str(round(statistics.mean(deltas), 3))


if __name__ == "__main__":
    pack = Path("task_pack.json")
    tasks = load_pack(pack)
    digest = pack_hash(tasks)
    trials = run_pack(tasks, stub_agent, seed=20260916)
    report = family_report(trials)
    print(json.dumps({
        "pack_sha256": digest,
        "seed": 20260916,
        "families": report,
        "global_delta": maybe_global_mean(report),
    }, indent=2))

A companion pack file keeps the contract visible. Fixture hashes are placeholders in this example and must be replaced with hashes of real tarballs before any public claim. The important property is that the pack is data, not a prompt stuffed into a README.

{
  "tasks": [
    {"task_id": "rf-001", "family": "refactor", "fixture_sha256": "ab", "timeout_s": 30},
    {"task_id": "rf-002", "family": "refactor", "fixture_sha256": "cd", "timeout_s": 30},
    {"task_id": "rf-003", "family": "refactor", "fixture_sha256": "ef", "timeout_s": 30},
    {"task_id": "bf-001", "family": "bugfix", "fixture_sha256": "11", "timeout_s": 30},
    {"task_id": "bf-002", "family": "bugfix", "fixture_sha256": "22", "timeout_s": 30},
    {"task_id": "bf-003", "family": "bugfix", "fixture_sha256": "33", "timeout_s": 30},
    {"task_id": "ts-001", "family": "tests", "fixture_sha256": "44", "timeout_s": 30},
    {"task_id": "ts-002", "family": "tests", "fixture_sha256": "55", "timeout_s": 30},
    {"task_id": "ts-003", "family": "tests", "fixture_sha256": "66", "timeout_s": 30},
    {"task_id": "sc-001", "family": "scaffold", "fixture_sha256": "77", "timeout_s": 45},
    {"task_id": "sc-002", "family": "scaffold", "fixture_sha256": "88", "timeout_s": 45},
    {"task_id": "sc-003", "family": "scaffold", "fixture_sha256": "99", "timeout_s": 45}
  ]
}

Running the stub locally should produce a family table and, quite often, a refused global mean. That refusal is the point of the method: mixed signs across families are information, not a rounding problem to hide inside one percentage. Replace the stub with a real agent only after the pack hash is recorded in the lab notebook or CI log.

python3 -c "import json,pathlib; p=pathlib.Path('task_pack.json'); json.loads(p.read_text()); print('pack json ok')"
python3 bench_pack.py | tee run_manifest.json

A cheap place to execute the same harness matters because environment drift is a second averaging sin. If the control ran on a laptop and the candidate ran on a crowded CI runner, the delta is partly hardware theater. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is relevant here only as a shared execution lane: free model access and a free server option let the control and the candidate share one runtime while the pack hash stays fixed. No model names, quotas, or hardware claims are added beyond that availability, because those details change and do not belong inside a methodology paper.

Numbers from this harness are not marketing when three conditions hold together. The pack hash is published beside the table, families remain visible, and the global mean is omitted whenever family deltas disagree in sign. A fourth informal check helps reviewers: if removing any one family reverses the winner, the headline was a blend, not a finding. That check is cheap, and it catches most accidental leaderboards before they leave the lab.

Limitations are part of the protocol rather than a closing apology. Twelve stub tasks cannot support a ranking of products, and the stub probabilities are instructional fakes that must never be quoted as measurements. The homogeneity gate is crude and can miss magnitude fights where every family moves the same direction but for different reasons. Timeout policy, flake retries, and human review of oracle mismatches still sit outside this script and will dominate real repositories.

This approach is a poor fit for live incident response, for single-ticket debugging, and for any team that needs a ship-or-not decision before a pack can be frozen. It is also the wrong tool when the product under test is allowed to mutate the fixtures, because a self-editing dataset is a moving target wearing a lab coat. Vendors who require a single public percentage as a procurement ritual will dislike the refused mean, which is evidence the method is working.

Readers who already keep agent evaluation notes can drop the harness beside those notes and require a pack hash on every new candidate row. The useful output is a family table that survives contact with a skeptical reviewer, not a slogan that could have been written before the run. Anyone who wants to try the same reporting contract on a shared runtime can start from the script, freeze a real fixture pack, and keep the mean split until the families agree.

Add a Cost Axis Before You Trust an Agent Score

Avery Wang — Mon, 14 Sep 2026 21:55:49 +0000

A pass rate without a cost axis is an advertisement, not a measurement, and that distinction decides whether a benchmark survives review. Teams comparing coding agents publish one percentage, then discover later that the cheapest configuration reached the same number for a fraction of the spend. The repair is rarely a stronger model; it is a harness that records what each solved task actually consumed.

Scores compress a messy run into one digestible number, which is precisely why they mislead so easily. Two configurations can converge on identical pass rates while diverging by an order of magnitude in input tokens, retries, and wall-clock time. A reader who sees only the percentage cannot tell whether the result reflects capability or budget, and neither can the team that published it.

Freeze the dataset before the first run

Every task should exist as a frozen file with a stable identifier, and the manifest should carry a hash of the prompt bytes so a silent edit cannot slip into a comparison. A holdout split, untouched during tuning, is what separates a measured result from a fitted one. Without that split, every reported gain contains an unknown amount of optimization against the test set.

{"id":"t-014","prompt_sha":"9f2c1ab7d004","check":"python grader.py --case 14","expect":"PASS","timeout":180}

The manifest above is deliberately boring, because boring manifests are the ones that still reproduce six weeks later. A reviewer should be able to regenerate the exact task set from version control without asking anyone for a private dataset. If that is impossible, the comparison is anecdotal and the article describing it is marketing.

Metrics that survive a skeptical reader

Three numbers belong in every published table: solved rate on the holdout split, tokens per solved task, and median wall-clock seconds per task. The middle one is the column most teams omit, and it is the one that changes conclusions. Tokens per call rewards agents that give up early, while tokens per solved task penalizes them, which is why the denominator deserves an explicit sentence.

import json, statistics

rows = [json.loads(line) for line in open("runs/arm_a.jsonl")]
solved = [r for r in rows if r["solved"]]

report = {
    "tasks": len(rows),
    "solved_rate": round(len(solved) / len(rows), 3),
    "tokens_per_solved": round(
        sum(r["tokens_in"] + r["tokens_out"] for r in solved) / max(len(solved), 1)
    ),
    "wall_median_s": round(statistics.median(r["wall_s"] for r in rows), 1),
}
print(json.dumps(report, indent=2))

None of these metrics requires a novel statistical method, and that restraint is the point. A benchmark becomes credible when a stranger can recompute it from raw rows rather than trusting a dashboard screenshot. Publish the JSONL file behind the table, or accept that the table is unverifiable.

Controls that make two arms comparable

The harness below is untested scaffolding rather than a finished tool, so treat it as a template and adapt the usage parser to whichever provider is being measured. Its only ambition is to attach a hash and a timestamp to every row, so that later disagreement becomes a diff instead of an argument. Two arms, one task set, one grader version.

# bench.py -- untested scaffolding; adapt before trusting any printed number.
import hashlib, json, subprocess, time
from pathlib import Path

def sha(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:12]

def grade(task: dict, cwd: Path) -> bool:
    proc = subprocess.run(task["check"], cwd=cwd, shell=True,
                          capture_output=True, text=True, timeout=task.get("timeout", 180))
    return proc.returncode == 0 and task["expect"] in proc.stdout

def run_task(task: dict, arm: dict, workdir: Path) -> dict:
    started = time.monotonic()
    agent = subprocess.run(arm["command"], cwd=workdir, shell=True,
                           capture_output=True, text=True, timeout=arm.get("timeout", 600))
    wall = time.monotonic() - started
    usage = parse_usage(agent.stdout)  # provider-specific: tokens_in, tokens_out, retries
    return {
        "task_id": task["id"], "arm": arm["name"], "seed": arm["seed"],
        "prompt_sha": sha(Path("tasks") / f"{task['id']}.md"),
        "grader_sha": sha(Path("grader.py")),
        "solved": grade(task, workdir), "wall_s": round(wall, 2), **usage,
    }

Holding prompt bytes, task order, grader version, and host constant matters more than holding model temperature, because the first four are invisible failure modes. A host change alone can move median wall-clock time enough to reverse a ranking, and nobody notices when the machine swap is unrecorded. The grader_sha field costs one line and prevents an entire class of accidental cheating, including well-intentioned grader edits.

A two-arm protocol you can repeat

A minimal protocol runs the same task list through a cheap arm and a strong arm, twice each, and compares only holdout rows. Two consecutive runs of the same arm expose variance that a single run hides, and variance larger than the gap between arms means no ranking should be published at all.

python bench.py --arm cheap  --tasks tasks/ --out runs/cheap-1.jsonl --seed 7
python bench.py --arm cheap  --tasks tasks/ --out runs/cheap-2.jsonl --seed 7
python bench.py --arm strong --tasks tasks/ --out runs/strong-1.jsonl --seed 7
python bench.py --arm strong --tasks tasks/ --out runs/strong-2.jsonl --seed 7
python summarize.py runs/*.jsonl --holdout-only

Claim someone wants to make	Column that must be published	Control that must hold constant
"faster on our tasks"	median wall seconds	host, task order, timeouts
"cheaper per result"	tokens per solved task	grader version, retry policy
"better at the job"	solved rate on holdout	prompt bytes, task manifest

A table like that turns a vague boast into a falsifiable sentence, and falsifiable sentences are the only ones worth shipping to a review channel. When a claim cannot fill all three cells, it belongs in a changelog rather than a benchmark post.

Where free access changes the arithmetic

This protocol is budget-hungry for an unglamorous reason: it wants repeated runs across at least two arms, and per-run spend is the usual justification for deleting the control arm entirely. The MonkeyCode project is presented by its operator as an open-source coding agent offering free model access and a free server option, which removes both excuses at once for someone who just wants to validate the method. A free server option also eliminates the laptop-versus-CI host difference that quietly corrupts wall-clock comparisons, provided the run records where it executed.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability terms, allowances, and supported models change, so any reader planning a long experiment should confirm the current terms on the project's own documentation before designing a multi-week run around them. Nothing in the harness above depends on a specific provider, which is deliberate: the method should outlive whichever free tier made it affordable this month.

Limits, and who should skip this

A cost axis systematically favours cheap configurations, so a team whose real constraint is latency or correctness on a narrow, high-stakes task may find the metric actively unhelpful. Token counts also ignore human review time, which is often the largest line item and the hardest to attribute per task. Graded synthetic tasks remain proxies, and a high solved rate on twelve frozen prompts says very little about a two-million-line repository.

Teams operating under regulated audit, certified environments, or contractual throughput guarantees should not substitute this harness for their formal validation process. The method measures relative behaviour between two configurations on one task set, and it offers no absolute claim about production reliability. Anyone needing a signed attestation should look elsewhere, because a JSONL file is not an attestation.

The practical takeaway is narrow and durable: publish the cost column, freeze the dataset, pin the grader, and run each arm twice before ranking anything. A benchmark that survives those four habits will still be quoted a year later, while a lone percentage quietly expires. If the harness above helps someone replace a suspicious number with a reproducible one, it has done its job.

Holdout Ledgers Keep Agent Scores Honest

Avery Wang — Sun, 13 Sep 2026 20:46:27 +0000

A headline agent percentage is not a measurement until the task ledger, metrics, and controls can be re-run by a stranger. Teams that skip that freeze often publish marketing numbers dressed as evaluation, then discover they cannot reproduce the run. This note records a compact methodology that treats the dataset as a versioned artifact rather than a mood. The scripts below are labeled examples for a local harness, not results claimed from a private benchmark farm.

The temptation is familiar because leaderboards reward a single number more than they reward a durable paper trail. A coding agent that appears to solve most tasks may have seen near-duplicates, burned unbounded retries, or shared context with its own grader. An honest ranking therefore starts with provenance written before the first model call, not with a product name. The sections that follow walk through dataset construction, a metric vector, and isolation controls that keep those figures from sliding into advertisement.

Treat the Dataset as a Sealed Evidence Bag

A usable coding dataset is a sequence of immutable task records with hashes, licenses, and difficulty labels assigned before any agent runs. Each record should name the source repository, the commit that defined the failing test, and the oracle that will later judge a patch. If that metadata is written after scores exist, the ledger is already contaminated by the ranking it pretends to justify. The analogy is a sealed evidence bag in a hallway, because once the tape is broken the later story no longer holds.

Holdout is not a folder named test created on the same afternoon as the leaderboard announcement. Tasks that share a parent commit, a near-identical failing assertion, or a copied docstring should be grouped so leakage stays visible. Difficulty labels belong on the ledger before agents run, because labeling after seeing failures turns the scale into a story about the winner. A stranger should be able to check the ledger hash and rebuild the sandboxes without asking the original authors for a private archive.

The JSON Lines record below is a proposal for that freeze, and it is not an executed public contest. Operators should reject extra fields that appear only after a candidate has already been scored on the same tasks. Cluster identifiers exist so near-duplicate bugs cannot be counted as independent victories during a later ranking. Licenses travel with the record because a score built on uncleared code is not a public measurement at all.

{"task_id":"ledger-0042","repo":"https://example.com/org/widget.git","base_sha":"8f3c1a9e","failing_tests":["tests/test_queue.py::test_drain"],"failing_assertion":"assert queue.drain() == []","oracle_sha256":"c4f19a22","license":"MIT","difficulty":"B","holdout":true,"cluster_id":"queue-drain","created_at":"2026-09-14T00:00:00Z"}

A tiny Python gate can refuse to score any run whose ledger digest does not match the pinned value. That refusal is the entire point of the freeze, because a mutable file is not a dataset in this method. Operators can store the digest in the same commit as the harness and block later task additions.

# proposal: refuse scoring when the ledger is not pinned
import hashlib, json, pathlib, sys

def ledger_digest(path: str) -> str:
    raw = pathlib.Path(path).read_bytes()
    return hashlib.sha256(raw).hexdigest()

def assert_pinned(expected: str, path: str = "tasks.jsonl") -> None:
    digest = ledger_digest(path)
    if digest != expected:
        sys.stderr.write(f"ledger mismatch: {digest}\n")
        raise SystemExit(2)

if __name__ == "__main__":
    assert_pinned(sys.argv[1])
    print("ledger pinned; scoring may proceed")

After the file is hashed, a labeled git sequence can freeze the ledger before any model token is spent. Pull requests that add tasks after a ranking exists should fail that gate rather than negotiate a new percentage. The commands below are an example of that freeze, not a claim about any contest already run.

python pin_ledger.py "$(sha256sum tasks.jsonl | awk '{print $1}')"
git add tasks.jsonl pin_ledger.py
git commit -m "freeze task ledger before agent runs"

Report a Vector Instead of a Trophy

A single pass rate is a trophy, and trophies travel farther than the footnotes that made them possible. A re-runnable score should carry pass rate, spent tokens, wall-clock seconds, retry count, and the harness commit beside every agent name. Without those companions, two agents can share a percentage while one quietly spent an afternoon the other was denied. The metric vector is the practical difference between a lab notebook and a brightly printed billboard.

Controls belong in the same table as the candidate, or the candidate is competing against an unnamed ghost. A cheap frozen baseline run on the identical ledger shows whether the environment itself drifted between Tuesday and Thursday. Pairwise movement against that baseline matters more than absolute percentages, because the grader and the sandbox are shared. If the baseline moves by itself, the ranking is measuring the laboratory rather than measuring the agent.

# proposal: emit a score vector instead of a single percentage
from dataclasses import dataclass, asdict
import json, time

@dataclass
class ScoreVector:
    agent_id: str
    ledger_sha256: str
    harness_sha: str
    passed: int
    total: int
    tokens: int
    wall_seconds: float
    retries: int
    baseline_id: str
    delta_pass: float

def finalize(agent_id, ledger, harness, passed, total, tokens, t0, retries, baseline_pass):
    vec = ScoreVector(
        agent_id=agent_id,
        ledger_sha256=ledger,
        harness_sha=harness,
        passed=passed,
        total=total,
        tokens=tokens,
        wall_seconds=time.time() - t0,
        retries=retries,
        baseline_id="control-frozen",
        delta_pass=(passed / total) - baseline_pass,
    )
    print(json.dumps(asdict(vec), indent=2))

Publishing that JSON next to a blog claim is what keeps the number from becoming marketing copy. Readers can reject the ranking when the ledger hash, the harness commit, or the baseline identifier is absent. They can also reject it when wall-clock is missing while an unbounded retry loop is left undescribed in the write-up. Silence in those fields is a result, and it should grade the evaluation as incomplete rather than praise the agent.

Isolate the Grader From the Agent

The grader must not share a context window, a working directory, or a network namespace with the agent that produced the patch. Otherwise the oracle becomes a collaborator, and the percentage measures a conversation rather than a concrete repair. A practical control is to run the agent in one sandbox, copy only the patch artifact, and grade in a second process. That split is dull plumbing, which is exactly why glossy write-ups skip it in favor of a single percentage.

A second isolation axis is the machine that hosts the harness, because authoring laptops leak helper files into eval trees. Running evaluations on the same workstation that authors the tasks invites extra scripts, cached credentials, and accidental fixtures. A dedicated evaluation host, even a modest free server, keeps the development tree from leaking into the published score. Cheap frozen models, when available as a free control stratum, belong on that host so the baseline cannot load local plugins.

MonkeyCode is relevant here only as one place that currently offers free model access and a free server option for running such a separated harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims are the only product facts used in this workflow, and they are not a substitute for ledger hashes. Teams that already own an isolated runner can apply the same method without that product, which is the intended test of remaining usefulness.

# proposal: agent writes a patch, grader sees only the artifact
python run_agent.py --task-id ledger-0042 --out /tmp/patch.diff
scp /tmp/patch.diff eval-host:/eval/incoming/ledger-0042.diff
ssh eval-host 'python grade_patch.py --ledger tasks.jsonl --task ledger-0042 --patch /eval/incoming/ledger-0042.diff'

The grader should hash the incoming diff, apply it on a clean worktree at the recorded base revision, and execute only listed tests. It should not fetch extra context from the agent transcript, and it should not retry tests with mutated interpreter flags. If the patch fails to apply, the result is a failed task, not an invitation to edit the frozen ledger. That severity is what prevents the dataset from becoming a living document of the winner's private taste.

Near-duplicate clustering should happen before the first agent call, using the failing assertion text rather than later model output. The proposal script below groups task identifiers that would otherwise inflate a pass rate through copied bugs. Clusters can keep one task in the reported holdout and move siblings to a leakage watch list that never enters the headline rate. Doing this after seeing scores would be another way to write the dataset around the winner, so the script belongs in the freeze commit.

# proposal: cluster tasks by failing assertion text before any agent run
import hashlib, json, pathlib, re
from collections import defaultdict

def normalize(text: str) -> str:
    return re.sub(r"\s+", " ", text.strip().lower())

def cluster_file(path: str) -> dict:
    groups = defaultdict(list)
    for line in pathlib.Path(path).read_text().splitlines():
        rec = json.loads(line)
        key = normalize(rec["failing_assertion"])
        digest = hashlib.sha256(key.encode()).hexdigest()[:12]
        groups[digest].append(rec["task_id"])
    return dict(groups)

if __name__ == "__main__":
    clusters = cluster_file("tasks.jsonl")
    for digest, task_ids in clusters.items():
        if len(task_ids) > 1:
            print(digest, ",".join(task_ids))

Why the Numbers Are Not Marketing

Marketing numbers omit the freeze, the vector, and the isolation because those details make the story slower to read. A methodology that publishes the ledger digest, the harness commit, the baseline identifier, and the spent budget cannot hide extra retries. The same methodology also cannot claim permanence, because a ledger that never grows will eventually become stale against real repositories. Honesty here is a process of re-freezing on a calendar, not a promise that one percentage will remain interesting.

This workflow does not certify models for regulated domains, and it does not replace human review on security-sensitive patches. It assumes operators can pin git commits, hash files, and keep the grader offline from the agent under test. Small demos often cannot meet that bar, and deadline pressure often collapses the rule against labeling difficulty after failures. Readers who need a vendor bake-off with contractual service promises should hire an independent lab rather than adopt this notebook.

Authors who cannot publish the task sources or licenses should not present a percentage as a public ranking of agents. Teams that must mutate tests after seeing agent patches are iterating on a product, not measuring one in public. Anyone hoping free model access or a free server option will remain unchanged should treat those options as convenience rather than constants. Convenience can host the harness for a weekend experiment, yet it cannot replace the frozen ledger or the score vector.

A reasonable next step is to freeze a dozen licensed tasks, run one cheap control model and one candidate, and publish both hashes. Operators who want a separated host for that control run may look at MonkeyCode's free server option after the ledger already exists. The ranking should still make sense if that product line is removed from the paragraph, because the measurement lives in the freeze.

Pin the Oracle Bundle Before the Agent Percentage

Avery Wang — Sat, 12 Sep 2026 18:48:52 +0000

A coding-agent percentage is not a measurement until dataset, oracle, metrics, and environment pin are frozen as one bundle. Naked success rates behave like product screenshots because later readers cannot reconstruct the track beneath the stopwatch. The method below treats that bundle as the unit of publication and withholds a percentage when any piece is missing. The harness is a labeled proposal, not a report of a completed vendor bake-off.

Marketing numbers fail the way a race fails when officials forget to paint the lanes. The athlete still runs, the clock still ticks, and a time still appears on a slide. Nobody can name the distance, and nobody can prove the finish line stayed put. Coding-agent demos repeat that failure when a chat log is graded by taste or when hidden edits move the tests after the answer lands.

The dataset is a directory of task folders rather than a prompt buried in a README file. Each task needs a stable identifier, a frozen prompt, a fixture tree, and a short license note. Tasks should be held out from any corpus the operator actually fine-tunes, and they should stay small enough for a reviewer to read. The layout that follows is clerical infrastructure, not evidence that a public leaderboard was executed here.

benchmark/
  manifest.json
  env_pin.json
  tasks/
    T001_csv_join/
      prompt.md
      fixtures/
      oracle/
        expected_files.json
        tests.py
    T002_header_normalize/
      prompt.md
      fixtures/
      oracle/
        expected_files.json
        tests.py

The manifest is the table of contents, and it should hash every file that can change a later score. Publishing a percentage without those hashes resembles publishing a checksum while withholding the bytes. Ordinary shell tools can write the manifest before any agent process is allowed to start. That ordering matters because a hash computed after a green run can hide an oracle that moved.

#!/usr/bin/env bash
set -euo pipefail
python3 - <<'PY'
import hashlib, json, os, time
root = "benchmark"
files = []
for dirpath, _, names in os.walk(root):
    for name in sorted(names):
        path = os.path.join(dirpath, name)
        if name in {"manifest.json", "env_pin.json"}:
            continue
        with open(path, "rb") as handle:
            digest = hashlib.sha256(handle.read()).hexdigest()
        files.append({"path": os.path.relpath(path, root), "sha256": digest})
payload = {
    "schema": "oracle-bundle/v1",
    "created_unix": int(time.time()),
    "file_count": len(files),
    "files": files,
}
os.makedirs(root, exist_ok=True)
with open(os.path.join(root, "manifest.json"), "w", encoding="utf-8") as handle:
    json.dump(payload, handle, indent=2)
print("wrote benchmark/manifest.json", "files", len(files))
PY

The oracle is the benchmark because it is the only object that can disagree with the agent in binary terms. An oracle folder should list required files, forbidden files, size caps, and tests whose exit status counts as success. Taste, summary charm, and fluent self-explanation belong in a reviewer note, which is a different instrument. Mixing those notes into the same average is how a demo transcript becomes a leaderboard cell.

{
  "must_exist": ["src/join.py", "tests/test_join.py"],
  "must_not_exist": ["secrets.env"],
  "max_bytes_per_file": 65536
}

# proposed file: score_task.py
from __future__ import annotations

import json
import subprocess
import sys
from pathlib import Path


def load_oracle(task_dir: Path) -> dict:
    path = task_dir / "oracle" / "expected_files.json"
    return json.loads(path.read_text(encoding="utf-8"))


def check_files(workspace: Path, oracle: dict) -> dict:
    missing = [item for item in oracle["must_exist"] if not (workspace / item).exists()]
    forbidden = [item for item in oracle["must_not_exist"] if (workspace / item).exists()]
    limit = int(oracle["max_bytes_per_file"])
    oversized = []
    for rel in oracle["must_exist"]:
        path = workspace / rel
        if path.exists() and path.stat().st_size > limit:
            oversized.append(rel)
    return {
        "missing": missing,
        "forbidden": forbidden,
        "oversized": oversized,
        "files_ok": not (missing or forbidden or oversized),
    }


def run_oracle_tests(task_dir: Path, workspace: Path) -> dict:
    test_file = task_dir / "oracle" / "tests.py"
    if not test_file.exists():
        return {"ran": False, "passed": False, "reason": "missing_tests"}
    proc = subprocess.run(
        [sys.executable, str(test_file), str(workspace)],
        capture_output=True,
        text=True,
        timeout=30,
    )
    return {
        "ran": True,
        "passed": proc.returncode == 0,
        "stdout_tail": proc.stdout[-500:],
        "stderr_tail": proc.stderr[-500:],
    }


def score_task(task_dir: Path, workspace: Path) -> dict:
    oracle = load_oracle(task_dir)
    files = check_files(workspace, oracle)
    tests = run_oracle_tests(task_dir, workspace)
    passed = bool(files["files_ok"] and tests.get("passed"))
    return {
        "task_id": task_dir.name,
        "passed": passed,
        "files": files,
        "tests": tests,
    }


if __name__ == "__main__":
    result = score_task(Path(sys.argv[1]), Path(sys.argv[2]))
    print(json.dumps(result, indent=2))
    raise SystemExit(0 if result["passed"] else 1)

Metrics are functions of oracle output, not adjectives sprayed across a transcript after the fact. A defensible starter set stays small: binary pass, count of failed checks, wall-clock seconds, and a budget declared before the run. Those fields are stored on failures as well, because a trophy cabinet cannot explain collapse later. Rolling them into one glossy figure should remain optional inside the published envelope.

# proposed file: bundle_report.py
from __future__ import annotations

import json
from pathlib import Path


def failed_check_count(task: dict) -> int:
    files = task.get("files", {})
    tests = task.get("tests", {})
    n = 0
    n += len(files.get("missing", []))
    n += len(files.get("forbidden", []))
    n += len(files.get("oversized", []))
    if tests.get("ran") and not tests.get("passed"):
        n += 1
    if not tests.get("ran"):
        n += 1
    return n


def summarize(results: list[dict]) -> dict:
    n_tasks = len(results)
    n_passed = sum(1 for item in results if item["passed"])
    return {
        "n_tasks": n_tasks,
        "n_passed": n_passed,
        "failed_checks": sum(failed_check_count(item) for item in results),
        "pass_rate": None if n_tasks == 0 else round(n_passed / n_tasks, 4),
        "incomplete": n_tasks == 0,
        "tasks": results,
    }


def reject_naked_percentage(report: dict, manifest_ok: bool, env_pin: dict) -> dict:
    pin_ready = bool(env_pin.get("python") and env_pin.get("platform"))
    if not manifest_ok or not pin_ready or report["incomplete"]:
        report["pass_rate"] = None
        report["publishable"] = False
        report["reason"] = "oracle_bundle_incomplete"
        return report
    report["publishable"] = True
    report["reason"] = "oracle_bundle_complete"
    return report


if __name__ == "__main__":
    raw = json.loads(Path("results.json").read_text(encoding="utf-8"))
    pin = json.loads(Path("benchmark/env_pin.json").read_text(encoding="utf-8"))
    report = reject_naked_percentage(summarize(raw), True, pin)
    print(json.dumps(report, indent=2))

The environment pin stops the same oracle from drifting under a new interpreter or a new testing plugin. Recording the Python version, a lockfile hash, and a platform string is usually enough for tiny coding tasks. Without that pin, a failed reproduction can be dismissed as the machine, which is how marketing numbers outlive other laptops. The snippet below writes the pin as JSON beside the manifest rather than in a slide footnote.

python3 - <<'PY'
import hashlib, json, pathlib, platform, sys
lock = pathlib.Path("requirements.lock")
digest = hashlib.sha256(lock.read_bytes()).hexdigest() if lock.exists() else None
payload = {
    "python": sys.version.split()[0],
    "platform": platform.platform(),
    "lock_sha256": digest,
}
pathlib.Path("benchmark").mkdir(exist_ok=True)
pathlib.Path("benchmark/env_pin.json").write_text(json.dumps(payload, indent=2), encoding="utf-8")
print(json.dumps(payload, indent=2))
PY

Contamination is a dataset defect, not a model personality trait, and it belongs in the methodology before any ranking is discussed. If a task prompt already lives in a public fine-tune dump, a pass may measure memorization of the oracle rather than repair skill. A local n-gram check will not prove cleanliness, yet it will catch the embarrassing copies that sit on disk. The proposed scanner below flags overlapping shingles between prompt files and a corpus the operator actually possesses.

# proposed file: shingle_scan.py
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path


def shingles(text: str, width: int = 12) -> set[str]:
    tokens = text.lower().split()
    grams = []
    for index in range(0, max(0, len(tokens) - width + 1)):
        piece = " ".join(tokens[index : index + width])
        grams.append(hashlib.sha1(piece.encode("utf-8")).hexdigest())
    return set(grams)


def scan(task_prompt: Path, corpus: Path) -> dict:
    left = shingles(task_prompt.read_text(encoding="utf-8"))
    right = shingles(corpus.read_text(encoding="utf-8"))
    overlap = left & right
    denom = max(len(left), 1)
    return {
        "prompt": str(task_prompt),
        "corpus": str(corpus),
        "overlap": len(overlap),
        "prompt_shingles": len(left),
        "overlap_ratio": round(len(overlap) / denom, 4),
        "flag": (len(overlap) / denom) >= 0.05 if left else False,
    }


if __name__ == "__main__":
    print(json.dumps(scan(Path(sys.argv[1]), Path(sys.argv[2])), indent=2))

A five percent overlap flag is a tripwire, not a scientific threshold copied from a paper. Operators should tune it against their own corpus and then freeze the constant inside the bundle so later scores stay comparable. The point is to keep dataset hygiene visible, because an invisible hygiene step is where marketing numbers usually hide. Tasks that trip the flag should be replaced or disclosed, not silently left in the average.

Controls belong in the same envelope because a pass rate without a baseline is a vibe wearing extra decimal places. The control is a weaker or cheaper agent scored on the identical oracle, not a theatrical rival review. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option that can host that control lane on the same frozen files.

export BUNDLE_HASH="$(python3 -c 'import hashlib,pathlib; print(hashlib.sha256(pathlib.Path("benchmark/manifest.json").read_bytes()).hexdigest())')"
python3 shingle_scan.py benchmark/tasks/T001_csv_join/prompt.md local_corpus.txt
python3 score_task.py benchmark/tasks/T001_csv_join /tmp/control-workspace > /tmp/control-T001.json
python3 score_task.py benchmark/tasks/T001_csv_join /tmp/candidate-workspace > /tmp/candidate-T001.json
python3 bundle_report.py

A published report should look like a sealed envelope rather than a keynote headline with a large percentage. It carries the manifest hash, the environment pin, per-task oracle outcomes, control outcomes, and only then a pass rate. If a vendor cannot attach those objects, the percentage should be read as a demo caption. The rule is unkind to slideware and kind to whoever must maintain the generated code.

The approach is the wrong instrument for several honest jobs that still matter to working teams. It will not rank chat quality, architectural taste, or the pleasure of a generated explanation, because those properties have no binary oracle here. It will not certify a model for production incident response, where the environment cannot be frozen and the task mix shifts. Teams that cannot publish even hashed fixtures should not ask strangers to trust the resulting percentage.

Readers who already keep agent evaluations in a lab notebook can adopt the oracle bundle without switching models. The extra discipline is mostly clerical, and it rewards anyone who later has to reproduce a disputed number. A cheaper control lane matters only because it makes that clerical loop repeatable on the same files. Operators who need that lane can try the free access already noted above, then publish the envelope rather than the headline.

Control Deltas Keep Agent Percentages Honest

Avery Wang — Fri, 11 Sep 2026 16:21:16 +0000

A coding-agent percentage is marketing until the dataset, the metrics, and the control runs sit beside it. Public claims that models already outcode most developers compress those three pieces into one headline number. The compression is convenient for sharing and disastrous for comparison, because two labs can report eighty percent on incomparable work. This protocol keeps the denominator visible even when evaluation rides on free endpoints and shared servers.

Think of an agent leaderboard as a race photographed without the track length, the wind, or a paced control runner. The finish-line smile is real, yet the photograph cannot tell a later team whether the course moved. Benchmarks fail the same way when tasks drift, oracles leak, or the harness itself authors the passing tests. A frozen scorecard is simply the same photo with the track markings still clearly in frame.

Seal Dataset and Contract Together

The dataset is the track itself, and it must be sealed before the first candidate agent run. A JSONL file of tasks, each with a prompt, a workspace tarball, and an oracle command, is enough for a lab notebook. The important property is immutability after the freeze, not size, because a moving corpus turns yesterday's score into an anecdote. The Python snippet below writes a compact lock file over the corpus and the metric contract together.

{"id": "t01", "prompt": "Create hello.txt with one line: ok", "oracle": "test -f hello.txt && grep -qx ok hello.txt"}
{"id": "t02", "prompt": "Create sum.py that prints 3 for 1+2", "oracle": "python3 sum.py | grep -qx 3"}
{"id": "t03", "prompt": "Write README.md containing the word sealed", "oracle": "grep -q sealed README.md"}

#!/usr/bin/env python3
"""Seal tasks.jsonl and metric_contract.json into one lock digest."""
from __future__ import annotations

import argparse
import hashlib
import json
from pathlib import Path


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            digest.update(chunk)
    return digest.hexdigest()


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--tasks", required=True)
    parser.add_argument("--contract", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()
    tasks = Path(args.tasks)
    contract = Path(args.contract)
    payload = {
        "tasks_path": tasks.name,
        "tasks_sha256": sha256_file(tasks),
        "contract_path": contract.name,
        "contract_sha256": sha256_file(contract),
        "task_count": sum(1 for line in tasks.open() if line.strip()),
    }
    payload["lock_sha256"] = hashlib.sha256(
        json.dumps(payload, sort_keys=True).encode("utf-8")
    ).hexdigest()
    Path(args.out).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
    print(payload["lock_sha256"])


if __name__ == "__main__":
    main()

The lock is not a trophy; it is a join key that later rows must match or be discarded as mixed. Regenerating tasks after seeing a candidate fail is a silent course change, even when the new tasks look fair. The laboratory either amends the contract in public and restarts both controls, or it keeps the original lock. Private edits that exist only on one laptop are how honest percentages quietly become marketing copy.

Define Rates Before Any Agent Runs

Metrics become marketing when a single ratio absorbs infrastructure failures, skipped tasks, and luck on a tiny sample. A contract should name the numerator, the denominator, and every exclusion before any agent is allowed to run. Pass rate uses only eligible tasks, while infrastructure faults occupy a separate column that cannot rescue the headline. The formulas below are deliberately boring, because boredom is what keeps a lab from renaming a weak result.

{
  "version": "1.0",
  "numerator": "oracle_pass_on_eligible_tasks",
  "denominator": "eligible_task_count",
  "exclusions": ["unpack_error", "runner_abort", "oracle_crash"],
  "ranking_allowed_when": {
    "min_eligible": 30,
    "min_control_delta": 0.15,
    "min_echo_delta": 0.15
  },
  "rates_to_publish": ["pass_rate", "infra_rate", "control_delta", "echo_delta"]
}

def classify(oracle_rc: int, runner_fault: bool) -> str:
    if runner_fault:
        return "infra"
    if oracle_rc == 0:
        return "pass"
    if oracle_rc < 0:
        return "infra"
    return "fail"


def rates(rows: list[dict]) -> dict:
    eligible = [row for row in rows if row["class"] != "infra"]
    infra = [row for row in rows if row["class"] == "infra"]
    passed = [row for row in eligible if row["class"] == "pass"]
    denom = max(len(eligible), 1)
    return {
        "n_total": len(rows),
        "n_eligible": len(eligible),
        "n_infra": len(infra),
        "pass_rate": len(passed) / denom if eligible else None,
        "infra_rate": len(infra) / max(len(rows), 1),
    }


def deltas(candidate: dict, null: dict, echo: dict) -> dict:
    if candidate["pass_rate"] is None:
        return {"control_delta": None, "echo_delta": None}
    return {
        "control_delta": candidate["pass_rate"] - (null["pass_rate"] or 0.0),
        "echo_delta": candidate["pass_rate"] - (echo["pass_rate"] or 0.0),
    }

Eligible means the oracle ran to completion, the workspace unpacked, and the harness did not abort on a runner fault. A timeout that belongs to the shared machine is an infrastructure fault, not evidence that the agent cannot code. Folding those faults into failures inflates drama, while folding them into passes inflates the brochure instead. The contract therefore stores three rates, and only the eligible pass rate may appear in a ranking sentence.

Run Null and Echo Through the Same Harness

Controls are the paced runners, and a candidate that barely beats them has not demonstrated skill. A null agent submits an empty diff, exposing tasks whose oracles already pass on the untouched tree. An echo agent copies comments from the prompt into a dummy file, catching oracles that grade presence rather than behavior. Both controls must travel through the same harness, clock, and exclusion rules as the candidate under test.

from pathlib import Path


def null_agent(workspace: Path, prompt: str) -> None:
    del workspace, prompt  # empty diff on purpose


def echo_agent(workspace: Path, prompt: str) -> None:
    (workspace / "echo.txt").write_text(prompt[:200] + "\n", encoding="utf-8")


def dummy_candidate(workspace: Path, prompt: str) -> None:
    # Synthetic stand-in, not a product measurement.
    text = prompt.lower()
    if "hello.txt" in text:
        (workspace / "hello.txt").write_text("ok\n", encoding="utf-8")
    elif "sum.py" in text:
        (workspace / "sum.py").write_text("print(1 + 2)\n", encoding="utf-8")
    else:
        (workspace / "notes.txt").write_text(prompt[:80] + "\n", encoding="utf-8")

Clock and environment still leak into scores when one run uses a warm cache and another pays a cold start. The harness therefore records start time, runner fingerprint, and a boolean for cache reuse, then refuses mixed rows. Mixed rows are how a free shared server accidentally becomes a hidden independent variable across candidate comparisons. Pairing candidate and controls on the same fingerprint is more important than collecting extra tasks after the environment drifted.

Teams that lack dedicated GPU budget often evaluate on whatever endpoint remains free that week, which is a constraint rather than a sin.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access and a free server option that can host this harness without rewriting the scorecard rules. The product participates in the run as infrastructure, not as a substitute for published controls, metrics, or dataset identity. If a team needs a no-cost runner, MonkeyCode's free model access and free server remain one option after the manifest is committed.

Emit a Scorecard or Emit Nothing

The scorecard writer is the laboratory notebook, and it should refuse to print a percentage without the lock digest. Each row records candidate identity, control scores, eligible count, infrastructure faults, and the metric contract version. Downstream dashboards may render a chart, but they should read this JSON rather than recomputing ad hoc ratios from logs. The command sequence that follows is labeled as an unexecuted example, not as a reported benchmark result.

#!/usr/bin/env python3
"""Unexecuted laboratory template: emit a scorecard or refuse ranking."""
from __future__ import annotations

import hashlib
import json
import os
import platform
import subprocess
import tempfile
from datetime import datetime, timezone
from pathlib import Path


def fingerprint() -> str:
    raw = f"{platform.platform()}|{platform.python_version()}|{os.getcwd()}"
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16]


def run_oracle(workspace: Path, command: str) -> int:
    try:
        completed = subprocess.run(
            command,
            cwd=workspace,
            shell=True,
            check=False,
            capture_output=True,
            text=True,
            timeout=20,
        )
        return completed.returncode
    except (OSError, subprocess.TimeoutExpired):
        return -1


def evaluate(agent, tasks: list[dict]) -> list[dict]:
    rows = []
    for task in tasks:
        with tempfile.TemporaryDirectory() as tmp:
            workspace = Path(tmp)
            runner_fault = False
            try:
                agent(workspace, task["prompt"])
            except Exception:
                runner_fault = True
            rc = -1 if runner_fault else run_oracle(workspace, task["oracle"])
            kind = classify(rc, runner_fault)
            rows.append({"id": task["id"], "class": kind, "oracle_rc": rc})
    return rows


def ranking_allowed(contract: dict, candidate: dict, gap: dict) -> bool:
    gate = contract["ranking_allowed_when"]
    if candidate["n_eligible"] < gate["min_eligible"]:
        return False
    if gap["control_delta"] is None or gap["echo_delta"] is None:
        return False
    return (
        gap["control_delta"] >= gate["min_control_delta"]
        and gap["echo_delta"] >= gate["min_echo_delta"]
    )


def emit_scorecard(lock: dict, contract: dict, candidate: dict, null: dict, echo: dict) -> dict:
    gap = deltas(candidate, null, echo)
    allowed = ranking_allowed(contract, candidate, gap)
    return {
        "lock_sha256": lock["lock_sha256"],
        "contract_version": contract["version"],
        "access_date": datetime.now(timezone.utc).date().isoformat(),
        "runner_fingerprint": fingerprint(),
        "cache_reuse": False,
        "null": null,
        "echo": echo,
        "candidate": candidate,
        "control_delta": gap["control_delta"],
        "echo_delta": gap["echo_delta"],
        "ranking_allowed": allowed,
        "ranking_sentence": None,
    }

# Unexecuted laboratory template, not a reported result.
python3 freeze_lock.py --tasks tasks.jsonl --contract metric_contract.json --out scorecard.lock
python3 scorecard_lab.py --lock scorecard.lock --out scorecard.json
python3 -m json.tool scorecard.json

A skeptical reader starts at control_delta, not at pass_rate, because a high pass on a contaminated oracle is still noise. Echo_delta answers a different contamination, namely oracles that reward any file the prompt already described in prose. When both deltas sit near zero, the laboratory publishes the method and withholds ranking language without embarrassment. The JSON also keeps infra_rate visible so a flaky free runner cannot be laundered into a story about intelligence.

A number stops being marketing when a skeptical reader can rebuild the denominator without emailing the authors. That means the lock file, the metric contract, the control binaries, and the exclusion log are published with the percentage. It also means the lab withholds ranking language until the candidate beats both controls by a pre-registered margin. Absent that margin, the honest sentence is that the method could not separate the agent from an empty diff.

This protocol will not satisfy a regulated audit, a production SLA, or a procurement process that needs certified workloads. Teams with fewer than thirty eligible tasks should not publish a ranking, because control deltas will swing with single cases. Vendors selling a percentage, and readers hunting a model name to copy into a pitch deck, should use a different genre. Free endpoints also change under operators, so the scorecard records an access date without pretending the lane will remain identical.

The industry does not lack agent percentages; it lacks published denominators that survive a week of scrutiny. Sealing the dataset, naming the metrics, and running null and echo controls is slower than a screenshot of a chat window. That slowness is the point, because a comparable ranking is a laboratory result rather than a vibe with extra decimal places. Keep the scorecard boring, keep the controls in the same harness, and let the percentage earn the right to travel.

Canary Gates Make Agent Rankings Comparable

Avery Wang — Thu, 10 Sep 2026 04:34:57 +0000

A coding-agent percentage remains marketing until protocol, canaries, and a cost ledger freeze beside it. Teams still quote a lonely pass rate as if that number could travel without a suitcase of hidden choices. Prompt text, tool allowlists, sandbox images, and grader prompts often move the score more than the model does. The honest unit of publication is therefore a registered protocol rather than a percentage standing alone on a slide.

The protocol resembles a flight plan much more than a souvenir photograph of cruising altitude. A photograph of thirty thousand feet says nothing about fuel load, weather, or whether the altimeter was calibrated. Agent leaderboards behave the same way when they publish altitude without publishing the plan that produced it. Pre-registration does not make models stronger; it only makes later numbers comparable to the original claim.

A useful registry records the corpus identifier, the hidden holdout rule, and the grader isolation boundary before scored runs. It also records decoding settings, the tool allowlist, and the network policy that the sandbox will enforce during execution. Those fields look bureaucratic until a rerun drifts after someone quietly upgrades a linter inside the evaluation image. The registry exists to make that drift visible to readers, not to decorate a repository README with extra YAML.

Dataset construction starts with task families that a human can grade without watching the agent think out loud. Each family needs a public cousin for debugging and a hidden cousin reserved until the prompt hash is locked. Difficulty labels are operator judgments and should be treated as metadata, not as a second score competing with pass rate. Duplicate statements of the same bug should collapse to one holdout item so the percentage does not double-count a skill.

Holdout tasks must be sampled after the system prompt is frozen, not before that freeze is committed. Otherwise the prompt is tuned against the same families that later appear inside the headline metric. University exam design offers a close analogy: writing the study guide after seeing the final is not measurement. A protocol that cannot state when the holdout was sampled is not ready to emit a public percentage.

Negative-control canaries sit beside that holdout as a pulse check on the harness rather than on the model. A canary is a task the operator already knows should fail, or should pass, for reasons independent of model quality. If a known-dead canary starts passing, the grader is leaking, the sandbox is dirty, or string matching is too generous. If a known-easy canary starts failing, the environment moved, and the later pass rate is describing that movement.

Pass rate remains a useful column in the report, but it cannot serve as the entire published result. A registered run should also carry attempt count, wall-clock time, and an approximate token ledger for agent and grader. Variance across seeds belongs in the same row, because a small gap inside the noise band is not a ranking. Cost belongs there because two agents that pass the same hidden tests at very different spend are not interchangeable.

None of those extra fields require a glamorous dashboard; they require a schema that refuses incomplete score files. Marketing copy prefers a single integer because a single integer is easy to screenshot and easy to rank. A methodology that wants citation rather than advertising treats a missing canary result as a hard error. The refusal is closer to a test suite that never ran than to a footnote about optional diagnostics.

Grader isolation remains the control most often skipped when a small team assembles a weekend harness. The agent should not see hidden tests, and the grader should not see chain of thought unless the protocol says so. Shared context is how a model appears to understand a constraint that never appeared in the user-visible prompt. Isolation is not hostility toward agents; it is the same wall that separates student answers from the answer key.

Controls include a frozen temperature, a capped tool-output budget, and a network policy that defaults to deny. The sandbox image digest belongs in the lockfile because package churn inside an image is an unlisted independent variable. Clock source matters for wall-time: measuring on a busy laptop mixes agent latency with thermal throttling and browser updates. The token ledger should separate agent tokens from grader tokens so a verbose judge cannot masquerade as model spend.

The following example is a proposed workflow, not a claim about any executed leaderboard or vendor result. It stores a protocol document, hashes that document, runs canaries first, and only then writes a scored pass rate. Operators can adapt the same shape to their own corpus; the point is the refusal path, not a particular framework.

# protocol.yaml — proposed schema, not a published result
schema_version: 1
protocol_id: agent-eval-2026-09-10
prompt_ref: prompts/system.v3.txt
corpus_ref: datasets/public_tasks.sha256
holdout_rule: sample_after_prompt_freeze
holdout_seed: 20260910
canaries:
  - id: dead_import_cycle
    expect: fail
  - id: hello_fs_write
    expect: pass
controls:
  temperature: 0.0
  max_tool_output_bytes: 65536
  network: deny
  tools: [edit, shell, pytest]
  sandbox_image: eval-runner@sha256:replace-me
grader:
  isolation: hidden_tests_only
  sees_chain_of_thought: false
metrics_required:
  - pass_rate
  - n_tasks
  - n_seeds
  - wall_clock_s
  - token_in_agent
  - token_out_agent
  - token_in_grader
  - token_out_grader
  - canary_status

A tiny registrar can refuse to proceed when the document is incomplete or when the hash has not been committed. The commands below are ordinary and local; they do not imply any particular vendor quota, model name, or hardware. The helper reads YAML, checks required keys, and writes a lockfile that later score files must cite. Operators should commit that lockfile before any holdout run that they intend to quote.

# register_protocol.py — example helper, unexecuted in this article
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

import yaml

REQUIRED = (
    "protocol_id",
    "prompt_ref",
    "corpus_ref",
    "holdout_rule",
    "canaries",
    "controls",
    "grader",
    "metrics_required",
)


def load_protocol(path: Path) -> dict:
    data = yaml.safe_load(path.read_text())
    missing = [key for key in REQUIRED if key not in data]
    if missing:
        raise SystemExit(f"protocol missing fields: {missing}")
    if data.get("holdout_rule") != "sample_after_prompt_freeze":
        raise SystemExit("holdout must be sampled after the prompt freeze")
    if not data["canaries"]:
        raise SystemExit("refusing to register a protocol without canaries")
    return data


def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> None:
    protocol_path = Path(sys.argv[1])
    protocol = load_protocol(protocol_path)
    record = {
        "protocol_id": protocol["protocol_id"],
        "protocol_sha256": digest(protocol_path),
        "prompt_sha256": digest(Path(protocol["prompt_ref"])),
        "corpus_sha256": Path(protocol["corpus_ref"]).read_text().strip(),
    }
    out = Path("protocol_lock.json")
    out.write_text(json.dumps(record, indent=2) + "\n")
    print(f"registered {record['protocol_id']} -> {out}")


if __name__ == "__main__":
    main()

python register_protocol.py protocol.yaml
test -f protocol_lock.json
sha256sum protocol.yaml prompts/system.v3.txt
python -c "import json; print(json.load(open('protocol_lock.json'))['protocol_sha256'])"
git add protocol.yaml protocol_lock.json prompts/system.v3.txt
git commit -m "lock evaluation protocol before scored runs"

The runner should treat canaries as a gate rather than as extra rows averaged into the headline percentage. Averaging a known-dead task into the pass rate hides harness failure inside what looks like model failure. The sketch below prints a score only after every canary matches its expected pass or fail polarity. Replace the stubbed sandbox call before any real use; the article does not report executed benchmark numbers.

# run_gated_eval.py — proposed control flow, not an executed result
from __future__ import annotations

import json
import time
from pathlib import Path


def run_task(task_id: str) -> dict:
    """Replace with a real sandbox call. This stub is a shape, not a result."""
    raise NotImplementedError("wire this to an isolated runner")


def assert_canaries(canaries: list[dict]) -> None:
    for spec in canaries:
        result = run_task(spec["id"])
        passed = bool(result["passed"])
        should_pass = spec["expect"] == "pass"
        if passed != should_pass:
            raise SystemExit(
                f"canary {spec['id']} broke the harness; refusing score"
            )


def summarize(results: list[dict], started: float) -> dict:
    n = len(results)
    passes = sum(1 for row in results if row["passed"])
    return {
        "n_tasks": n,
        "pass_rate": None if n == 0 else passes / n,
        "wall_clock_s": time.time() - started,
        "token_in_agent": sum(row.get("token_in_agent", 0) for row in results),
        "token_out_agent": sum(row.get("token_out_agent", 0) for row in results),
        "token_in_grader": sum(row.get("token_in_grader", 0) for row in results),
        "token_out_grader": sum(row.get("token_out_grader", 0) for row in results),
        "canary_status": "ok",
    }


def main() -> None:
    lock = json.loads(Path("protocol_lock.json").read_text())
    protocol = json.loads(Path("protocol_expanded.json").read_text())
    required = {"pass_rate", "n_tasks", "wall_clock_s", "canary_status"}
    assert_canaries(protocol["canaries"])
    started = time.time()
    results = [run_task(task_id) for task_id in protocol["holdout_task_ids"]]
    summary = summarize(results, started)
    missing = sorted(required - set(summary))
    if missing:
        raise SystemExit(f"score file missing metrics: {missing}")
    summary["protocol_sha256"] = lock["protocol_sha256"]
    Path("score.json").write_text(json.dumps(summary, indent=2) + "\n")


if __name__ == "__main__":
    main()

The important branch is the hard exit on a mismatched canary, which keeps a dirty sandbox from looking like progress. Without that branch, the rest of the file is only a pretty printer for whatever the environment happened to do. With it, an upgraded linter, a leaked test file, or a broken import cannot mint a number that resembles improvement. Readers who cannot reproduce the exit condition cannot reproduce the claim, regardless of how tidy the percentage looks.

Local laptops are poor clocks for wall-time comparisons and poor isolation boundaries when the agent receives a shell. A remote runner with a known image digest is closer to a laboratory bench than to a developer notebook full of extras. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option that can host this gated harness off a single workstation.

That pairing matters because protocol hashes are weak if every rerun happens on a different unmanaged laptop. The free server option is a stable place to execute the locked document, not a substitute for canaries or isolation. The free model access is only one interchangeable backend behind the same allowlist and the same hidden tests. Swapping backends without swapping the protocol is the entire practical point of registration in this workflow.

Marketing selects a peak result, then discards the envelope that made the peak possible to observe. A registered protocol publishes that envelope, including the canaries that would have forbidden the run entirely. Readers can disagree with the corpus and still reproduce the envelope, which differs from arguing with a screenshot. If a later post omits the protocol hash, the cost ledger, or the canary gate, the percentage has returned to advertising.

The method remains narrow and should not be sold as a general verdict on coding agents or developer productivity. It does not estimate real-user utility, and it does not certify safety properties of tools that execute arbitrary code. It also does not rescue a corpus that is too small or too close to widely circulated training material. Teams that need a launch headline, or that cannot isolate a grader from the agent trace, should not use this workflow.

Researchers comparing decoding settings and platform engineers checking sandbox drift are the intended readers of this protocol. They already suspect that a lonely pass rate is a costume worn by an unpublished bundle of choices. Pre-registration simply gives that suspicion a file format, a hash, and a refusal path when a dead canary stands up. The percentage can come last, after the lockfile exists, which is the opposite order from a marketing screenshot.

Write the Protocol Before the First Agent Run

Avery Wang — Wed, 09 Sep 2026 16:35:04 +0000

A coding-agent percentage is a measurement only when a protocol file exists before the first task runs. Screenshots, live demos, and leaderboard rows can look precise while the dataset and the scorer still move. Reviewers should treat an unpublished protocol like a lab treats an unlabeled reagent on a crowded bench. The reading may be physically real, and it still cannot be compared with any later reading.

The working analogy is a kitchen scale that is zeroed after the flour is already in the bowl. Moving the tare after the pour does not invent extra grain, yet it quietly rewrites every later claim. Agent evaluations repeat that gesture when tasks are added because a model looked weak on Tuesday. A frozen protocol does not make the agent smarter; it makes tomorrow's percentage talk about the same object.

A useful dataset for this method is a holdout corpus with an explicit split, not a folder of favorite bugs. Each task needs a stable identifier, a language, a fixture path, a timeout, and a public-versus-hidden test flag. Tasks that authors used while prompting the agent belong in a development split and must never enter the reported score. The reported split should stay sealed until the protocol document is written, because curiosity itself is a contamination path.

The protocol file is the original artifact of the method, and it should be boring on purpose. YAML is enough, because reviewers can diff it, hash it, and refuse results that lack the file. The example below is labeled as a template rather than as a published benchmark, since no live corpus is claimed here.

# protocol.yaml — template, not a published leaderboard
schema: agent-eval-protocol/v1
name: sealed-holdout-demo
created: "2026-09-09"
splits:
  dev: ["task-001", "task-002"]
  report: ["task-101", "task-102", "task-103"]
tasks:
  task-101:
    language: python
    fixture: fixtures/task-101.tar
    timeout_s: 120
    public_tests: tests/public_test.py
    hidden_tests: tests/hidden_test.py
    network: deny
metrics:
  - hidden_pass
  - wall_clock_ms
  - undeclared_network
  - writes_outside_workspace
aggregation: macro_by_language
controls:
  dummy_agent: empty_diff
  comment_agent: apply_public_hint
  toolchain_lock: ".tool-versions"
limits:
  max_output_bytes: 1048576

That file is the contract between the authors and anyone who will later cite the percentage. Changing a timeout after seeing a trace is not tuning; it is a new experiment that needs a new name. Teams that keep the YAML in git and refuse to score untagged commits already have most of this discipline. The remaining work is to make the scorer read the file instead of a hallway conversation.

Metrics have to be named in that same document, or they will drift toward whichever number looks kind in a screenshot. Hidden-test pass rate is one metric, not a synonym for quality, and it should sit beside wall-clock time and undeclared network use. An agent that memorizes public tests the way a student memorizes an answer key will inflate pass rate while the companion metrics stay ugly. Aggregation is part of the metric: micro-average across tasks is not macro-average across languages, and mixing them after the run rewrites the claim.

A small local scorer can enforce that contract without a platform and without a private dashboard. The Python below is a proposal for local use, not a report of production numbers. It should refuse to compute a percentage when the protocol hash is missing from the log. Labeled examples like this one are for method illustration and should not be cited as benchmark results.

# harness_score.py — proposal for a local sealed-split scorer
import hashlib, json, subprocess, sys, time, yaml
from collections import defaultdict
from pathlib import Path

def protocol_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def load_protocol(path: Path) -> dict:
    doc = yaml.safe_load(path.read_text(encoding="utf-8"))
    if "splits" not in doc or "report" not in doc["splits"]:
        raise ValueError("protocol missing sealed report split")
    return doc

def run_agent(cmd, workspace, timeout):
    start = time.monotonic()
    proc = subprocess.run(cmd, cwd=workspace, capture_output=True, timeout=timeout)
    elapsed_ms = int((time.monotonic() - start) * 1000)
    return proc.returncode, elapsed_ms

def macro_by_language(rows):
    bucket = defaultdict(list)
    for row in rows:
        bucket[row["language"]].append(row["hidden_pass"])
    means = [sum(v) / len(v) for v in bucket.values() if v]
    return sum(means) / len(means) if means else 0.0

if __name__ == "__main__":
    proto_path = Path(sys.argv[1])
    digest = protocol_hash(proto_path)
    protocol = load_protocol(proto_path)
    print(json.dumps({"protocol_sha256": digest, "aggregation": protocol["aggregation"]}))

Controls are the unglamorous half of the method, and they are the half that marketing posts usually skip. The host should pin the toolchain and deny outbound network unless a task declares a need. It should always run a dummy agent that emits an empty diff before any candidate is scored. That dummy is a floor control, and a score above zero means the tests or the scorer are leaking.

A second control can apply one obvious patch taken from a comment in the public tests. That weak agent estimates how much of the task was already solved in the prose of the tests. Together the floor and the weak ceiling give the later candidate percentage a scale, the way a photograph needs a ruler in the frame. Without those two control runs, a ninety percent row is a cropped photo, not a measurement.

# dummy_agent.py — floor control: must score zero on a sealed hidden split
from pathlib import Path
import sys

def main(workspace: str) -> None:
    # Write a marker only. A nonzero hidden_pass means the harness is leaking.
    Path(workspace, "AGENT_RAN").write_text("dummy\n", encoding="utf-8")

if __name__ == "__main__":
    main(sys.argv[1])

# comment_agent.py — weak ceiling: apply a hint only if public tests mention it
import re, sys
from pathlib import Path

HINT = re.compile(r"HINT:\s+(\S+)\s+->\s+(\S+)")

def main(workspace: str) -> None:
    root = Path(workspace)
    public = root.joinpath("tests", "public_test.py")
    if not public.exists():
        return
    match = HINT.search(public.read_text(encoding="utf-8"))
    if not match:
        return
    src, dst = match.group(1), match.group(2)
    src_root = root.joinpath("src")
    if not src_root.exists():
        return
    for path in src_root.rglob("*.py"):
        body = path.read_text(encoding="utf-8")
        if src in body:
            path.write_text(body.replace(src, dst), encoding="utf-8")

if __name__ == "__main__":
    main(sys.argv[1])

A short command sequence keeps the floor visible in the log before any candidate is allowed to run. Operators who skip the dummy run are asking the candidate agent to grade its own classroom. The commands below assume a Unix shell and a locked toolchain file sitting beside the protocol. Printing the hashes next to the dummy score makes later candidate rows harder to detach from the protocol.

sha256sum protocol.yaml .tool-versions
python dummy_agent.py ./workspaces/task-101
python harness_score.py protocol.yaml
# Continue only if dummy hidden_pass is exactly 0.0 on the report split.
python comment_agent.py ./workspaces/task-101
python harness_score.py protocol.yaml

When a local machine cannot host the candidate agent, the protocol file should travel with the run. Rewriting the protocol for a new host creates a second experiment under the same percentage label. MonkeyCode currently offers free model access and a free server option for teams that need a spare runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A remote runner still has to print the protocol hash, the dummy floor, and the candidate rows in one log. If the server image cannot pin the toolchain, the comparison is a different experiment and needs a different protocol name. Pasting a failing assertion back into the agent breaks the holdout seal in a single keystroke. The cheap runner is useful when it preserves isolation, not when it becomes a second editor.

The method does not detect training-set leakage against the public internet, and it does not certify fairness across languages or difficulty bands. Teams that need a sales slide by Friday should not use it, because a sealed holdout often lowers the published percentage. Groups without authority to freeze the task list will produce theater no matter how pretty the YAML looks. This approach also does not belong in suites that fail under the dummy agent on a second run.

A number that survives the dummy floor, the comment ceiling, and a hashed protocol is still not a verdict on intelligence. It is a comparable observation, which is a narrower and more honest claim than a marketing row. Readers who already keep protocol files can ignore any vendor runner and keep the YAML. Readers who need a spare machine for the same sealed split may try that free server on their own holdout.

Coding Agents Fail Differently by Stratum

Avery Wang — Mon, 07 Sep 2026 14:22:15 +0000

Coding-agent leaderboards still collapse unlike tasks into one pass rate, and that collapse is the product. A suite that mixes short Python katas with multi-file refactors is not one experiment. The mix of languages, difficulty, and hidden tests moves the number before any model runs. A protocol that freezes strata, sample size, and seeds can publish an interval another lab can rerun.

Sports desks would not average a marathon clock with a hundred-meter sprint and call the blend fitness. Coding evaluations still fold regex patches and schema migrations into one cheerful percent. Readers then treat that percent as a property of the model rather than of the basket. The honest unit of reporting is the stratum, with any overall figure labeled as a weighted summary only.

This article proposes a frozen evaluation protocol, not a vendor bake-off and not a claim about any named model. The protocol treats dataset mix, trial count, and seed list as controls that must exist before a percentage is allowed to print. The scripts below are a reproducible method other teams can run on their own traces, not unpublished private scores. No pass rates from a hidden leaderboard appear, because those numbers would recreate the original problem.

The dataset is the first control, and it is a mix rather than a pile of files. Each task needs a stable identifier, a language, a difficulty band, and a visibility flag for hidden tests. A holdout that never appears in prompt text belongs in a separate column so leakage can be audited later. If SQL migrations and Python katas share one average, the published figure mostly reports how the author weighted the homework.

A useful analogy is a kitchen scale that tares itself after every ingredient except salt. The displayed mass then tracks the salt habit of the cook, not the recipe. Coding scores behave the same way when easy autocompletes dominate the suite and rare multi-file repairs hide in the tail. Stratified sampling does not make models smarter; it stops the tail from being outvoted by trivia.

Metrics have to survive a rerun on another machine, which a naked percentage never does. Pass rate remains allowed, but only beside trial count, a confidence interval, and the per-stratum rates that produced it. Wall-clock timeout belongs in the same record because a slow agent can look identical to a wrong agent after the harness kills the process. Temperature and decoding seed belong there too, since they decide whether two labs even ran the same stochastic experiment.

The Wilson score interval is a boring choice on purpose, because coding trials are often small and extreme. A point estimate of eight successes in ten looks decisive until the interval reminds the reader that ten is a rumor. Binomial independence is an assumption, not a gift from the runtime, and shared caches can still correlate trials. The scorecard therefore prints the assumption in the same JSON object as the percentage, instead of burying it in a blog caveat.

Controls have to be frozen before the first generation, not edited after a disappointing cell in a spreadsheet. Sample size n is a budget, not a knob for hunting a prettier headline after seeing the traces. Seeds are a list committed to version control, not a random draw that changes when the author dislikes a failure. Timeout, temperature, and the stratum weights are likewise protocol fields; changing them midstream creates a new benchmark that should not keep the old name.

The proposed harness never calls a model. It grades recorded Bernoulli outcomes against a frozen protocol so the statistics can be checked without network access. Teams may fill outcomes.csv from local runners, continuous integration, or a hosted coding assistant, as long as the protocol file does not change under the data. The command surface is intentionally small so the methodology stays visible in review.

Create a protocol file that refuses to be vague. The weights must sum to one, and the seed list must match n exactly, or the loader should stop.

# PROTOCOL.toml — proposed frozen controls, not an executed vendor study
name = "coding-agent-strata-v1"
n = 10
timeout_s = 120
temperature = 0.0
seeds = [101, 102, 103, 104, 105, 106, 107, 108, 109, 110]

[weights]
python_easy = 0.40
python_refactor = 0.25
sql_migration = 0.20
ts_api = 0.15

Pair that file with a tiny task catalog. Hidden tests stay out of any prompt template that later models will see, and each row names exactly one stratum.

{
  "tasks": [
    {"id": "py-kata-017", "stratum": "python_easy", "hidden_tests": true},
    {"id": "py-ref-004", "stratum": "python_refactor", "hidden_tests": true},
    {"id": "sql-mig-009", "stratum": "sql_migration", "hidden_tests": true},
    {"id": "ts-api-021", "stratum": "ts_api", "hidden_tests": true}
  ]
}

Record one row per trial rather than a pre-averaged percentage. A later script can recompute intervals; a spreadsheet cell cannot reconstruct the seed that produced a lucky pass.

task_id,stratum,seed,passed,elapsed_ms
py-kata-017,python_easy,101,1,1840
py-kata-017,python_easy,102,1,1912
py-ref-004,python_refactor,101,0,120000
sql-mig-009,sql_migration,101,1,4022
ts-api-021,ts_api,101,0,8871

The grader loads those three artifacts, checks that every stratum in the catalog has a frozen weight, and refuses to print when trial counts drift. Wilson intervals are computed per stratum first. A weighted headline prints only after that gate, and it is labeled as a summary rather than as the result.

# protocol.py — proposed scorecard, not a published leaderboard
from __future__ import annotations

import csv, json, math, sys
from collections import defaultdict
from pathlib import Path

try:
    import tomllib
except ModuleNotFoundError:  # Python 3.10
    import tomli as tomllib  # type: ignore

Z = 1.959963984540054  # ~95% Wilson


def wilson(successes: int, n: int) -> tuple[float, float, float]:
    if n <= 0:
        raise ValueError("n must be positive before a rate is printed")
    p = successes / n
    denom = 1.0 + Z * Z / n
    center = (p + Z * Z / (2 * n)) / denom
    half = (Z / denom) * math.sqrt(p * (1 - p) / n + Z * Z / (4 * n * n))
    return p, max(0.0, center - half), min(1.0, center + half)


def load_protocol(path: Path) -> dict:
    proto = tomllib.loads(path.read_text())
    seeds = list(proto["seeds"])
    n = int(proto["n"])
    if len(seeds) != n or len(set(seeds)) != n:
        raise SystemExit("seed list must be unique and exactly length n")
    weights = proto["weights"]
    total = sum(weights.values())
    if abs(total - 1.0) > 1e-9:
        raise SystemExit(f"stratum weights must sum to 1.0, got {total}")
    return proto


def main() -> None:
    proto = load_protocol(Path("PROTOCOL.toml"))
    catalog = json.loads(Path("tasks.json").read_text())["tasks"]
    by_id = {row["id"]: row for row in catalog}
    wanted = set(proto["weights"])
    found = {row["stratum"] for row in catalog}
    if wanted != found:
        raise SystemExit(f"catalog strata {found} != protocol weights {wanted}")

    grouped: dict[str, list[int]] = defaultdict(list)
    with Path("outcomes.csv").open() as handle:
        for row in csv.DictReader(handle):
            task = by_id[row["task_id"]]
            if task["stratum"] != row["stratum"]:
                raise SystemExit(f"stratum drift on {row['task_id']}")
            if int(row["seed"]) not in proto["seeds"]:
                raise SystemExit(f"seed {row['seed']} is not in the frozen list")
            grouped[row["stratum"]].append(int(row["passed"]))

    strata = {}
    headline = 0.0
    for name, weight in proto["weights"].items():
        trials = grouped.get(name, [])
        if len(trials) != proto["n"] * sum(1 for t in catalog if t["stratum"] == name):
            raise SystemExit(f"{name} does not have n trials for every task")
        p, lo, hi = wilson(sum(trials), len(trials))
        strata[name] = {
            "n": len(trials),
            "pass_rate": round(p, 4),
            "wilson95": [round(lo, 4), round(hi, 4)],
            "weight": weight,
        }
        headline += weight * p

    scorecard = {
        "protocol": proto["name"],
        "controls": {
            "n_per_task": proto["n"],
            "timeout_s": proto["timeout_s"],
            "temperature": proto["temperature"],
            "seeds": proto["seeds"],
        },
        "independence_assumption": "trials treated as Bernoulli; shared caches can invalidate this",
        "strata": strata,
        "weighted_summary_only": round(headline, 4),
        "do_not_cite_without_strata": True,
    }
    Path("scorecard.json").write_text(json.dumps(scorecard, indent=2))
    json.dump(scorecard, sys.stdout, indent=2)
    sys.stdout.write("\n")


if __name__ == "__main__":
    main()

Run the method against fixture traces before wiring any generator. The exit status is part of the protocol: a missing stratum is a failed experiment, not a blank cell to fill later.

python3 protocol.py
python3 -c "import json; print(json.load(open('scorecard.json'))['strata'])"

A scorecard that cannot be cited without strata is the point of the extra JSON noise. Reviewers can reject a pull request that quotes weighted_summary_only in a title while omitting wilson95. Continuous integration can store PROTOCOL.toml beside the outcomes so later authors cannot quietly reweight Python trivia after the run. The percentage is then a derived view, like a chart, rather than the scientific object.

Hosted runners become relevant when the protocol needs many independent trials and local machines are already booked. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant with free model access and a free server option, which can replay the same frozen protocol without turning this text into a quota sheet. The method still requires recorded outcomes, frozen strata, and an interval; a free endpoint does not repair a blended task mix.

The method has sharp limits, and those limits are part of the result. Wilson intervals assume independent Bernoulli trials, which shared repositories, warm language servers, and leaked hidden tests can quietly violate. Stratification cannot save a catalog whose easy band is still drawn from public interview questions that models have already seen. Small n produces honest intervals that are too wide to rank two close systems, which is an answer, not a defect in the printer.

Procurement teams should not use this scorecard as a sole vendor gate, because mix design still encodes the buyer's taste. Marketing teams should not harvest weighted_summary_only for a banner, because the field is labeled as a summary on purpose. Classrooms that need a binary lab grade can keep pass or fail per student and skip intervals entirely. Researchers who cannot freeze seeds or hidden tests should not pretend the percentage is comparable across weeks.

Cheap generation has made it easy to emit code and even easier to emit leaderboards that look like evidence. The expensive part is now the boring bookkeeping: a versioned mix, a committed seed list, and an interval that survives contact with another lab. The next honest publication is a per-stratum record with n written beside the rate, not a single percent asked to represent five languages at once.