A coding-agent percentage is marketing until the dataset, the metrics, and the control runs sit beside it. Public claims that models already outcode most developers compress those three pieces into one headline number. The compression is convenient for sharing and disastrous for comparison, because two labs can report eighty percent on incomparable work. This protocol keeps the denominator visible even when evaluation rides on free endpoints and shared servers.
Think of an agent leaderboard as a race photographed without the track length, the wind, or a paced control runner. The finish-line smile is real, yet the photograph cannot tell a later team whether the course moved. Benchmarks fail the same way when tasks drift, oracles leak, or the harness itself authors the passing tests. A frozen scorecard is simply the same photo with the track markings still clearly in frame.
Seal Dataset and Contract Together
The dataset is the track itself, and it must be sealed before the first candidate agent run. A JSONL file of tasks, each with a prompt, a workspace tarball, and an oracle command, is enough for a lab notebook. The important property is immutability after the freeze, not size, because a moving corpus turns yesterday's score into an anecdote. The Python snippet below writes a compact lock file over the corpus and the metric contract together.
{"id": "t01", "prompt": "Create hello.txt with one line: ok", "oracle": "test -f hello.txt && grep -qx ok hello.txt"}
{"id": "t02", "prompt": "Create sum.py that prints 3 for 1+2", "oracle": "python3 sum.py | grep -qx 3"}
{"id": "t03", "prompt": "Write README.md containing the word sealed", "oracle": "grep -q sealed README.md"}
#!/usr/bin/env python3
"""Seal tasks.jsonl and metric_contract.json into one lock digest."""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(65536), b""):
digest.update(chunk)
return digest.hexdigest()
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--tasks", required=True)
parser.add_argument("--contract", required=True)
parser.add_argument("--out", required=True)
args = parser.parse_args()
tasks = Path(args.tasks)
contract = Path(args.contract)
payload = {
"tasks_path": tasks.name,
"tasks_sha256": sha256_file(tasks),
"contract_path": contract.name,
"contract_sha256": sha256_file(contract),
"task_count": sum(1 for line in tasks.open() if line.strip()),
}
payload["lock_sha256"] = hashlib.sha256(
json.dumps(payload, sort_keys=True).encode("utf-8")
).hexdigest()
Path(args.out).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
print(payload["lock_sha256"])
if __name__ == "__main__":
main()
The lock is not a trophy; it is a join key that later rows must match or be discarded as mixed. Regenerating tasks after seeing a candidate fail is a silent course change, even when the new tasks look fair. The laboratory either amends the contract in public and restarts both controls, or it keeps the original lock. Private edits that exist only on one laptop are how honest percentages quietly become marketing copy.
Define Rates Before Any Agent Runs
Metrics become marketing when a single ratio absorbs infrastructure failures, skipped tasks, and luck on a tiny sample. A contract should name the numerator, the denominator, and every exclusion before any agent is allowed to run. Pass rate uses only eligible tasks, while infrastructure faults occupy a separate column that cannot rescue the headline. The formulas below are deliberately boring, because boredom is what keeps a lab from renaming a weak result.
{
"version": "1.0",
"numerator": "oracle_pass_on_eligible_tasks",
"denominator": "eligible_task_count",
"exclusions": ["unpack_error", "runner_abort", "oracle_crash"],
"ranking_allowed_when": {
"min_eligible": 30,
"min_control_delta": 0.15,
"min_echo_delta": 0.15
},
"rates_to_publish": ["pass_rate", "infra_rate", "control_delta", "echo_delta"]
}
def classify(oracle_rc: int, runner_fault: bool) -> str:
if runner_fault:
return "infra"
if oracle_rc == 0:
return "pass"
if oracle_rc < 0:
return "infra"
return "fail"
def rates(rows: list[dict]) -> dict:
eligible = [row for row in rows if row["class"] != "infra"]
infra = [row for row in rows if row["class"] == "infra"]
passed = [row for row in eligible if row["class"] == "pass"]
denom = max(len(eligible), 1)
return {
"n_total": len(rows),
"n_eligible": len(eligible),
"n_infra": len(infra),
"pass_rate": len(passed) / denom if eligible else None,
"infra_rate": len(infra) / max(len(rows), 1),
}
def deltas(candidate: dict, null: dict, echo: dict) -> dict:
if candidate["pass_rate"] is None:
return {"control_delta": None, "echo_delta": None}
return {
"control_delta": candidate["pass_rate"] - (null["pass_rate"] or 0.0),
"echo_delta": candidate["pass_rate"] - (echo["pass_rate"] or 0.0),
}
Eligible means the oracle ran to completion, the workspace unpacked, and the harness did not abort on a runner fault. A timeout that belongs to the shared machine is an infrastructure fault, not evidence that the agent cannot code. Folding those faults into failures inflates drama, while folding them into passes inflates the brochure instead. The contract therefore stores three rates, and only the eligible pass rate may appear in a ranking sentence.
Run Null and Echo Through the Same Harness
Controls are the paced runners, and a candidate that barely beats them has not demonstrated skill. A null agent submits an empty diff, exposing tasks whose oracles already pass on the untouched tree. An echo agent copies comments from the prompt into a dummy file, catching oracles that grade presence rather than behavior. Both controls must travel through the same harness, clock, and exclusion rules as the candidate under test.
from pathlib import Path
def null_agent(workspace: Path, prompt: str) -> None:
del workspace, prompt # empty diff on purpose
def echo_agent(workspace: Path, prompt: str) -> None:
(workspace / "echo.txt").write_text(prompt[:200] + "\n", encoding="utf-8")
def dummy_candidate(workspace: Path, prompt: str) -> None:
# Synthetic stand-in, not a product measurement.
text = prompt.lower()
if "hello.txt" in text:
(workspace / "hello.txt").write_text("ok\n", encoding="utf-8")
elif "sum.py" in text:
(workspace / "sum.py").write_text("print(1 + 2)\n", encoding="utf-8")
else:
(workspace / "notes.txt").write_text(prompt[:80] + "\n", encoding="utf-8")
Clock and environment still leak into scores when one run uses a warm cache and another pays a cold start. The harness therefore records start time, runner fingerprint, and a boolean for cache reuse, then refuses mixed rows. Mixed rows are how a free shared server accidentally becomes a hidden independent variable across candidate comparisons. Pairing candidate and controls on the same fingerprint is more important than collecting extra tasks after the environment drifted.
Teams that lack dedicated GPU budget often evaluate on whatever endpoint remains free that week, which is a constraint rather than a sin.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode currently offers free model access and a free server option that can host this harness without rewriting the scorecard rules. The product participates in the run as infrastructure, not as a substitute for published controls, metrics, or dataset identity. If a team needs a no-cost runner, MonkeyCode's free model access and free server remain one option after the manifest is committed.
Emit a Scorecard or Emit Nothing
The scorecard writer is the laboratory notebook, and it should refuse to print a percentage without the lock digest. Each row records candidate identity, control scores, eligible count, infrastructure faults, and the metric contract version. Downstream dashboards may render a chart, but they should read this JSON rather than recomputing ad hoc ratios from logs. The command sequence that follows is labeled as an unexecuted example, not as a reported benchmark result.
#!/usr/bin/env python3
"""Unexecuted laboratory template: emit a scorecard or refuse ranking."""
from __future__ import annotations
import hashlib
import json
import os
import platform
import subprocess
import tempfile
from datetime import datetime, timezone
from pathlib import Path
def fingerprint() -> str:
raw = f"{platform.platform()}|{platform.python_version()}|{os.getcwd()}"
return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16]
def run_oracle(workspace: Path, command: str) -> int:
try:
completed = subprocess.run(
command,
cwd=workspace,
shell=True,
check=False,
capture_output=True,
text=True,
timeout=20,
)
return completed.returncode
except (OSError, subprocess.TimeoutExpired):
return -1
def evaluate(agent, tasks: list[dict]) -> list[dict]:
rows = []
for task in tasks:
with tempfile.TemporaryDirectory() as tmp:
workspace = Path(tmp)
runner_fault = False
try:
agent(workspace, task["prompt"])
except Exception:
runner_fault = True
rc = -1 if runner_fault else run_oracle(workspace, task["oracle"])
kind = classify(rc, runner_fault)
rows.append({"id": task["id"], "class": kind, "oracle_rc": rc})
return rows
def ranking_allowed(contract: dict, candidate: dict, gap: dict) -> bool:
gate = contract["ranking_allowed_when"]
if candidate["n_eligible"] < gate["min_eligible"]:
return False
if gap["control_delta"] is None or gap["echo_delta"] is None:
return False
return (
gap["control_delta"] >= gate["min_control_delta"]
and gap["echo_delta"] >= gate["min_echo_delta"]
)
def emit_scorecard(lock: dict, contract: dict, candidate: dict, null: dict, echo: dict) -> dict:
gap = deltas(candidate, null, echo)
allowed = ranking_allowed(contract, candidate, gap)
return {
"lock_sha256": lock["lock_sha256"],
"contract_version": contract["version"],
"access_date": datetime.now(timezone.utc).date().isoformat(),
"runner_fingerprint": fingerprint(),
"cache_reuse": False,
"null": null,
"echo": echo,
"candidate": candidate,
"control_delta": gap["control_delta"],
"echo_delta": gap["echo_delta"],
"ranking_allowed": allowed,
"ranking_sentence": None,
}
# Unexecuted laboratory template, not a reported result.
python3 freeze_lock.py --tasks tasks.jsonl --contract metric_contract.json --out scorecard.lock
python3 scorecard_lab.py --lock scorecard.lock --out scorecard.json
python3 -m json.tool scorecard.json
A skeptical reader starts at control_delta, not at pass_rate, because a high pass on a contaminated oracle is still noise. Echo_delta answers a different contamination, namely oracles that reward any file the prompt already described in prose. When both deltas sit near zero, the laboratory publishes the method and withholds ranking language without embarrassment. The JSON also keeps infra_rate visible so a flaky free runner cannot be laundered into a story about intelligence.
A number stops being marketing when a skeptical reader can rebuild the denominator without emailing the authors. That means the lock file, the metric contract, the control binaries, and the exclusion log are published with the percentage. It also means the lab withholds ranking language until the candidate beats both controls by a pre-registered margin. Absent that margin, the honest sentence is that the method could not separate the agent from an empty diff.
This protocol will not satisfy a regulated audit, a production SLA, or a procurement process that needs certified workloads. Teams with fewer than thirty eligible tasks should not publish a ranking, because control deltas will swing with single cases. Vendors selling a percentage, and readers hunting a model name to copy into a pitch deck, should use a different genre. Free endpoints also change under operators, so the scorecard records an access date without pretending the lane will remain identical.
The industry does not lack agent percentages; it lacks published denominators that survive a week of scrutiny. Sealing the dataset, naming the metrics, and running null and echo controls is slower than a screenshot of a chat window. That slowness is the point, because a comparable ranking is a laboratory result rather than a vibe with extra decimal places. Keep the scorecard boring, keep the controls in the same harness, and let the percentage earn the right to travel.
Top comments (1)
The 'race photographed without the track length' framing is the most honest description of agent leaderboards I've read. We tried to keep a frozen control task set for our own multi-agent pipeline and the maintenance cost was the surprise: the control quietly stops being a control once the harness version drifts. Did you solve version-stamping the harness itself into the scorecard, or is that still manual in your protocol?