An agent percentage is not a measurement until the scoring harness itself can be replayed against frozen golden traces. Leaderboards still publish a single success rate as if the grader, the timeouts, and the parse rules were as stable as the task files. Those hidden pieces move whenever a prompt template, a JSON schema, or a retry policy changes underneath the same dataset name. This article treats harness calibration as the first published result, and the candidate agent as a later comparison that inherits that protocol.
Public conversation keeps returning to a familiar worry that evaluation suites age faster than the systems they claim to rank. That worry is reasonable, because a test that once discriminated among tools can become a puzzle that many large models memorize. Expanding the suite without locking the grader simply multiplies an unstable instrument across more task files. A more honest step is to freeze a small golden-trace pack and refuse ranking until the harness scores those traces identically twice.
A golden trace is not a unit test for the agent; it is a unit test for the judge that assigns credit. Each trace records the task identifier, the exact stdout, the exit status, and the wall time from the original run. The protocol card then states how those fields map onto pass, fail, skip, and parse_error, including the timeout ceiling. If two independent checkouts of the harness disagree on a golden trace, the candidate ranking is invalid rather than merely late.
The dataset side of the card is a digest, not a friendly folder name on someone's laptop. Task files, fixtures, and the golden traces themselves are hashed together so a renamed repository cannot quietly swap a prompt. The metric contract sits beside that digest and names the aggregation rule, because micro-averages and family macros are not interchangeable summaries. Controls in this design are golden replay, an order shuffle, and an idempotent rescore, which catch graders that depend on global state.
Protocol card
{
"protocol_id": "harness.v3.golden",
"dataset_glob": "tasks/*.json",
"dataset_sha256": "replace-after-hashing",
"golden_glob": "golden/*.json",
"metrics": {
"pass_rule": "exit_code==0 and schema_ok and wall_ms<=timeout_ms",
"timeout_ms": 15000,
"aggregate": "macro_by_task_family"
},
"controls": ["golden_replay", "idempotent_rescore", "order_shuffle"]
}
That object is intentionally boring, because a timeout and a hash travel better than a rounded marketing percentage. The card also refuses to mention a candidate agent, since the harness must stand on its own before any comparison is fair. Teams that skip this file usually discover the omission later, when a one-line JSON parsing change moves every historical score. A leaderboard percentage without a calibrated judge resembles a stopwatch that was never zeroed at the starting line.
Replay kit
The following Python module implements the three controls against a directory of traces and prints a calibration report. It is labeled as a local replay proposal, not as evidence from a production contest or a vendor bake-off. Readers should replace the digest placeholders after the first real run, rather than copying a fictional hash into a blog table.
# harness_calibrate.py
from __future__ import annotations
import hashlib
import json
import random
from pathlib import Path
from typing import Any, Callable, Dict, List, Tuple
Trace = Dict[str, Any]
Score = Dict[str, Any]
def sha256_tree(root: Path, pattern: str) -> str:
digest = hashlib.sha256()
paths = sorted(root.glob(pattern))
if not paths:
raise FileNotFoundError(f"no files for {pattern}")
for path in paths:
digest.update(path.name.encode("utf-8"))
digest.update(path.read_bytes())
return digest.hexdigest()
def load_json_dir(root: Path, pattern: str) -> List[Trace]:
items: List[Trace] = []
for path in sorted(root.glob(pattern)):
payload = json.loads(path.read_text())
payload["_source"] = path.name
items.append(payload)
return items
def score_trace(trace: Trace, timeout_ms: int) -> Score:
schema_ok = isinstance(trace.get("stdout"), str)
if schema_ok and trace.get("expect_json"):
try:
json.loads(trace["stdout"])
except json.JSONDecodeError:
schema_ok = False
elif trace.get("expect_json"):
schema_ok = False
wall = int(trace.get("wall_ms", timeout_ms + 1))
passed = (
int(trace.get("exit_code", 1)) == 0
and schema_ok
and wall <= timeout_ms
)
return {
"id": trace.get("id", trace.get("_source")),
"family": trace.get("family", "ungrouped"),
"pass": passed,
"parse_error": not schema_ok,
"timeout": wall > timeout_ms,
}
def macro_by_family(scores: List[Score]) -> Dict[str, float]:
buckets: Dict[str, List[int]] = {}
for row in scores:
buckets.setdefault(row["family"], []).append(int(row["pass"]))
return {
name: (sum(vals) / len(vals) if vals else 0.0)
for name, vals in sorted(buckets.items())
}
def idempotent_rescore(traces: List[Trace], scorer: Callable[[Trace], Score]) -> bool:
first = [scorer(item) for item in traces]
second = [scorer(item) for item in traces]
return first == second
def order_shuffle_stable(
traces: List[Trace],
scorer: Callable[[Trace], Score],
seeds: Tuple[int, ...] = (0, 1, 7, 13),
) -> bool:
baseline = macro_by_family([scorer(item) for item in traces])
for seed in seeds:
shuffled = list(traces)
random.Random(seed).shuffle(shuffled)
again = macro_by_family([scorer(item) for item in shuffled])
if again != baseline:
return False
return True
def calibrate(root: Path, timeout_ms: int = 15000) -> Dict[str, Any]:
traces = load_json_dir(root, "golden/*.json")
scorer = lambda item: score_trace(item, timeout_ms)
scored = [scorer(item) for item in traces]
expected = {item["id"]: bool(item["must_pass"]) for item in traces}
mismatches = [row["id"] for row in scored if row["pass"] != expected.get(row["id"])]
report = {
"dataset_sha256": sha256_tree(root, "tasks/*.json"),
"golden_sha256": sha256_tree(root, "golden/*.json"),
"golden_n": len(traces),
"mismatches": mismatches,
"idempotent": idempotent_rescore(traces, scorer),
"order_stable": order_shuffle_stable(traces, scorer),
"macro_by_family": macro_by_family(scored),
}
report["ready_for_candidates"] = (
not mismatches and report["idempotent"] and report["order_stable"]
)
return report
if __name__ == "__main__":
print(json.dumps(calibrate(Path(".")), indent=2))
A matching golden file makes the contract visible to anyone who clones the repository and runs the module. The fixture below is synthetic on purpose, so it should not be cited as a result from a live agent evaluation. Known-good traces prove the happy path still parses; they do not prove that the candidate model understood the task.
{
"id": "json.extract.001",
"family": "extract",
"exit_code": 0,
"wall_ms": 820,
"expect_json": true,
"stdout": "{\"ok\": true, \"value\": 42}",
"must_pass": true
}
A second fixture should fail on purpose, because a harness that only sees successes cannot detect a grader that always returns true. Known-bad traces act like a laboratory blank, proving the instrument can still reject malformed output. Without that blank, a broken parser that treats every blob as valid JSON will inflate later agent scores by one hidden constant. Publishing the blank is what separates a measurement from a brochure, even when the candidate percentage looks impressive.
{
"id": "json.extract.002",
"family": "extract",
"exit_code": 0,
"wall_ms": 640,
"expect_json": true,
"stdout": "almost json but not",
"must_pass": false
}
Commands for a stranger replaying the kit stay small and should live in continuous integration beside the percentage job. The hash step belongs in that pipeline so a silent fixture edit cannot land next to an unchanged published score. A failing calibration job is the desired outcome when someone improves the parser without updating golden expectations.
python -m pip install pytest
python harness_calibrate.py
python - <<'PY'
from pathlib import Path
from harness_calibrate import calibrate
report = calibrate(Path("."))
assert report["ready_for_candidates"], report
print("harness calibrated:", report["dataset_sha256"][:12])
PY
A short pytest file turns those assertions into a gate that ranking jobs must pass before any candidate is scored. The test does not claim that any hosted model is strong; it claims that the local judge is boringly consistent. Timeout handling is part of the metric contract, so a slow but pretty answer is still a failure under this protocol.
# test_harness_calibrate.py
from pathlib import Path
from harness_calibrate import calibrate, score_trace
def test_golden_pack_locks_the_judge():
report = calibrate(Path("."))
assert report["mismatches"] == []
assert report["idempotent"] is True
assert report["order_stable"] is True
assert report["ready_for_candidates"] is True
def test_timeout_is_part_of_the_metric():
slow = {
"id": "slow.001",
"family": "extract",
"exit_code": 0,
"wall_ms": 20000,
"stdout": "{}",
"expect_json": True,
}
row = score_trace(slow, timeout_ms=15000)
assert row["pass"] is False
assert row["timeout"] is True
Only after that gate is green does a candidate agent deserve a percentage stored beside the protocol hashes. People reading a blog table rarely notice a missing digest, which is why marketing scores travel farther than protocol cards. A fair write-up lists the candidate score as a comparison that inherits the frozen harness, not as an isolated trophy. Family-level rates should remain visible, because a two-item family must not be summarized as if it contained a hundred tasks.
When a team wants a second machine to rerun the candidate arm after calibration, a free remote shell can be enough. Disclosure: This article was prepared as part of MonkeyCode's product outreach, and the product notes below are availability claims rather than benchmark results. MonkeyCode currently offers free model access and a free server option that can host the same replay commands after local calibration. Those notes are not a quota, a hardware spec, or a claim that any particular model won the synthetic tasks above.
What the numbers still cannot mean
The numbers stay non-marketing for a dull reason: every published figure names the digest it depends on. A reader who cannot obtain the golden traces cannot audit the judge, and a reader who cannot hash the tasks cannot detect silent edits. Golden traces still freeze yesterday's failures, so they will not detect a novel cheating strategy the original agents never tried. A purely syntactic JSON check will miss semantic wrongness, and an unpinned model judge will reintroduce the drift this protocol tried to remove.
The shuffle test only proves order independence of the scorer, not independence from the host clock or a polluted virtualenv. Teams that should skip this workflow include daily prompt shootouts and classrooms that want a live demo more than a ledger. Vendors that refuse to publish failing traces cannot use this kit without turning the blank into another marketing surface. Safety-critical evaluations still need human review on top of schema checks, because a well-formed object can still be a harmful action.
Stale tests will keep aging, and public models will keep saturating yesterday's puzzles without asking anyone's permission. Ranking can still be honest if the first published artifact is the harness report, with hashes, mismatches, and a ready_for_candidates bit. Candidate percentages then become a second paragraph instead of the only sentence a reader remembers from the table.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Top comments (0)