An agent score is trustworthy only when the dataset, the metrics, and the controls are frozen before the first scored run. A convenient pass on a free server can debug the harness, yet that same pass cannot become a public claim. The method below treats a dry run as rehearsal and reserves the headline number for a locked protocol with a refusal slice. Readers who skip that lock will publish a demonstration of fluency rather than a measurement of task completion.
Public write-ups often celebrate a clean streak after several quiet retries, which hides the policy that actually produced the streak. An agent that never abstains can look stronger than an agent that refuses an underspecified task and asks for the missing file. Those two behaviors are not the same capability, even when both transcripts end with a green check. A score that counts only final success will reward confident fabrication and will punish a careful refusal.
The dataset in this method is a small sealed pack of thirty tasks, split into twenty ordinary tasks and ten negative-control tasks. Ordinary tasks have a single expected artifact, such as a function, a failing test that must turn green, or a short migration note. Negative-control tasks omit a required input on purpose, so the honest outcome is an abstention rather than a guessed patch. The pack is hashed before any model call, and the hash travels with every later report.
Four metrics travel together, and none of them is allowed to stand alone as a slogan. Task success is the fraction of ordinary tasks whose emitted artifact matches the frozen oracle byte for byte. Abstention accuracy is the fraction of negative-control tasks that stop without inventing a missing file or a hidden requirement. Invalid completion is the fraction of negative-control tasks that still emit a confident patch, which is a failure even if the patch compiles.
A report that quotes task success without abstention accuracy is incomplete, because a perfect streak can hide total failure on the refusal slice. Retry count, tool-call count, and wall time are covariates, not decorations, and they must be printed beside the rates. Changing the retry ceiling after a disappointing run creates a new experiment, even when the dataset hash stays the same. That is why these numbers cannot be lifted into marketing copy: they describe one locked procedure, not a general property of a product.
Controls are boring on purpose, and that boredom is exactly the point of a publishable method. The decoder settings, the tool allow-list, the retry ceiling, and the route name are written into a manifest before the scored lane starts. A route lock means the scored lane may not silently fall over to a different model when the first route is busy. If the route changes, the run is labeled exploratory and is excluded from the published comparison table.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source project and supplies free model access plus a free server as the exploratory lane for this harness. That lane is useful for checking that the hash step and the scorer actually execute before any scored claim is attempted. Current model names, quotas, hardware, and availability windows are not assumed here and must be read from the project documentation at run time.
A dry run on the free lane never inherits the authority of the sealed pack, because convenience and measurement answer different questions. The exploratory lane may use whatever free model access is currently offered, and it may be repeated while the scorer is still being debugged. The scored lane must name the route, freeze the manifest, and refuse to start if the pack hash disagrees with the recorded digest. Mixing those lanes in one table is how a debugging session turns into an accidental advertisement.
Availability claims for the rehearsal lane stay operator-supplied and are not turned into a quota, a hardware profile, or a permanence promise. A team should confirm the current terms in the project documentation before a scored night, because free access can change without notice. The same methodology remains usable with any other route that can emit the transcript shape the scorer expects. A changed term sheet does not rewrite an older scored manifest, so yesterday's number cannot be silently rebranded.
The artifact is a small Python harness that hashes the task pack, rejects a drifted manifest, and scores success separately from abstention. It is a proposal for local reproduction, not a log from an executed vendor benchmark, and it calls no network endpoint. A reviewer can run the harness against local fixture files before any agent route is connected to the scored lane. The point of the code is to make an invalid comparison fail closed, rather than to crown a particular model.
#!/usr/bin/env python3
"""Proposal scorer for a sealed agent pack. Not an executed vendor result."""
import hashlib
import json
import sys
from pathlib import Path
REQUIRED = ("route", "temperature", "retry_ceiling", "tool_allowlist", "lane")
def pack_digest(task_dir: Path) -> str:
hasher = hashlib.sha256()
files = sorted(path for path in task_dir.glob("*.json") if path.is_file())
for path in files:
hasher.update(path.name.encode())
hasher.update(b"\0")
hasher.update(path.read_bytes())
hasher.update(b"\0")
return hasher.hexdigest()
def load_manifest(path: Path) -> dict:
manifest = json.loads(path.read_text(encoding="utf-8"))
missing = [key for key in REQUIRED if key not in manifest]
if missing:
raise SystemExit("manifest missing controls: " + ", ".join(missing))
if manifest["lane"] != "scored":
raise SystemExit("refusing to score an exploratory lane")
if int(manifest["retry_ceiling"]) < 0:
raise SystemExit("retry_ceiling must be non-negative")
return manifest
def score_task(task: dict, transcript: dict) -> str:
abstained = bool(transcript.get("abstained"))
artifact = transcript.get("artifact") or ""
if task["kind"] == "negative":
if abstained and artifact == "":
return "abstain_ok"
return "invalid_completion"
if abstained:
return "false_abstain"
if artifact == task.get("oracle", ""):
return "success"
return "miss"
def rate(rows, kind, label):
subset = [row for row in rows if row["kind"] == kind]
if not subset:
return None
hits = sum(row["label"] == label for row in subset)
return round(hits / len(subset), 4)
def main() -> None:
task_dir = Path(sys.argv[1])
manifest = load_manifest(Path(sys.argv[2]))
transcripts = json.loads(Path(sys.argv[3]).read_text(encoding="utf-8"))
digest = pack_digest(task_dir)
if manifest.get("pack_sha256") != digest:
raise SystemExit("pack hash drifted; scored lane aborted")
rows = []
for path in sorted(task_dir.glob("*.json")):
task = json.loads(path.read_text(encoding="utf-8"))
label = score_task(task, transcripts[path.stem])
rows.append({"task": path.stem, "kind": task["kind"], "label": label})
report = {
"pack_sha256": digest,
"route": manifest["route"],
"retry_ceiling": manifest["retry_ceiling"],
"lane": manifest["lane"],
"task_success": rate(rows, "ordinary", "success"),
"abstention_accuracy": rate(rows, "negative", "abstain_ok"),
"invalid_completion": rate(rows, "negative", "invalid_completion"),
"rows": rows,
}
json.dump(report, sys.stdout, indent=2)
sys.stdout.write("\n")
if __name__ == "__main__":
main()
A minimal fixture keeps one ordinary task and one negative task, which is enough to prove the scorer before the thirty-task pack exists. The ordinary file stores an oracle string, and the negative file stores an empty oracle plus a note that the required schema was withheld. The transcript file then records whether the agent abstained and what artifact it emitted, using the task stem as the key. Running the scorer against a copied manifest with a stale hash should abort, and that abort is the test the reviewer should see first.
{
"fix-parser.json": {
"kind": "ordinary",
"oracle": "def parse(line):\n return line.strip().split(',')\n"
},
"missing-schema.json": {
"kind": "negative",
"oracle": "",
"withheld": "schema"
}
}
{
"lane": "scored",
"route": "REPLACE_WITH_ROUTE_FROM_CURRENT_DOCS",
"temperature": 0,
"retry_ceiling": 1,
"tool_allowlist": ["read_file"],
"pack_sha256": "PASTE_DIGEST_BEFORE_SCORING"
}
mkdir -p tasks/sealed
python3 -m py_compile score_pack.py
python3 score_pack.py tasks/sealed manifest.json transcripts.json
python3 -c 'import hashlib, pathlib; d=pathlib.Path("tasks/sealed"); h=hashlib.sha256();
[ (h.update(p.name.encode()), h.update(b"\0"), h.update(p.read_bytes()), h.update(b"\0")) for p in sorted(d.glob("*.json")) ]; print(h.hexdigest())'
The command exits before scoring when a control is missing, when the lane is exploratory, or when the pack digest disagrees. A zero exit therefore means only that the protocol held, not that the agent produced an impressive result. The printed rates should travel with the route name and the retry ceiling, because those fields define the experiment. A later reader who cannot recover the digest cannot audit the number, and an unauditable number should stay unpublished.
Consider a hypothetical pack of twenty ordinary tasks and ten negative controls, scored under one named route and a retry ceiling of one. Suppose sixteen ordinary tasks match the oracle, while seven negative controls abstain and three emit a patch anyway. Task success is then 0.80, abstention accuracy is 0.70, and invalid completion is 0.30, which is a mixed result rather than a victory lap. Publishing only the 0.80 figure would conceal the slice where the agent invented missing inputs and still looked fluent.
The refusal slice works like a control sample in a lab notebook, sitting beside the treatment so a clean result cannot stand alone. Without that neighboring sample, a green streak is closer to a demo reel than to a careful measurement. The free server option fits the notebook's margin, where a reviewer checks pens, seals, and arithmetic before the official page is written. It does not fit the official page itself, unless the same route lock and the same pack hash governed that run.
A fair comparison between two routes keeps the same pack digest, retry ceiling, and tool allow-list, changing only the route field. If one route abstains more often and succeeds less often, the table should show both movements instead of declaring a winner from success alone. If one route needed hidden retries inside the runtime, that fact belongs in the covariate column beside the manifest ceiling. The absence of a paired manifest is itself a reason to withhold the comparison from any public note.
Recent public arguments about retries and agreeable models point at the same gap this protocol tries to close. A streak can be an artifact of a generous retry ceiling, and agreement can be an artifact of an undeserving task. Neither observation requires a named leaderboard, a prize purse, or a fresh model announcement to be useful on a Tuesday afternoon. The practical response is to freeze the pack, score the refusal slice, and refuse to promote a lane that was only meant for rehearsal.
This approach is deliberately narrow, and several familiar evaluation problems sit entirely outside the scope of the harness. The oracle comparison assumes a deterministic artifact, so multi-answer tasks need a rubric or a property test first. Thirty tasks can expose a broken harness, yet they cannot support a leaderboard claim about a model family. Temperature drift, hidden tool retries, and a nondeterministic sandbox can still move the rates even when the manifest looks frozen.
Teams that need a procurement number, a regulated benchmark, or a cross-vendor ranking should not use this pack as evidence. A lunch-bound product demo should also skip it, because the refusal slice will slow the story and may embarrass a polished agent. Researchers who already run a large public suite gain little from a thirty-task rehearsal unless they adopt the same route lock on that suite. The method is for engineers who want a publishable local protocol before they quote any agent percentage in public.
Engineers who keep benchmark notes can place this scorer beside a free-server rehearsal, then decide whether a sealed night is warranted. That placement is a workflow choice, not an invitation to treat a rehearsal transcript as a public result. The pack hash, the refusal slice, and the route lock remain the parts worth copying even if that free lane is never used. If those three controls are missing, the percentage should remain in the lab notebook and off the public page.
Top comments (0)