Hash the Task Pack Before Ranking Coding Agents

A coding-agent ranking is trustworthy only after the task pack, the metric, and the run controls are hashed together. Unpinned models, live network calls, and unpublished graders turn yesterday's score into unverifiable advertising. This method treats agent evaluation like an API load test: freeze the workload, name the clocks, and publish the bundle. Strangers should be able to replay the same pack and obtain the same pass or fail vector.

Realistic API tests already refuse to measure a service against the public internet at noon. Agent benches that pull fresh GitHub issues, live docs, or today's model alias repeat that mistake in a prettier dashboard. The workload must be a file and the grader a command, or the published number is only a weather report. The rest of this article builds that closed loop with a small Python harness and a control sheet that travels with every score.

The task pack is a JSONL file of coding jobs, each with a frozen prompt, a fixture tree, and a shell grader. Tasks that need the open web are rewritten against local fixtures so the agent cannot harvest a newer answer from a search tool. Hidden tests stay off the prompt disk and run only after the agent writes an artifact, which keeps leakage from inflating pass rates. A SHA-256 of the sorted pack file becomes the dataset identity that every later score must cite.

Pass rate without weights is a trap when ten toy edits sit beside one multi-file refactor. Each task carries an integer weight for review cost, and the headline metric is weighted pass rate under a wall-clock budget. Median latency and recorded token spend travel as secondary axes so a cheap fast fail cannot hide behind a single percentage. Rankings that omit the budget or the weight vector are incomplete even when every test is green.

The control sheet pins the endpoint identifier, decoding temperature, sampler seed, tool-schema hash, and maximum wall seconds per task. Changing any field creates a new experiment rather than a better score on the old leaderboard. That rule is the same one load testers use when they refuse to mix mild and harsh arrival rates in one percentile chart. If a vendor renames a model alias, the sheet records the old pin as stale and the comparison stops.

A compact pack file looks ordinary on disk, which is the point: the dataset is boring, versioned, and diffable. Each line names one job, points at fixtures, and declares the only command that may grade the agent's output. Reviewers should read the grader before they read the leaderboard, because the command is the metric in executable form. The snippet below is a proposal for a three-task starter pack, not a claim about any public contest.

{"id":"fix-off-by-one","weight":1,"prompt_file":"prompts/fix-off-by-one.md","fixture_dir":"fixtures/fix-off-by-one","grade":["python","graders/fix-off-by-one.py"],"budget_s":90}
{"id":"parse-csv-dates","weight":2,"prompt_file":"prompts/parse-csv-dates.md","fixture_dir":"fixtures/parse-csv-dates","grade":["python","graders/parse-csv-dates.py"],"budget_s":120}
{"id":"split-god-module","weight":5,"prompt_file":"prompts/split-god-module.md","fixture_dir":"fixtures/split-god-module","grade":["python","graders/split-god-module.py"],"budget_s":240}

The harness never talks to the open web during a scored run, and it refuses to start when the pack hash drifts. It writes a bundle that contains the pack digest, the control sheet, per-task pass bits, wall times, and an optional spend field supplied by the adapter. Teams can swap the adapter for a local script, a vendor SDK, or a scratch endpoint without changing the grader. The listing below is labeled as a proposed runner; it is meant to be executed against fixtures the operator already owns.

#!/usr/bin/env python3
"""Proposed agent-benchmark runner. Freeze pack + controls, then emit a bundle."""
from __future__ import annotations

import hashlib, json, os, subprocess, sys, time
from pathlib import Path

PACK = Path("pack.jsonl")
BUNDLE = Path("score_bundle.json")

CONTROLS = {
    "endpoint_id": os.environ.get("AGENT_ENDPOINT", "local-adapter"),
    "temperature": 0.0,
    "seed": 7,
    "tool_schema_sha256": os.environ.get("TOOL_SCHEMA_SHA256", ""),
    "network": "blocked",
}

def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def load_pack(path: Path) -> tuple[str, list[dict]]:
    raw = path.read_bytes()
    digest = sha256_bytes(raw)
    tasks = [json.loads(line) for line in raw.splitlines() if line.strip()]
    if not tasks:
        raise SystemExit("empty pack")
    ids = [t["id"] for t in tasks]
    if len(ids) != len(set(ids)):
        raise SystemExit("duplicate task ids")
    return digest, tasks

def grade_task(task: dict, workdir: Path) -> dict:
    started = time.monotonic()
    budget = int(task["budget_s"])
    try:
        proc = subprocess.run(
            task["grade"],
            cwd=workdir,
            capture_output=True,
            text=True,
            timeout=budget,
            env={**os.environ, "NETWORK": "blocked"},
        )
        elapsed = time.monotonic() - started
        passed = proc.returncode == 0
        return {
            "id": task["id"],
            "weight": int(task["weight"]),
            "passed": passed,
            "returncode": proc.returncode,
            "elapsed_s": round(elapsed, 3),
            "timeout": False,
        }
    except subprocess.TimeoutExpired:
        return {
            "id": task["id"],
            "weight": int(task["weight"]),
            "passed": False,
            "returncode": None,
            "elapsed_s": float(budget),
            "timeout": True,
        }

def weighted_pass(rows: list[dict]) -> float:
    num = sum(r["weight"] for r in rows if r["passed"])
    den = sum(r["weight"] for r in rows)
    return round(num / den, 4) if den else 0.0

def main() -> None:
    pack_sha, tasks = load_pack(PACK)
    expected = os.environ.get("EXPECTED_PACK_SHA256")
    if expected and expected != pack_sha:
        raise SystemExit(f"pack hash mismatch: {pack_sha}")
    rows = []
    for task in tasks:
        workdir = Path("runs") / task["id"]
        workdir.mkdir(parents=True, exist_ok=True)
        # Adapter contract: write artifact into workdir before the grader runs.
        adapter = os.environ.get("AGENT_ADAPTER", "./adapters/local.sh")
        subprocess.run(
            [adapter, task["id"], str(workdir), task["prompt_file"]],
            check=False,
            timeout=int(task["budget_s"]),
        )
        rows.append(grade_task(task, workdir))
    elapsed_values = sorted(r["elapsed_s"] for r in rows)
    median = elapsed_values[len(elapsed_values) // 2]
    bundle = {
        "pack_sha256": pack_sha,
        "controls": CONTROLS,
        "headline_metric": "weighted_pass_under_budget",
        "weighted_pass": weighted_pass(rows),
        "median_elapsed_s": median,
        "tasks": rows,
    }
    BUNDLE.write_text(json.dumps(bundle, indent=2, sort_keys=True))
    print(json.dumps({"pack_sha256": pack_sha, "weighted_pass": bundle["weighted_pass"]}))

if __name__ == "__main__":
    main()

A matching adapter can be a thin shell wrapper that sends the frozen prompt to whatever endpoint the lab is measuring that day. The wrapper must not mutate fixtures, must not fetch extra context, and must exit when the task budget elapses. Spend, if the endpoint reports it, is copied into the bundle as a secondary field rather than folded into the headline pass rate. Mixing spend into the pass fraction without a published formula is how cost disappears from a ranking and reappears as surprise invoices.

export EXPECTED_PACK_SHA256="$(python3 - <<'PY'
import hashlib, pathlib
print(hashlib.sha256(pathlib.Path('pack.jsonl').read_bytes()).hexdigest())
PY
)"
export AGENT_ADAPTER=./adapters/local.sh
export AGENT_ENDPOINT=local-adapter
python3 harness.py

Numbers leave the lab only after the bundle names the pack hash, the control sheet, and the weighted pass rate in one document. A later run that cannot reproduce the pass vector under the same hash is not a regression in the agent; it is a broken experiment. Publishing the percentage alone, without those three fields, is the agent-eval equivalent of posting p99 latency without the traffic shape. Readers should treat such a percentage as marketing copy even when the underlying model is strong.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A lab that needs a scratch box for the adapter and harness can use MonkeyCode's free model access and free server option, then keep the signed bundle and discard the machine. That pairing matters only as a place to execute the same frozen pack; it does not replace the hash, the weights, or the wall-clock bound. Claims about named models, quotas, hardware, or lasting performance are out of scope here because they are not part of the method.

The approach fails for agents whose job is live browsing, incident response on production traffic, or research over a corpus that cannot legally be snapshotted. It also fails for vendor bake-offs that refuse to pin an endpoint and still want a single winner by Friday. Teams without rights to ship fixtures and hidden tests should not pretend a public demo repository is a dataset. In those settings the honest artifact is a qualitative report, not a ranked table.

Limitations remain even when the pack is frozen. Graders can be wrong, weights can encode taste, and a zero temperature pin does not make a non-deterministic tool stack repeatable. The harness records timeouts as failures, which punishes slow correct answers and can hide partial progress that a human reviewer would salvage. Secondary axes such as median time and reported spend need their own units and missing-data rules, or they become decorative columns beside the headline.

The load-test analogy is the check against self-deception. Nobody ships an API percentile from a laptop on café wifi and calls it capacity planning, yet agent scores still travel that way in launch posts. Hash the pack, print the controls, weight the tasks, and bound the clock before the ranking is allowed to leave the repository. The number that survives that gate is smaller, duller, and finally comparable.