Sign the Metric Before You Publish an Agent Score

A headline agent score is not a measurement until the metric function is frozen, hashed, and tested against golden fixtures. Teams still paste a single percentage into a README and treat that figure as evidence. A one-line change in the scorer can move the number more than any model swap. This article treats the metric as an instrument that must be signed before any ranking is quoted.

Measuring agents without a signed metric is like weighing samples on a scale whose firmware updates overnight. The reading looks precise and the units look familiar, yet the instrument is a different object every morning. Freezing task files helps, but it does not bind the function that turns traces into points. When that function drifts, yesterday’s ranking and today’s ranking are not comparable, even with identical task bytes.

A usable dataset for this protocol is a directory of task cards with stable identifiers and content hashes, not a living spreadsheet. Each card should name inputs, oracles, and a timeout that is recorded rather than silently extended after a slow run. The split between probe items and evaluation items belongs in the same digest, because a moved row is a new experiment. The commands below hash that tree so a published number can point at one object and no other.

# content-address the evaluation pack; do not edit files after this digest is quoted
find eval_pack -type f -print0 | sort -z | xargs -0 sha256sum > eval_pack.sums
sha256sum eval_pack.sums > eval_pack.tree.sha256
cat eval_pack.tree.sha256

Metrics need a contract: a versioned function, a fixture suite, and an interval estimate instead of a naked mean. The contract states which fields of a trace count and how missing patches, crashed shells, or truncated logs are scored. Golden fixtures are tiny traces with known scores, and they fail the build if a refactor of the scorer changes them. Bootstrap intervals then show whether a claimed gap is larger than sampling noise on the same frozen pack.

# metric_contract.py — labeled sketch for a frozen scorer, not a vendor result
from __future__ import annotations

import hashlib, json, random
from pathlib import Path
from typing import Callable

CONTRACT_ID = "agent-metric/v3-patch-apply"
PASS, FAIL, INVALID = 1.0, 0.0, float("nan")

def digest_source(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def score_trace(trace: dict) -> float:
    if trace.get("schema") != "trace.v1":
        return INVALID
    if trace.get("exit_code") not in (0, 1):
        return INVALID
    if not trace.get("patch"):
        return FAIL
    oracle = set(trace.get("oracle_files") or [])
    touched = set(trace.get("touched_files") or [])
    tests = trace.get("tests_passed")
    if oracle and not oracle.issubset(touched):
        return FAIL
    return PASS if tests is True else FAIL

FIXTURES = [
    ({"schema": "trace.v1", "exit_code": 0, "patch": "diff", "oracle_files": ["a.py"],
      "touched_files": ["a.py"], "tests_passed": True}, PASS),
    ({"schema": "trace.v1", "exit_code": 0, "patch": "", "oracle_files": ["a.py"],
      "touched_files": [], "tests_passed": False}, FAIL),
    ({"schema": "trace.v2", "exit_code": 0, "patch": "diff", "oracle_files": [],
      "touched_files": [], "tests_passed": True}, INVALID),
]

def assert_fixtures(fn: Callable[[dict], float]) -> None:
    for i, (trace, expected) in enumerate(FIXTURES):
        got = fn(trace)
        if got != expected and not (got != got and expected != expected):
            raise AssertionError(f"fixture {i}: expected {expected!r}, got {got!r}")

def valid_scores(rows: list[float]) -> list[float]:
    return [x for x in rows if x == x]

def bootstrap_mean_ci(rows: list[float], n: int = 2000, alpha: float = 0.05, seed: int = 7) -> dict:
    rng = random.Random(seed)
    xs = valid_scores(rows)
    if len(xs) < 8:
        return {"n_valid": len(xs), "mean": None, "ci95": None, "note": "too few valid runs"}
    means = []
    for _ in range(n):
        sample = [xs[rng.randrange(len(xs))] for _ in xs]
        means.append(sum(sample) / len(sample))
    means.sort()
    lo = means[int(alpha / 2 * n)]
    hi = means[int((1 - alpha / 2) * n)]
    return {
        "n_valid": len(xs),
        "n_invalid": len(rows) - len(xs),
        "mean": sum(xs) / len(xs),
        "ci95": [lo, hi],
        "seed": seed,
        "resamples": n,
    }

if __name__ == "__main__":
    assert_fixtures(score_trace)
    print(json.dumps({
        "contract_id": CONTRACT_ID,
        "metric_sha256": digest_source(Path(__file__)),
        "fixture_count": len(FIXTURES),
    }, indent=2))

Controls keep the instrument honest without turning the write-up into a brochure. A stub agent that returns empty patches should sit near the floor if the metric actually penalizes missing work. A shuffled-oracle pass should collapse toward chance if filenames or ordering leak the answer into the scorer. Seed locks and environment fingerprints sit beside the score so a later reader can see whether the runtime or the trace schema changed. None of these controls prove a product is better; they only show that the number was produced under a named procedure.

# controls.py — labeled sketch: stub agent and shuffled oracles as scorer sanity checks
import json, os, platform, random, subprocess, sys
from pathlib import Path
from metric_contract import score_trace, bootstrap_mean_ci, CONTRACT_ID, digest_source

def fingerprint() -> dict:
    py = subprocess.check_output([sys.executable, "-c", "import sys; print(sys.version)"], text=True).strip()
    return {
        "python": py,
        "platform": platform.platform(),
        "cwd": str(Path.cwd()),
        "metric_sha256": digest_source(Path("metric_contract.py")),
        "tree_sha256": Path("eval_pack.tree.sha256").read_text().split()[0] if Path("eval_pack.tree.sha256").exists() else None,
    }

def stub_agent(task: dict) -> dict:
    return {
        "schema": "trace.v1",
        "exit_code": 1,
        "patch": "",
        "oracle_files": task.get("oracle_files", []),
        "touched_files": [],
        "tests_passed": False,
        "agent": "stub.empty_patch",
    }

def shuffle_oracles(tasks: list[dict], seed: int = 13) -> list[dict]:
    rng = random.Random(seed)
    files = [t.get("oracle_files", []) for t in tasks]
    rng.shuffle(files)
    out = []
    for task, oracle in zip(tasks, files):
        item = dict(task)
        item["oracle_files"] = oracle
        out.append(item)
    return out

def run_pack(tasks: list[dict], agent) -> list[float]:
    return [score_trace(agent(t)) for t in tasks]

if __name__ == "__main__":
    tasks = json.loads(Path("eval_pack/tasks.json").read_text())
    stub_rows = run_pack(tasks, stub_agent)
    shuffled = shuffle_oracles(tasks)
    report = {
        "contract_id": CONTRACT_ID,
        "env": fingerprint(),
        "stub_empty_patch": bootstrap_mean_ci(stub_rows),
        "shuffled_oracles_on_stub": bootstrap_mean_ci(run_pack(shuffled, stub_agent)),
        "seed_lock": {"pythonhashseed": os.environ.get("PYTHONHASHSEED")},
    }
    Path("score_report.json").write_text(json.dumps(report, indent=2, default=str))
    print(json.dumps(report, indent=2, default=str))

export PYTHONHASHSEED=0
python metric_contract.py
python controls.py
# quote score_report.json as a whole; do not lift only the mean field into a title

The report is the artifact, and the mean is only one field inside it. A serious write-up quotes the contract identifier, the metric digest, the tree digest, the valid-run count, the invalid-run count, and the interval. If two agents overlap inside that interval on the same pack, the honest sentence is that the pack cannot separate them yet. Stretching a 0.04 gap into a crown graphic is how a measurement turns into marketing without any change in the underlying traces.

Reproducing that report is cheaper when the harness can run in a disposable workspace that already exposes models and a server, rather than on a laptop that silently differs from last week. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option can host the same frozen pack and the same signed scorer so a reviewer reruns metric_contract.py and controls.py without rebuilding the instrument by hand. The product is not the metric, and a convenience host does not enlarge a confidence interval or repair a leaking oracle.

Limitations follow from the statistics, not from taste. Bootstrap intervals treat tasks as roughly exchangeable, while agent packs are often clustered by repository, language, or bug class, which narrows the real uncertainty. Golden fixtures catch scorer regressions, but they cannot certify that partial credit matches a human maintainer’s judgment on messy diffs. Invalid runs must stay visible; dropping crashes until the mean looks tidy is another quiet rescoring. Wall-clock, token use, and retry counts belong in the report as context, yet they are not a substitute for the signed success function.

Teams that need regulatory evidence, customer acceptance, or safety certification should not treat this sketch as a qualification method. Groups that cannot freeze the pack, or that let prompt text and hidden tools change between runs, will hash a moving target and then argue about ghosts. Writers who only want a single percentage for a launch post will find the extra fields inconvenient, which is the point of putting them in the file. A ranking that cannot name its instrument is not a ranking that later engineers can refute.

The core conclusion stays small on purpose. Publish the contract, the digests, the fixtures, and the interval, then let the mean sit in the middle of that paragraph. Readers who rerun the stub and the shuffle already know whether the scorer still behaves like yesterday’s instrument. That habit is slower than a leaderboard screenshot, and it is the difference between a number that can be cited and a number that can only be advertised. If the signed report is reproduced in a shared workspace, the useful next step is to keep the pack and the scorer in lockstep rather than to chase a prettier headline.