Agent Scores Without a Null Pack Are Marketing

An agent percentage published without a versioned dataset, a locked metric function, and a passing null pack is marketing copy. Weekend evaluations often move because the judge prompt, the task mix, or the timeout changed, not because the agent improved. A measurement protocol therefore refuses to print a success rate until those three pieces are sealed and the negative controls behave. The small harness below treats that refusal as a hard gate rather than a polite style recommendation.

Public coding-agent numbers have started to resemble product landing pages, even when the underlying tasks are thin. Saturated unit tests and loosely written judges both inflate pass rates without showing whether the agent can miss on purpose. When a suite cannot produce a clean miss, the headline percentage is no longer a measurement; it is a selected screenshot. That failure mode matches a load test that never opens a connection to the real database.

A defensible agent benchmark is a sealed protocol, not a folder of prompts edited after the first chart appeared. The dataset hash, the metric implementation, the timeout budget, and the null pack travel together as one version. Changing any of those fields creates a new protocol identity, so older percentages cannot be compared as if they were the same exam. The useful analogy is pinning a compiler and a test runner before quoting a runtime score.

The null pack is the piece most leaderboards skip because it can kill a flattering number. It is a small set of tasks whose outcomes are known in advance: items that must pass, items that must fail, and items that must be refused rather than guessed. If a supposedly strong agent sails through poisoned files, or fails tasks a stub solver should finish, the run is invalid and no percentage is emitted. The pack is not extra credit; it is the instrument check that comes before the experiment.

The following protocol is a proposed, unexecuted example. It does not claim production results, vendor rankings, or model-specific scores. Teams can copy the files, seal them, and replace the solver command with whatever agent they already run locally.

# protocol.yaml — identity for one exam, not a leaderboard
schema: agent-bench-protocol/v1
protocol_id: coding-repair.nullpack.2026-09-18
dataset:
  path: ./tasks/repair_v3.jsonl
  sha256: "PENDING_SEAL"
metric:
  module: ./metric_exact_diff.py
  sha256: "PENDING_SEAL"
budget:
  timeout_sec: 90
  max_files_touched: 8
null_pack:
  path: ./tasks/null_pack_v1.jsonl
  sha256: "PENDING_SEAL"
  rules:
    must_pass: ["NP-STUB-01", "NP-STUB-02"]
    must_fail: ["NP-POISON-01", "NP-LEAK-02"]
    must_refuse: ["NP-SECRET-01"]
publish:
  min_n: 40
  require_ci: true
  forbidden_fields: ["single_percentage_only"]

Sealing replaces every PENDING_SEAL value with a digest and then refuses further edits without a new protocol_id. The command below is intentionally boring, because a benchmark that needs a clever seal is already drifting toward theater.

#!/usr/bin/env bash
# seal_protocol.sh — proposed example, run from the protocol directory
set -euo pipefail
sha256sum tasks/repair_v3.jsonl tasks/null_pack_v1.jsonl metric_exact_diff.py \
  | tee protocol.sums
python3 - <<'PY'
import hashlib, pathlib, re, sys
text = pathlib.Path("protocol.yaml").read_text()
sums = {}
for line in pathlib.Path("protocol.sums").read_text().splitlines():
    digest, name = line.split()
    sums[name] = digest
mapping = {
    "./tasks/repair_v3.jsonl": sums["tasks/repair_v3.jsonl"],
    "./metric_exact_diff.py": sums["metric_exact_diff.py"],
    "./tasks/null_pack_v1.jsonl": sums["tasks/null_pack_v1.jsonl"],
}
for path, digest in mapping.items():
    text = text.replace('sha256: "PENDING_SEAL"', f'sha256: "{digest}"', 1)
if "PENDING_SEAL" in text:
    sys.exit("seal incomplete")
pathlib.Path("protocol.sealed.yaml").write_text(text)
print("sealed protocol written")
PY

The metric module must be a function with its own digest, not a chat prompt that a reviewer rephrases between runs. Exact-diff scoring is deliberately unimpressive: it compares a normalized unified diff against an oracle patch and returns a boolean. Teams that want a language-model judge can still use one, but that judge becomes part of the sealed metric file, not a hidden sidebar comment. If the judge changes, the protocol identity changes, and yesterday's percentage is retired.

# metric_exact_diff.py — proposed example
from pathlib import Path

def normalize(diff: str) -> str:
    lines = []
    for line in diff.splitlines():
        if line.startswith(("index ", "diff --git", "---", "+++")):
            continue
        lines.append(line.rstrip())
    return "\n".join(lines)

def score(result_diff: str, oracle_path: str) -> dict:
    oracle = normalize(Path(oracle_path).read_text())
    got = normalize(result_diff)
    return {"pass": got == oracle, "metric": "exact_diff_v1"}

Null-pack records look like ordinary tasks, except each row carries an expected class instead of an open-ended success flag. must_pass rows are trivial repairs a stub can finish, which detects a broken runner. must_fail rows include poisoned instructions and leaked answers that a careful agent should reject. must_refuse rows mention secrets or out-of-scope files; a completion that writes those files fails the instrument check even if the rest of the suite looks green.

# run_protocol.py — proposed example; prints no headline until gates pass
import json, random, statistics, subprocess, sys, time, yaml
from pathlib import Path
from metric_exact_diff import score

def load_jsonl(path):
    rows = []
    with open(path) as handle:
        for line in handle:
            rows.append(json.loads(line))
    return rows

def run_solver(task, timeout):
    started = time.time()
    proc = subprocess.run(
        ["bash", "./solve.sh", task["id"]],
        capture_output=True, text=True, timeout=timeout,
    )
    return proc.stdout, time.time() - started, proc.returncode

def bootstrap_ci(flags, rounds=1000):
    means = []
    n = len(flags)
    for _ in range(rounds):
        sample = [flags[random.randrange(n)] for _ in range(n)]
        means.append(sum(sample) / n)
    means.sort()
    lo = means[int(0.025 * rounds)]
    hi = means[int(0.975 * rounds)]
    return lo, hi, statistics.mean(flags)

def main(protocol_path):
    proto = yaml.safe_load(Path(protocol_path).read_text())
    if "PENDING_SEAL" in Path(protocol_path).read_text():
        sys.exit("refuse: protocol is not sealed")
    null_rows = load_jsonl(proto["null_pack"]["path"])
    for row in null_rows:
        diff, _, _ = run_solver(row, proto["budget"]["timeout_sec"])
        passed = score(diff, row["oracle"])["pass"]
        expected = row["expected_class"]
        if expected == "must_pass" and not passed:
            sys.exit(f"null pack failed open: {row['id']}")
        if expected in {"must_fail", "must_refuse"} and passed:
            sys.exit(f"null pack failed closed: {row['id']}")
    tasks = load_jsonl(proto["dataset"]["path"])
    if len(tasks) < proto["publish"]["min_n"]:
        sys.exit("refuse: sample smaller than publish.min_n")
    flags = []
    for task in tasks:
        diff, elapsed, code = run_solver(task, proto["budget"]["timeout_sec"])
        flags.append(1.0 if score(diff, task["oracle"])["pass"] else 0.0)
    lo, hi, mean = bootstrap_ci(flags)
    report = {
        "protocol_id": proto["protocol_id"],
        "n": len(flags),
        "mean": round(mean, 4),
        "ci95": [round(lo, 4), round(hi, 4)],
        "null_pack": "pass",
    }
    Path("report.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report))

if __name__ == "__main__":
    main(sys.argv[1])

A team that wants a number for slides will dislike this runner, because the happy path is easy to block. That is the point. Marketing copy needs a single percentage with no denominator, no interval, and no story about the tasks that should have failed. A measurement needs the opposite: a protocol identity, a sample size, a confidence interval, and a null pack that actually bites. If the interval is wide, the honest publication is the interval, not a rounded headline.

Variance is not a footnote for later. Two agents can share a mean and still disagree on which files they break, which is why a protocol that only stores a percentage cannot be replayed. The report above keeps n and ci95 in the same object as protocol_id, so a later reader can see whether the exam was large enough to quote. Teams that need a ranking can still rank, but they rank sealed protocols against each other rather than mixing leftover prompts from different weeks.

Running the gate locally stays small on purpose. The solver is a shell entrypoint, the metric is a file, and the dataset is content-addressed. That shape fits a short-lived server as well as a laptop, which is where free model access can participate without becoming the subject of the exam. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host the sealed runner when a laptop is a poor place to keep timeouts honest; they do not replace the protocol, the null pack, or the interval. The measurement remains valid if those product pieces are removed.

Several limitations follow from the same design, and they are not cosmetic. Exact-diff metrics under-count valid alternate patches, so this harness is a conservative instrument, not a creativity contest. Bootstrap intervals assume the task rows are exchangeable; they are not a substitute for stratified sampling when the dataset is a pile of unrelated bugs. The null pack only detects the failures someone thought to encode, which means a clever leak outside those rows can still flatter a solver. Timeouts, network flakes, and non-deterministic tools can also fail the pack for reasons that have nothing to do with agent quality.

This approach is a poor fit for sales demos, for regulatory claims, and for any score that must be produced before the protocol can be sealed. It is also the wrong tool when the sample is smaller than the publish floor, when the oracle patches are themselves unreviewed, or when a team needs a qualitative read of design taste rather than a boolean metric. Those jobs want notebooks, design reviews, or hold-the-line tests of a different kind. They do not want a percentage that pretends to be an exam.

The core conclusion does not change after the code is copied. An agent score is a measurement only when the dataset, the metric, and the null pack share a version and the run can still fail closed. Numbers that cannot fail are advertisements with extra decimal places. Seal the protocol, watch the null pack, and publish the interval; the ranking, if it is still needed, can wait until those pieces exist.