Refuse Every Agent Score That Ships Without a Protocol Card

You drop a coding-agent eval into Slack on a Monday. The screenshot says 81 percent. Product wants that number in the launch post. Then a reviewer asks three questions the screenshot cannot answer: which task files ran, which metric produced 81, and whether the agent was allowed to edit the tests.

That gap is the whole problem. A score without a protocol is a press release. You need a card another engineer can replay on a different machine without asking you what you meant.

This is not a leaderboard tutorial. It is a method for turning an eval into a signed artifact. You will walk out with a protocol card, a validator that refuses incomplete cards, and a decision table for metrics that marketing cannot collapse into one headline.

The number is not the result

An agent score is a function of the dataset, the metric family, and the controls. Change any one of those and the number moves. If you cannot name all three, you do not have a result. You have a vibe.

Write the function down first. Run it second. Publish both, or publish neither.

The rest of this article is that function, as files you can check into git.

What belongs on a protocol card

A protocol card is a small YAML document that must be complete before any score printer is allowed to run. Keep it next to the suite. Version it with the suite. Treat a missing field as a failed eval, not as a TODO.

Here is a starter card. Label it as a template until you fill every field from your own repo.

# protocol_card.yaml — template, fill before any score is emitted
schema: agent-eval-protocol/v1
suite:
  name: intern-fix-suite
  git_sha: REPLACE_ME
  fixture_hash: REPLACE_ME   # sha256 of the frozen task tarball
  license: REPLACE_ME
  source: REPLACE_ME         # where the tasks came from
  contamination_note: REPLACE_ME
  task_count: 0
  taxonomy:
    bugfix: 0
    refactor: 0
    test-authoring: 0
dataset_split:
  train_like: []             # tasks the agent may have seen in prompts or docs
  holdout: []                # the only tasks that may appear in the published score
metrics:
  primary: REPLACE_ME        # one name, see decision table below
  secondary: []
  headline_scalar_allowed: false
controls:
  runtime: python-3.12
  lockfile: poetry.lock
  tool_allowlist: [read, edit, bash]
  test_command: pytest -q
  wall_clock_s: 180
  max_tool_calls: 40
  temperature: 0.0
  seed: 7
  human_intervention: none
  parallel_jobs: 1
environment:
  os: linux
  notes: "no network except the model endpoint"
publishing:
  negative_results: required
  confound_log: required

If a field still says REPLACE_ME, you do not have a benchmark. You have a draft.

1. Dataset: provenance, not a folder named tasks/

Name the origin of every task. A scraped GitHub issue, an internal bug, a synthetic fixture, and a homework prompt are not interchangeable. They leak into the model at different rates and they punish different failure modes.

Hash the fixture tarball. Record the license. Write one sentence about contamination: could this task appear in public pretraining, in your own docs, or in yesterday's prompt traces? If you cannot answer, put the task in train_like and keep it out of the published score.

Holdout is a list of task ids, not a feeling. The published number may only include holdout. Everything else is calibration.

2. Metrics: pick a family, then refuse the scalar

One percentage is how a launch post is written. It is not how an eval is read. Choose a primary metric that matches the decision you actually need to make, then keep two secondary metrics so a single lucky slice cannot dominate.

Use this table. Do not average the rows.

Decision you need	Primary metric	Required secondary	Do not report as
Can the agent land a fix at all?	holdout pass@1 with frozen tests	patch size in changed lines	"accuracy"
Is the agent cheaper than a human on this slice?	tokens + wall clock per passing task	human-minutes estimate, labeled as estimate	"10x productivity"
Does it stop touching tests?	share of runs that leave tests/ byte-identical	count of test-file diffs	"robustness"
Does it finish under the budget?	share of runs that exit before wall_clock_s	timeout vs tool-cap vs model error	"reliability"
Can another team replay it?	validator exit code on the protocol card	git sha + fixture hash	a screenshot

Set headline_scalar_allowed: false until you have at least one row from that table filled with real fields. If someone later wants a single number for a slide, they can derive it from the card in public. They cannot replace the card.

3. Controls: the environment is part of the score

Pin the runtime, the lockfile, the tool allowlist, the test command, the wall clock, the tool-call cap, temperature, seed, and the human-intervention policy. If a human may nudge the agent, the card must say so, and the run is no longer unattended.

Parallelism belongs on the card because hidden retries show up as extra jobs. Network policy belongs on the card because a suite that can pip install mid-run is not the suite you hashed.

A validator that refuses incomplete cards

Do not trust yourself to notice a blank field at 11pm. Make the score printer exit non-zero. The script below is a proposal you can copy into scripts/validate_protocol.py and run in CI.

#!/usr/bin/env python3
"""Refuse to print an agent score unless protocol_card.yaml is complete."""
from __future__ import annotations

import hashlib
import sys
from pathlib import Path

import yaml

REQUIRED = [
    ("suite", "git_sha"),
    ("suite", "fixture_hash"),
    ("suite", "license"),
    ("suite", "source"),
    ("suite", "contamination_note"),
    ("metrics", "primary"),
    ("controls", "test_command"),
    ("controls", "wall_clock_s"),
    ("controls", "tool_allowlist"),
    ("controls", "human_intervention"),
]
PLACEHOLDERS = {"", "REPLACE_ME", None}


def nested(doc, *keys):
    cur = doc
    for k in keys:
        if not isinstance(cur, dict) or k not in cur:
            return None
        cur = cur[k]
    return cur


def main(card_path: Path, fixture: Path | None) -> int:
    doc = yaml.safe_load(card_path.read_text())
    errors: list[str] = []

    for path in REQUIRED:
        val = nested(doc, *path)
        if val in PLACEHOLDERS or val == [] or val == 0:
            errors.append(".".join(path) + " is empty")

    holdout = nested(doc, "dataset_split", "holdout") or []
    if not holdout:
        errors.append("dataset_split.holdout must list task ids")

    if nested(doc, "metrics", "headline_scalar_allowed") is True:
        errors.append("headline_scalar_allowed must stay false for publish")

    if nested(doc, "controls", "human_intervention") not in {"none", "logged"}:
        errors.append("human_intervention must be none or logged")

    if fixture and fixture.exists():
        digest = hashlib.sha256(fixture.read_bytes()).hexdigest()
        listed = nested(doc, "suite", "fixture_hash")
        if listed != digest:
            errors.append(f"fixture_hash mismatch: card={listed} file={digest}")

    if errors:
        print("protocol card invalid:")
        for e in errors:
            print(f"  - {e}")
        return 1

    print("protocol card ok")
    print(f"primary metric: {doc['metrics']['primary']}")
    print(f"holdout tasks: {len(holdout)}")
    return 0


if __name__ == "__main__":
    card = Path(sys.argv[1] if len(sys.argv) > 1 else "protocol_card.yaml")
    tarball = Path(sys.argv[2]) if len(sys.argv) > 2 else None
    raise SystemExit(main(card, tarball))

Run it before any aggregation script:

python scripts/validate_protocol.py protocol_card.yaml fixtures/intern-fix-suite.tar.gz
# expected on a blank template:
# protocol card invalid:
#   - suite.git_sha is empty
#   - ...

Wire the same command into CI so a pull request that adds a score but not a card cannot merge. The failure message is the review.

Numbered workflow you can run this week

1. Freeze the task tarball. tar -C suites/intern-fix -cf fixtures/intern-fix-suite.tar.gz . then hash it. Put the hash on the card.
2. Split ids into train_like and holdout. If a task appeared in a blog, a system prompt, or a fine-tune mix, it is not holdout.
3. Pick one primary metric from the table. Write the exact command that computes it. Example: pytest -q must be the same string the agent is scored against.
4. Fill controls until the validator is quiet. Then, and only then, run the agent against holdout.
5. Store per-task rows: task id, primary, secondaries, exit reason, tool-call count, wall clock, patch hash. Do not reduce those rows until the card is in the same commit.
6. Publish the card, the row file, and the negative results. If half the holdout timed out, that is the result.

A replay then looks like this:

git checkout $GIT_SHA
python scripts/validate_protocol.py protocol_card.yaml fixtures/intern-fix-suite.tar.gz
# run your agent harness with controls copied from the card
# write rows.jsonl next to the card

If a second machine cannot get the validator to print protocol card ok, the original score is not replayable. Do not argue about the percentage. Fix the card.

Why this stops a marketing number

Marketing needs a scalar, a comparison, and a fade-out of the setup. A protocol card attacks all three. The scalar is banned at publish time. The comparison has to name the holdout and the controls. The setup cannot fade out because the validator will not start.

You still can lie, of course. You can put a decorative hash on the card. You can hide retries as parallel_jobs. That is why the confound log is required. Write every deviation in the same commit as the rows. Reviewers should read that file before they read the percentage.

A useful smell test: if deleting the protocol card would not change how you talk about the agent, you were never benchmarking.

Where a shared box fits

You can run this entire protocol on a laptop. Many teams should. The card does not care where the process lived.

If you need a disposable environment so the lockfile, the tool allowlist, and the wall clock are not mixed with your daily shell, a hosted box helps. MonkeyCode is an open-source coding environment with free model access and a free server option, which is enough to check out the card, run the validator, and keep the suite off your laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it as an execution host for the protocol, not as a substitute for the protocol. The product does not fill REPLACE_ME for you.

If you try that path, export the same protocol_card.yaml and rows.jsonl you would have produced locally. The artifact is the card plus the rows. The host is incidental.

Limitations

This method does not tell you which model is better. It tells you whether a number is admissible. A perfect card around a 12-task toy suite is still a toy suite.

The validator only checks presence, a few enums, and an optional hash. It cannot detect that your holdout leaked into a prompt last Thursday. It cannot detect that pytest -q was swapped for a weaker command after the hash was recorded. Those are process failures. Review them like you review production incidents.

YAML is not cryptography. A signed git tag on the card-plus-fixtures commit is stronger than a file sitting in a gist. If you need external audit, tag the commit and publish the tag URL next to the rows.

Token costs, hardware, model names, and quotas are out of scope here on purpose. They change. The card fields should not.

Who should not use this

Skip this if you are demoing an agent to yourself for ten minutes. A protocol card is overhead when the only reader is you and the only decision is "does this even boot."

Skip this if you do not own the suite. You cannot honestly fill source, license, or contamination_note for a benchmark you do not have rights to redistribute. Find another suite or do not publish.

Skip this if your org wants a weekly leaderboard more than it wants replay. The validator will look like bureaucracy, and people will route around it with screenshots. In that culture the card will rot, and a rotting card is worse than no card because it implies a rigor you did not perform.

If you do own a small holdout, you can fill the template in an afternoon. Start with twelve tasks, one primary metric, and a validator in CI. The first useful output is not a percentage. It is a commit where the card and the rows cannot drift apart.