Four Fields That Make a Coding-Agent Score Citeable

You are in standup. A teammate says the new coding agent “clears 71 percent.” You ask 71 percent of what. Nobody has a file.

Someone pastes a screenshot from a vendor blog. The screenshot has a green bar, a model nickname, and no dataset hash. That number will leak into a roadmap slide by Friday. Treat it as marketing until a card travels with it.

A citeable coding-agent score is not a percentage. It is a percentage plus the dataset, the oracles, the metric family, and the resource envelope that produced it. If any of those four are missing, you do not have a measurement. You have a story that happens to contain digits.

What a marketing number looks like

You have seen the pattern. A public benchmark name. A single pass rate. No timeout. No token envelope. No note on whether the tasks already live in training corpora. The chart is clean because the protocol is invisible.

Public leaderboards still matter as conversation. They do not matter as procurement evidence. Named suites get trained against, coached against, and blog-optimized against. When the test is famous, the score is partly a memory test.

You need a local protocol you can rerun. The rest of this article is that protocol. It is a proposal you can execute on a laptop or a small server. It does not require you to trust a vendor chart.

The four objects on the card

Keep four objects. Refuse to quote a score if one is blank.

1. Dataset. A frozen task pack with content hashes, license notes, and difficulty strata.
2. Oracle. Commands that decide pass or fail without a human reading the agent’s essay.
3. Metric family. More than one number, and never a silent average across unequal envelopes.
4. Envelope. Wall-clock, step cap, and output cap, written down before the run starts.

Contamination posture sits beside those four. You will not prove a task is unseen. You can still record how you tried.

Step 1: Freeze a private task pack

Do not start from a leaderboard dump and call it private. Start from bugs you already shipped.

Pick 20 to 40 defects from your own tracker. Prefer ones with a failing test that already exists. If you must synthesize tasks, write the failing test first. Then freeze.

mkdir -p bench/tasks bench/oracles
git -C . rev-parse HEAD > bench/SOURCE_COMMIT
find bench/tasks -type f -print0 | sort -z | xargs -0 sha256sum > bench/DATASET.sha256
git add bench && git commit -m "freeze: coding-agent task pack"

Store one JSON line per task. Keep any gold patch out of the prompt the agent sees.

{"id":"INV-1843","lang":"py","strata":"medium","prompt_file":"tasks/INV-1843.md","oracle":"oracles/INV-1843.sh","timeout_s":120}

If you cannot hash it, you cannot cite it. A moving folder is not a dataset. It is a mood.

Step 2: Make the oracle a command, not a vibe

Each task needs a script that exits 0 on pass and non-zero on fail. The script must not read the agent’s chain of thought. It applies the patch, runs tests, and maybe a tiny linter you already own.

#!/usr/bin/env bash
set -euo pipefail
TASK_DIR="$1"
PATCH="$2"
cd "$TASK_DIR"
git apply --check "$PATCH"
git apply "$PATCH"
python -m pytest -q tests/test_invoice_tax.py

Label this as the contract. If the oracle needs a human, the metric is an annotation study. That is valid work. It is not a pass rate.

Step 3: Choose a metric family

One headline pass rate is how marketing happens. You want at least three printed numbers, each with a denominator.

• oracle_pass: tasks where the oracle exited 0.
• oracle_fail: tasks where the patch applied and tests failed.
• infra_fail: tasks where git, the network, or the runner died. Do not fold these into oracle_fail.
• strata_pass: oracle_pass split by easy / medium / hard.

You may add exact-match on a gold patch if you have one. Do not let that replace the oracle. Agents can fix the bug with a different diff.

The scorer below is a proposal, not a claim about any product. It refuses to emit a single average.

from collections import defaultdict

def summarize(rows):
    by_strata = defaultdict(lambda: {"n": 0, "pass": 0})
    infra = 0
    for r in rows:
        if r["status"] == "infra_fail":
            infra += 1
            continue
        bucket = by_strata[r["strata"]]
        bucket["n"] += 1
        bucket["pass"] += int(r["status"] == "oracle_pass")
    return {
        "infra_fail": infra,
        "strata": {
            name: {
                "n": vals["n"],
                "pass": vals["pass"],
                "rate": None if vals["n"] == 0 else round(vals["pass"] / vals["n"], 3),
            }
            for name, vals in sorted(by_strata.items())
        },
    }

Notice there is no mean of means. If someone later averages easy and hard, they are writing a new paper. They are not citing yours.

Step 4: Write the envelope before the first call

An agent with 40 steps and an agent with 8 steps are not the same method. Neither is an agent that may emit ten times the tokens. Put the cap in the card. Kill the run when it trips.

{
  "name": "invoice-tax-pack-2026-09",
  "dataset_sha256_file": "DATASET.sha256",
  "n_tasks": 24,
  "envelope": {
    "max_wall_s": 180,
    "max_steps": 12,
    "max_output_tokens": 8192
  },
  "metrics": ["oracle_pass", "infra_fail", "strata_pass"],
  "contamination": {
    "public_overlap_check": "manual filenames vs known suite lists",
    "training_guarantee": "none"
  },
  "print_policy": "refuse_headline_if_any_field_blank"
}

Save that as bench/card.json. If you raise the envelope after you see a disappointing rate, you started a new experiment. Rename the card. Do not overwrite the old one.

Step 5: Record contamination posture honestly

You will not get a cryptographic proof that a model never saw your bug. Say so.

Do three cheap checks anyway:

1. Search your prompt text against public benchmark names and famous function identifiers.
2. Keep proprietary identifiers in the fixtures so a memorized public snippet cannot pass.
3. Hold back a slice you never paste into a chat product, even “just to see.”

Write the outcome on the card: unchecked, filename_scan, or private_ids. Unchecked is allowed. Pretending you checked is not.

Step 6: Print the card, or print nothing

The script below refuses a headline if the card is incomplete. Run it after every batch. It uses only the standard library.

#!/usr/bin/env python3
"""Proposal: refuse a headline pass rate when the card is incomplete."""
from __future__ import annotations

import json
import sys
from collections import defaultdict
from pathlib import Path

REQUIRED = (
    "name",
    "dataset_sha256_file",
    "n_tasks",
    "envelope",
    "metrics",
    "contamination",
)

def summarize(rows):
    by_strata = defaultdict(lambda: {"n": 0, "pass": 0})
    infra = 0
    for r in rows:
        if r["status"] == "infra_fail":
            infra += 1
            continue
        bucket = by_strata[r["strata"]]
        bucket["n"] += 1
        bucket["pass"] += int(r["status"] == "oracle_pass")
    return {
        "infra_fail": infra,
        "strata": {
            name: {
                "n": vals["n"],
                "pass": vals["pass"],
                "rate": None if vals["n"] == 0 else round(vals["pass"] / vals["n"], 3),
            }
            for name, vals in sorted(by_strata.items())
        },
    }

def load_card(path: Path) -> dict:
    card = json.loads(path.read_text())
    missing = [k for k in REQUIRED if k not in card or card[k] in (None, "", [])]
    if missing:
        raise SystemExit(f"unprintable: missing {missing}")
    sha_file = path.parent / card["dataset_sha256_file"]
    if not sha_file.exists():
        raise SystemExit("unprintable: dataset hash file missing")
    env = card["envelope"]
    for key in ("max_wall_s", "max_steps", "max_output_tokens"):
        if key not in env:
            raise SystemExit(f"unprintable: envelope.{key} missing")
    return card

def main() -> None:
    root = Path(sys.argv[1] if len(sys.argv) > 1 else "bench")
    card = load_card(root / "card.json")
    result_path = root / "results.jsonl"
    if not result_path.exists():
        raise SystemExit("unprintable: results.jsonl missing")
    rows = [json.loads(line) for line in result_path.read_text().splitlines() if line.strip()]
    if len(rows) != card["n_tasks"]:
        raise SystemExit("unprintable: result count != n_tasks")
    summary = summarize(rows)
    print(json.dumps({"card": card["name"], "envelope": card["envelope"], **summary}, indent=2))

if __name__ == "__main__":
    main()

A tiny results.jsonl for a smoke test looks like this:

{"id":"INV-1843","strata":"medium","status":"oracle_pass"}
{"id":"INV-1844","strata":"hard","status":"oracle_fail"}
{"id":"INV-1845","strata":"easy","status":"infra_fail"}

Wire the scorer as a command, not a ritual:

chmod +x score_card.py
python3 score_card.py bench > bench/SCORE.json
test -s bench/SCORE.json

If the command exits non-zero, you have nothing to paste into Slack. That is the point.

Decision table: cite or do not cite

Situation	Citeable?	What you may say
Pass rate only, no card	No	“We do not have a measurement yet.”
Card complete, infra_fail folded into fail	No	“Rerun: infra is unlabeled.”
Card complete, strata hidden, only a mean	No	“Publish the strata or do not publish.”
Envelope raised after seeing results	No	“New experiment. New card name.”
Card complete, strata + infra + envelope present	Yes	Quote rates with n, envelope, and hash.
Public-benchmark name with no overlap note	No	“That is a conversation, not a citation.”

Tape this table next to the dashboard. The dashboard will try to grow a single tile. Do not let it.

Run the protocol where you can afford to rerun it

A protocol you cannot afford to rerun will rot into a one-off demo. That is how vendor charts get cited. You need a boring place to execute the oracle, keep the frozen pack, and call a model.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you do not already have an inference stack, MonkeyCode’s free model access and free server option are one way to exercise this card without first procuring GPUs. The card still lives in your repo. The oracles still belong to you. The product is only a runner you can actually repeat.

Do not treat a free runner as a published lab. Same envelope. Same hashes. Same refusal to print a naked percentage.

Limitations

This protocol does not estimate statistical power. Twenty-four tasks will not rank two agents. It will tell you whether your own card is honest.

It does not detect training-data leakage beyond cheap filename and identifier checks. Anyone who claims otherwise is selling comfort.

It does not score partial credit, human preference, or security impact. If those are your questions, design those oracles. Do not overload pytest until it lies.

The Python above is a proposal. It is not a result from a production eval run for this article. Copy it, then replace summarize with the metrics your oracles actually emit.

Who should not use this

Skip this if you are writing a launch tweet. A card slows you down. That is a feature for measurement and a bug for announcements.

Skip this if your tasks have no automatic oracle. You need an annotation protocol, not a bash exit code.

Skip this if you need a legally defensible comparison for procurement. Hire an eval team, pre-register the card, and size the task pack. This article is a hygiene kit, not a lab accreditation.

What you do Monday

Open the last agent score you quoted. Ask for the dataset hash, the oracle command, the strata table, and the envelope. If those four files do not exist, the number was marketing.

Write the card first. Then, and only then, quote a rate.