Freeze the Retry Budget Before You Trust an Agent Pass Rate

You finish a Saturday bake-off between two coding agents on the same five tickets. Agent A lands 4/5. Agent B lands 2/5. You screenshot the table, write “clear winner,” and close the laptop.

Monday you replay the traces. Agent A called the test runner eleven times, edited the same file six times, and burned a quiet pile of tokens after the first failure. Agent B stopped after one compile error and asked for a missing fixture. Same tasks. Same model family. Different retry policy. The leaderboard was a loop counter wearing a lab coat.

That is the failure this protocol is built to stop. You already know to freeze the oracle. You still need to freeze the budget that feeds the oracle.

What a number has to survive

A pass rate is marketing when a reader cannot reconstruct the run. Reconstruction needs four frozen pieces: the task pack, the grader, the interaction budget, and the clock. Miss any one and two honest engineers can publish opposite rankings from the same model.

You do not need a giant public leaderboard. You need a folder you can hash, a script you can rerun, and a results file that refuses to print a score unless the controls are present.

This article gives you that folder. The code is a protocol, not a trophy. Do not treat the tiny sample tasks as a ranking of any product.

The four controls

Name them before you write a single adapter.

1. Dataset. A JSONL pack with stable task_id values, a prompt, a hidden test command, and a SHA-256 of the pack. If a prompt changes, the hash changes. No silent edits.
2. Oracle. The grader binary or script is checked into the same tree. You do not “eyeball” diffs. You run the command recorded on the task.
3. Retry / tool budget. Max tool calls, max file-edit rounds, max tokens, max wall time. These are independent variables, not implementation details you hide in a framework default.
4. Environment. Same language runtime, same packages, no network unless the task says so. Record python --version and a lockfile hash.

If your write-up omits (3), you published a retry contest. If it omits (1) and (2), you published a vibe check.

A frozen mini pack you can actually inspect

Keep the first pack embarrassing and small. Three tasks. Each one is designed to punish a different failure mode: a missing import, an off-by-one, and a fabricated helper that does not exist.

{"task_id":"t01_missing_import","prompt":"Write sum_csv(path) that returns the sum of column n in a CSV with headers n,v. Do not invent extra helpers.","test_cmd":"python -m pytest tests/test_t01.py -q","must_not_contain":["fetch_csv_from_s3"]}
{"task_id":"t02_off_by_one","prompt":"Implement clamp_index(i, n) so i is kept in [0, n). Reject n <= 0.","test_cmd":"python -m pytest tests/test_t02.py -q","must_not_contain":[]}
{"task_id":"t03_no_ghost_api","prompt":"Parse a .env file into dict[str,str]. Use only the stdlib. If a line is illegal, raise ValueError.","test_cmd":"python -m pytest tests/test_t03.py -q","must_not_contain":["dotenv.load_dotenv","requests.get"]}

Hash the file. Put the digest in the results header. If someone “improves” a prompt after seeing failures, the digest will not match and the old score is void.

sha256sum tasks.jsonl
python --version

Label the tests as a proposal pack. They are not a public standard. They exist so you can practice the controls before you bring in real tickets.

Outcome taxonomy: stop collapsing everything into pass/fail

A binary pass rate hides why the agent spent the budget. Classify every task into one bucket. Do not invent extra buckets on the fly mid-study.

Code	Meaning	Counts as pass?
PASS	Hidden tests exited 0 inside budget	yes
FAIL_TEST	Tests ran, at least one assertion died	no
FAIL_ASSUME	Output referenced a banned symbol or a helper you never provided	no
EXHAUSTED	Hit tool, token, or wall-clock cap before a green test	no
ERROR	Adapter crash, timeout at the OS layer, or missing artifact	no

FAIL_ASSUME is the one most weekend bake-offs never count. An agent that invents fetch_csv_from_s3 can look “productive” in a chat log and still be wrong. You want that visible next to the pass rate, not folded into a shrug.

The harness

The runner below is intentionally boring. It does not talk to a vendor SDK. It talks to a function you inject: propose_edit(task, history) -> {files, tool_name}. Swap that function for a local script, an HTTP wrapper, or a no-op stub while you debug the bookkeeping.

# agent_bench.py — protocol harness, not a product score
from __future__ import annotations

import hashlib, json, os, subprocess, time, traceback
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Callable, Dict, List

@dataclass(frozen=True)
class Budget:
    max_tool_calls: int = 4
    max_tokens: int = 8000  # adapter-reported; 0 if unknown
    max_seconds: float = 90.0

@dataclass
class Row:
    task_id: str
    outcome: str
    tool_calls: int
    seconds: float
    tokens: int
    note: str

def pack_hash(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()

def run_oracle(test_cmd: str, cwd: Path, remaining: float) -> int:
    proc = subprocess.run(
        test_cmd, shell=True, cwd=cwd,
        capture_output=True, text=True, timeout=max(5.0, remaining),
    )
    return proc.returncode

def evaluate(
    tasks_path: Path,
    workdir: Path,
    propose: Callable[[dict, List[dict]], dict],
    budget: Budget,
) -> Dict:
    rows: List[Row] = []
    digest = pack_hash(tasks_path)
    for line in tasks_path.read_text().splitlines():
        task = json.loads(line)
        history: List[dict] = []
        tokens = 0
        t0 = time.monotonic()
        outcome, note = "ERROR", ""
        try:
            for step in range(budget.max_tool_calls):
                elapsed = time.monotonic() - t0
                if elapsed >= budget.max_seconds:
                    outcome, note = "EXHAUSTED", "wall_clock"
                    break
                if budget.max_tokens and tokens >= budget.max_tokens:
                    outcome, note = "EXHAUSTED", "tokens"
                    break
                proposal = propose(task, history)
                tokens += int(proposal.get("tokens", 0))
                history.append(proposal)
                banned = task.get("must_not_contain") or []
                blob = json.dumps(proposal)
                if any(s in blob for s in banned):
                    outcome, note = "FAIL_ASSUME", "banned_symbol"
                    break
                for rel, src in proposal.get("files", {}).items():
                    dest = workdir / rel
                    dest.parent.mkdir(parents=True, exist_ok=True)
                    dest.write_text(src)
                remaining = budget.max_seconds - (time.monotonic() - t0)
                rc = run_oracle(task["test_cmd"], workdir, remaining)
                if rc == 0:
                    outcome, note = "PASS", f"step={step+1}"
                    break
            else:
                if outcome == "ERROR":
                    outcome, note = "EXHAUSTED", "tool_calls"
            if outcome == "ERROR":
                # last oracle run failed inside budget
                outcome = "FAIL_TEST"
                note = note or "assertions"
        except subprocess.TimeoutExpired:
            outcome, note = "EXHAUSTED", "oracle_timeout"
        except Exception:
            outcome, note = "ERROR", traceback.format_exc(limit=1)
        rows.append(Row(
            task_id=task["task_id"], outcome=outcome,
            tool_calls=len(history),
            seconds=round(time.monotonic() - t0, 3),
            tokens=tokens, note=note,
        ))
    return {
        "pack_sha256": digest,
        "budget": asdict(budget),
        "python": os.popen("python --version").read().strip(),
        "rows": [asdict(r) for r in rows],
    }

def summarize(payload: Dict) -> Dict[str, float]:
    rows = payload["rows"]
    n = max(len(rows), 1)
    def rate(code: str) -> float:
        return round(sum(1 for r in rows if r["outcome"] == code) / n, 3)
    return {
        "n": len(rows),
        "pass_rate": rate("PASS"),
        "assume_rate": rate("FAIL_ASSUME"),
        "exhaust_rate": rate("EXHAUSTED"),
        "mean_tool_calls": round(sum(r["tool_calls"] for r in rows) / n, 3),
    }

Notice what the harness will not do. It will not retry outside max_tool_calls. It will not “helpfully” bump the wall clock because a run felt close. It will not drop FAIL_ASSUME into FAIL_TEST. Those refusals are the product.

Stub the agent while you test the bookkeeping:

def silent_agent(task, history):
    return {"files": {}, "tokens": 0, "tool_name": "noop"}

if __name__ == "__main__":
    payload = evaluate(Path("tasks.jsonl"), Path("work"), silent_agent, Budget())
    print(json.dumps({"controls": payload["budget"], "hash": payload["pack_sha256"],
                      "summary": summarize(payload), "rows": payload["rows"]}, indent=2))

You should see EXHAUSTED / tool_calls across the board. That is a successful control test. If a silent agent can still print a pretty pass rate, the harness is lying.

How to report so the table cannot pose as a brand ad

Print the controls above the score. Always. A reader who only sees 0.80 has been marketed to.

pack_sha256:  <digest>
budget:       tool_calls=4  tokens=8000  seconds=90
env:          python 3.x  + lockfile sha
n=3           pass=0.00  assume=0.00  exhaust=1.00  mean_tools=4.0

Then run the same pack at two more budgets, for example 2 and 8 tool calls. If the ranking flips when the budget changes, you have found the real story: the agent is budget-sensitive. Publish the flip. Do not pick the budget that flatters the tool you like.

A decision table keeps you honest when someone asks for “the number.”

Question you want to answer	Budget to freeze	Metric to lead with
Does it solve the ticket at all?	generous tool + time cap	pass rate + exhaust rate
Does it stop inventing APIs?	modest tool cap	assume rate
Can a free-tier box finish overnight?	tight wall clock	exhaust rate by wall_clock
Is Agent A actually better than B?	identical budgets and pack hash	pass rate and mean tool calls

If you cannot fill the middle column, you are not ready to publish the right column.

Running it where the meter is quiet

You want the harness to sit on a cheap always-on box so overnight sweeps do not depend on your laptop lid. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding-agent project with operator-supplied free model access and a free server option, which is enough to park this runner and an adapter without turning the protocol into a paid bake-off.

Use that box as an execution venue, not as a source of magic numbers. Do not paste a vendor dashboard into the paper. Copy the JSON the harness emitted. If the adapter cannot report tokens, set tokens to 0 and say so in the report. Missing token accounting is a limitation, not a rounding error you hide.

A minimal remote loop looks like this:

python agent_bench.py > results_budget4.json
# change Budget(max_tool_calls=8) and rerun
python agent_bench.py > results_budget8.json
python - <<'PY'
import json
from pathlib import Path
for p in sorted(Path('.').glob('results_budget*.json')):
    print(p.name, json.loads(p.read_text()).get('summary'))
PY

If you later swap adapters, keep the pack hash and the Budget dataclass identical. That is the entire point of the protocol. A new model on a mutated pack is a new experiment, not a continuation.

Limitations

This harness does not measure code quality, security, or review cost. A PASS can still be an unreadable patch. Hidden tests on three synthetic tasks will overfit any agent you iterate against them; rotate the pack or keep a held-out slice.

Token counts are only as good as the adapter. Some wrappers silently omit them. Wall-clock includes your disk and the pytest collection time, so do not compare a cold box to a warm one without saying so.

The banned-symbol check is a string needle. It will miss clever aliases. It will also false-positive if your own tests mention the banned name in an error string that gets logged into the proposal blob. Keep grader output out of the proposal.

None of these numbers are a company benchmark. They are a way to stop you from ranking retry policies.

Who should not use this

Skip this protocol if you are grading open-ended design work with no hidden tests. Skip it if your agent is allowed unbounded human-in-the-loop edits; the “budget” then lives in the human, and the JSON will pretend otherwise. Skip it if you need statistically tight comparisons across dozens of models—the pack is too small, and you will need a proper paired design on a frozen, larger set.

Use it when you are about to post a pass rate from a free-tier coding agent and you want the post to still make sense after someone asks, “How many times did it get to retry?”

Freeze the pack. Freeze the grader. Freeze the retry budget. Then, and only then, print the rate. If you need a quiet machine to run those three budget sweeps, MonkeyCode’s free server option is one place to leave the harness running while you read the traces instead of the marketing table.