Don't Average Pass Rates Across Unequal Token Budgets

You taped two terminal panes to a Friday review. The left pane showed Agent A with eight greens; the right pane showed Agent B with six. Someone asked which one to keep. You almost said A.

Then you scrolled the traces. Agent A retried until the cap cried. Agent B stopped after one compile and one test run. Same tasks. Different permission to spend. The ranking was a poster, not a measurement.

A pass rate answers one question: did the tests go green. It is silent on three others. How many tokens did the run burn before that green. Did a budget cap get scored as a model failure. Were the two agents even allowed the same tool-call envelope. If you cannot answer those, you do not have a benchmark. You have a vibe with a percentage sign.

This article is a protocol, not a leaderboard. You will pin a tiny task pack, declare a budget before the first call, write every attempt to a JSONL ledger, and publish three numbers together. The scripts below are a proposal you can copy. They are not a claim about any vendor eval, and they are not executed numbers from a production fleet.

Why the single percentage is usually marketing

Engineering reviews launder vibe-coding into a metric when the budget is a footnote. Two agents can share a task list and still live in different universes. One is allowed 40 tool calls and four repair loops. The other gets one shot and a 2k-token ceiling. Averaging their pass rates is like averaging two race times after giving one car a second tank of fuel.

You need the budget in the same table as the outcome. Not in a blog paragraph. In the file you commit.

The protocol below treats tokens, tool calls, and truncations as first-class results. Pass/fail is column four, not the headline.

What you will actually produce

Work in a single directory. Keep it boring.

1. A pinned task pack with SHA-256 hashes.
2. A budget envelope declared before any model call.
3. A JSONL ledger with one object per attempt, including the dull failures.
4. Three numbers you publish together, or you publish none of them: pass rate, truncation rate, and tokens-per-success.

You can run this on a laptop. You can also run it on a free server so the cost of being honest is not why the controls get skipped.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option matter here only as a lab: you can execute the ledger without folding a billed retry policy into the ranking. Strip the product name out and the method still stands. Free is still a cap. Write that cap in the same file as the tasks.

Step 1 — Pin a task pack you can hash

Do not point at "the latest suite on the internet." Snapshot files. Hash them. If a fixture changes, the run id changes. That is the whole point.

Proposed layout:

eval_pack/
  TASKS.md
  budget.yaml
  tasks/
    01_csv_summary/
      prompt.md
      src/app.py
      tests/test_app.py
    02_retry_header/
      prompt.md
      src/app.py
      tests/test_app.py
  hashes.sha256

Keep the pack small on purpose. Eight to twelve tasks is enough to debug the protocol. It is not enough to crown a winner. Label it that way in TASKS.md.

Each task needs three things: a prompt the agent may read, a hidden test file the agent must not edit, and a definition of done that is a command, not a feeling.

# tasks/01_csv_summary/prompt.md
Read src/app.py. Make summarize(path) return a dict with keys
rows, empty_rows, numeric_columns. Do not edit tests/.

Hash after you freeze the files:

cd eval_pack
find tasks -type f | sort | xargs sha256sum > hashes.sha256
sha256sum budget.yaml TASKS.md >> hashes.sha256
cat hashes.sha256

If hashes.sha256 drifts, you are not comparing agents. You are comparing days.

Step 2 — Declare the budget before the first call

Write the envelope first. Then refuse to start a run that does not load it. This is a control, not a comment.

# budget.yaml
envelope_id: "lab-2026-09-19-a"
max_input_tokens_per_task: 8000
max_output_tokens_per_task: 2000
max_tool_calls_per_task: 12
max_repair_loops: 2
wall_clock_seconds_per_task: 180
hidden_tests_are_read_only: true

Pick numbers that fit your lab. Do not copy these as if they were a standard. The standard is that both agents get the same envelope, and that truncation is recorded as its own outcome class.

Outcome classes you will allow:

• pass — hidden tests exit 0 inside the envelope.
• fail — tests exit non-zero, budget still remaining.
• truncated — a cap fired before tests could pass.
• invalid — the agent edited a forbidden path, or the runner crashed in a way you cannot attribute.

truncated is not fail. Mixing them is how a stingy envelope looks like a weak model.

Step 3 — Log every attempt, including the boring ones

JSONL beats a screenshot of a dashboard. One object per task attempt. No silent retries.

Proposed record (label: schema proposal):

{
  "run_id": "20260919T1400Z-lab-a",
  "envelope_id": "lab-2026-09-19-a",
  "task_id": "01_csv_summary",
  "agent_id": "agent-a",
  "pack_hash": "REPLACE_WITH_SHA256_OF_hashes.sha256",
  "input_tokens": 2410,
  "output_tokens": 880,
  "tool_calls": 5,
  "repair_loops": 1,
  "wall_clock_seconds": 47,
  "outcome": "pass",
  "test_exit_code": 0,
  "notes": ""
}

Append only. If a row is wrong, write a correction row. Do not silently edit history.

mkdir -p runs
touch runs/20260919T1400Z-lab-a.jsonl

Your runner should dump token counts from the provider response, not from a guess. Tokenizers differ. That is a limitation, not an excuse to skip the column.

Step 4 — Score three numbers, never one

Proposal, not a published score. After the JSONL exists, compute:

• pass_rate = passes / tasks
• truncation_rate = truncated / tasks
• tokens_per_success = sum(input_tokens + output_tokens for passes) / max(passes, 1)

If passes is zero, tokens_per_success is undefined. Print NA. Do not print infinity and call it a ranking.

# score_ledger.py — proposal, unexecuted example
import json
import sys
from collections import defaultdict

REQUIRED = {
    "run_id", "envelope_id", "task_id", "agent_id", "pack_hash",
    "input_tokens", "output_tokens", "tool_calls", "repair_loops",
    "wall_clock_seconds", "outcome",
}

def load(path):
    rows = []
    with open(path, encoding="utf-8") as handle:
        for line_no, line in enumerate(handle, 1):
            line = line.strip()
            if not line:
                continue
            row = json.loads(line)
            missing = REQUIRED - row.keys()
            if missing:
                raise SystemExit(f"line {line_no} missing {sorted(missing)}")
            rows.append(row)
    return rows

def score(rows):
    grouped = defaultdict(list)
    for row in rows:
        grouped[(row["agent_id"], row["envelope_id"], row["pack_hash"])].append(row)
    reports = []
    for key, items in grouped.items():
        n = len(items)
        passes = [r for r in items if r["outcome"] == "pass"]
        trunc = [r for r in items if r["outcome"] == "truncated"]
        tokens = sum(r["input_tokens"] + r["output_tokens"] for r in passes)
        reports.append({
            "agent_id": key[0],
            "envelope_id": key[1],
            "pack_hash": key[2],
            "n": n,
            "pass_rate": round(len(passes) / n, 3) if n else None,
            "truncation_rate": round(len(trunc) / n, 3) if n else None,
            "tokens_per_success": round(tokens / len(passes), 1) if passes else "NA",
        })
    return reports

if __name__ == "__main__":
    for report in score(load(sys.argv[1])):
        print(json.dumps(report, indent=2))

Run it:

python3 score_ledger.py runs/20260919T1400Z-lab-a.jsonl

Compare agents only when envelope_id and pack_hash match. If they do not match, the script still prints two blocks. You do the adult thing and refuse the average.

A useful derived check: if truncation_rate differs by more than one task, you are mostly measuring the envelope, not the agent. Widen the budget and rerun both. Or keep the tight budget and say so in the first sentence of the writeup.

Step 5 — Decision table: measurement vs advertisement

Use this table before anyone pastes a percentage into Slack.

Situation	Quote a single pass rate?	Publish instead
Same pack_hash, same envelope_id, both agents logged	No	The three-number triple, plus n
Different token caps, "same" tasks	No	Two separate triples, never averaged
Hidden tests were edited by the agent	No	invalid count and a patch diff
n under 8 tasks	No	The ledger, labeled as a protocol debug
Truncation rate > 0.25 for one agent only	No	"envelope-bound" and the raw traces
Latency on a shared free server	No	Tokens and outcomes only; drop wall-clock from the headline
Homepage hero or vendor one-pager	No	Link the recipe, or do not cite the lab

If a cell says no, the percentage is marketing even when the arithmetic is correct.

Step 6 — Pairwise only inside the same envelope

When you must pick a winner, do not subtract two pass rates. Count paired outcomes on the same task_id.

Proposal rule:

1. Drop any task where either agent is invalid.
2. On the rest, score +1 for the agent that passed when the other did not.
3. If both passed, award +1 to the lower token total only when the gap is larger than 15 percent. Otherwise call it a tie.
4. If both truncated, call it a tie. You learned about the envelope, not the model.

That 15 percent band is a knob. Write the knob in budget.yaml. Do not tune it after you see which agent you like.

What this protocol refuses to claim

It does not detect training-set contamination. It does not prove an agent is safe to ship. It does not turn a free server into a latency benchmark. Shared CPUs lie. Tokenizers lie in smaller ways. Your hidden tests lie if they only cover the happy path.

It also does not make a 12-task pack into a scientific paper. Twelve tasks can falsify a sloppy ranking. They cannot support a superlative.

Who should not use this

Skip this approach if you need a homepage number this afternoon. Skip it if your "agent" is allowed network access the other agent is not. Skip it if you will not log truncated runs because they look ugly. Skip it if the real question is production SLA, not coding-agent comparison. And skip it if you planned to treat free-tier queue time as intelligence.

If you want a lab that already offers free model access and a free server option, MonkeyCode is one place to run the ledger. The protocol does not depend on it. The only number worth quoting is the triple you can rebuild from JSONL and a hash file.