Your Benchmark Has No Cost Axis: Score Free Endpoints by Quota per Accepted Patch

A team I traded notes with ran the same 40-task patch suite against two free endpoints. Endpoint A passed 71% of tasks, endpoint B passed 64%. They shipped A into their CI helper. Twenty-two days later A's monthly allowance was gone and the helper started failing open — while B, the "worse" endpoint, would have finished the month with quota to spare.

The pass rate was not wrong. It was just answering a different question than the one that mattered. On a free tier, the budget is finite, so the objective is not patches per run — it is patches per unit of quota.

Below is the accounting layer I add to agent benchmarks now: a spend ledger, three derived metrics, and a set of controls that keep the numbers from turning into marketing.

Why a pass rate without a cost axis misleads

A single number collapses two very different failure modes. An endpoint that fails a task on the first attempt and then fails it again on retries has burned budget for nothing. An endpoint that fails once and succeeds on attempt two has burned some budget for something.

Pass rate sees those as identical. Your quota does not.

There is a second distortion: retry policy. If your harness allows three attempts, an endpoint that flails is charged three times, and the flailing is silently hidden inside a "pass". Change the attempt budget and the ranking can flip. That is a harness artifact, not a capability difference.

The three metrics worth reporting

Fix the task set, fix the attempt budget, then report all three of these together. None of them is meaningful alone.

1. Acceptance rate — accepted tasks ÷ total tasks. The familiar number. Keep it for continuity.
2. QAP (quota per accepted patch) — total tokens consumed ÷ accepted tasks. This is the cost axis. Lower is better.
3. Retry tax — tokens spent on attempts that did not produce the accepted patch ÷ total tokens. A high retry tax means your pass rate is being bought with silent spend.

Add a fourth flag, censoring: did the run hit your token cap before finishing the task set? A censored run's acceptance rate is not comparable to an uncensored one, because the hardest tasks never got their full attempt budget.

Step 1: Freeze the task set and the attempt budget

Write both to a file and hash it. If the hash changes, the run is a new benchmark and the old numbers do not carry over.

sha256sum tasks/frozen.jsonl harness_config.json | tee runs/2026-10-26.manifest.sha256

harness_config.json should pin max_attempts, temperature (0 for comparability), cap_tokens, and the prompt template version. Nothing that can drift silently.

Step 2: Instrument the call, not the result

The ledger records one row per attempt, not one row per task. That is the whole trick — retries are where the cost hides.

# spend_ledger.py — template, not a tested library. Adapt the fields to your harness.
from dataclasses import dataclass
import json, time

@dataclass
class Attempt:
    run_id: str
    task_id: str
    arm: str              # endpoint label, e.g. "free-arm-a"
    attempt: int          # 1-based retry index
    prompt_tokens: int
    completion_tokens: int
    wall_ms: int
    accepted: bool        # set by YOUR checker, never by the model
    checker_error: str | None = None

    @property
    def tokens(self) -> int:
        return self.prompt_tokens + self.completion_tokens

def run_task(call, checker, task, arm, run_id, max_attempts=3):
    rows = []
    for i in range(1, max_attempts + 1):
        t0 = time.monotonic()
        resp = call(task.prompt, temperature=0)
        wall = int((time.monotonic() - t0) * 1000)
        verdict = checker(task, resp.text)      # runs the test suite, returns pass/fail
        rows.append(Attempt(
            run_id=run_id, task_id=task.id, arm=arm, attempt=i,
            prompt_tokens=resp.usage.prompt_tokens,
            completion_tokens=resp.usage.completion_tokens,
            wall_ms=wall, accepted=verdict.ok,
            checker_error=None if verdict.ok else verdict.reason[:200],
        ))
        if verdict.ok:
            break
    return rows

If your provider does not return usage, log the request and response text and estimate tokens from a tokenizer you keep in the repo. Estimating is fine — but state that you estimated, and use the same estimator for every arm.

Step 3: Aggregate with censoring made explicit

# summarize.py — same caveat: template code, run it against your own data first.
def summarize(rows, cap_tokens):
    spend = sum(r.tokens for r in rows)
    accepted_ids = {r.task_id for r in rows if r.accepted}
    accepted = len(accepted_ids)
    first_try = sum(1 for r in rows if r.accepted and r.attempt == 1)
    wasted = sum(r.tokens for r in rows if not r.accepted)

    return {
        "tasks": len({r.task_id for r in rows}),
        "accepted": accepted,
        "acceptance_rate": round(accepted / max(1, len({r.task_id for r in rows})), 3),
        "tokens": spend,
        "qap": round(spend / accepted, 1) if accepted else None,
        "retry_tax": round(wasted / spend, 3) if spend else None,
        "first_try_share": round(first_try / accepted, 3) if accepted else None,
        "censored": spend >= cap_tokens,
    }

Run each arm with the same cap and the same order. Order matters more than people expect: a per-minute rate limit turns task ordering into a latency confounder, and latency is the quiet reason harnesses crash mid-run.

python -m bench.run --arm free-arm-a --tasks tasks/frozen.jsonl \
  --max-attempts 3 --cap-tokens 250000 --out runs/2026-10-26-a.jsonl

Step 4: Add the controls that make the table honest

Four controls, all cheap:

• Negative control. One task whose correct patch you already know fails the checker. If an arm "passes" it, your checker is broken and every number above is void.
• Floor task. A one-line fix any competent agent should pass. It measures harness overhead and token accounting, not capability.
• Paired arms. Run both endpoints on the same task order in the same window. Sequential runs on different days pick up provider drift and file it as endpoint quality.
• Retry budget parity. Same max_attempts everywhere. Three for one arm and one for another is not a comparison, it is two benchmarks sharing a folder.

Step 5: Read the numbers as a decision table, not a podium

Arm	Acceptance	QAP (tokens/patch)	Retry tax	Censored	Practical read
A	0.71	6,400	0.44	no	Wins on quality, eats quota on retries
B	0.64	3,100	0.12	no	Loses 7 points, survives the month
C	0.68	—	—	yes	Unusable: cap hit before the suite finished

A is the right pick for a burst workload with a deadline. B is the right pick for a background job that must keep running unattended. C should be dropped from the table entirely — a censored run is not a low score, it is a missing measurement.

Where a free tier genuinely fits this method

This accounting layer exists because finite allowances are the normal case, not an edge case. MonkeyCode is relevant here for two operator-stated reasons: free model access, and a free server option to host the harness itself.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes the free model allowance as 10 million tokens; confirm the current figure and its reset behavior in-product before you put it in a table, because free-tier terms move and a stale quota number invalidates every QAP figure derived from it. The server option matters for a narrower reason: running the harness next to the endpoint removes your laptop's thermal throttling and home upload speed from the latency column, so wall_ms measures something closer to the provider.

Neither of those makes the numbers better. They just make the measurement less contaminated — which is the only thing a benchmark host should do.

Limitations, stated up front

• Token accounting is provider-shaped. Cached prompt tokens, tool-call overhead, and system-prompt reuse are billed differently across providers. QAP compares arms only when measured by the same accounting method.
• Free-tier availability is not a guarantee. Rate limits, model availability, and allowances change without notice. Re-run the manifest before you compare a new month's numbers to an old table.
• QAP is not a price. It is quota consumption per accepted patch. If you later map it to money, keep the mapping in a separate column.
• Small suites produce wide intervals. With 40 tasks, a 7-point gap is inside the noise of a single re-run. Report the re-run spread beside the point estimate.

Who should not use this

Skip this method if you need a latency SLA — an unattended free server and a free model allowance are the wrong substrate for p99 guarantees. Skip it if your checker is still a human eyeballing diffs; QAP without an automated checker just measures how long a person stared at the screen. And skip it if you cannot reserve quota for the benchmark separately from production, because a benchmark that starves your real workload will be quietly disabled by the first person it inconveniences.

If you already have a checker and a frozen suite, the ledger is about thirty lines. Start by adding attempt and the two token fields to whatever you currently log — that alone usually explains why last month's "best" endpoint stopped answering.