Break Agent Ties With Cost Per Fix and a Bootstrap Interval

You walk into the same meeting that has been eating coding-agent reviews all year. Two harnesses ran on the same forty-task pack. Agent A prints 62.5 percent resolved. Agent B prints 57.5 percent. Someone already put A on the slide.

Then the invoice, or the GPU hours, or the queue delay shows up. A looped the test runner, rewrote one file three times, and still needed a human merge. B failed more tasks. When it passed, it passed in one shot.

If you published only the pass rate, you published a brochure. Rank agents by cost per resolved task, and refuse a winner when the confidence intervals overlap. The numbers below are a worked example, not a lab report.

The ranking that the slide hid

Pass rate answers one question: did the suite go green. It does not answer how much work the agent burned to get there. It does not tell you whether a three-point gap would survive a different seed.

Hot takes keep claiming coding agents already outrun most developers. Treat those claims as untrusted until the protocol shows the unit of cost and the uncertainty. You need both, or you are ranking vibes.

Freeze a price table if money matters. Freeze token counts and wall-clock even when the invoice is zero. Tokens and seconds are still scarce on a free endpoint.

What the ledger must record

Do not start with a dashboard. Start with one JSONL row per task attempt. If a field is missing, you do not get to quote a ranking.

Each row needs:

1. run_id, agent_id, task_id, seed
2. resolved as a boolean from a frozen checker, not from the model’s self-report
3. prompt_tokens, completion_tokens, tool_calls
4. wall_ms measured outside the vendor’s stream
5. price_table_id so a later reader can see which rates you applied
6. notes for human merges, skipped tests, or harness crashes

Keep monetary cost as a derived column. Multiply tokens by the frozen table. If the table is all zeros because you used a free model, you still rank on tokens and wall-clock. Zero dollars is not zero work.

A sample pack you can score without an API key

Label this data as illustrative. Eight rows, two agents, four tasks. Tiny on purpose. The point is the method, not a leaderboard.

{"run_id":"r1","agent_id":"a","task_id":"t1","seed":7,"resolved":true,"prompt_tokens":1800,"completion_tokens":900,"tool_calls":4,"wall_ms":14000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"a","task_id":"t2","seed":7,"resolved":true,"prompt_tokens":4200,"completion_tokens":3100,"tool_calls":11,"wall_ms":41000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"a","task_id":"t3","seed":7,"resolved":false,"prompt_tokens":5100,"completion_tokens":4400,"tool_calls":14,"wall_ms":56000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"a","task_id":"t4","seed":7,"resolved":true,"prompt_tokens":2600,"completion_tokens":1200,"tool_calls":5,"wall_ms":18000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"b","task_id":"t1","seed":7,"resolved":true,"prompt_tokens":900,"completion_tokens":400,"tool_calls":2,"wall_ms":8000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"b","task_id":"t2","seed":7,"resolved":false,"prompt_tokens":1600,"completion_tokens":800,"tool_calls":3,"wall_ms":12000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"b","task_id":"t3","seed":7,"resolved":true,"prompt_tokens":1100,"completion_tokens":500,"tool_calls":2,"wall_ms":9000,"price_table_id":"local-0"}
{"run_id":"r1","agent_id":"b","task_id":"t4","seed":7,"resolved":false,"prompt_tokens":1500,"completion_tokens":700,"tool_calls":4,"wall_ms":11000,"price_table_id":"local-0"}

Save it as sample_ledger.jsonl. Agent A looks like the winner if you only count greens. Wait for the division.

Step 1: Derive the three columns that matter

Resolved rate is still useful. It is not sufficient. Compute three numbers per agent, then stop talking until the intervals exist.

• Resolved rate: greens divided by tasks
• Tokens per resolved (TPR): total tokens divided by greens. If greens is zero, the agent has no TPR. It failed the suite.
• Seconds per resolved (SPR): total wall-clock divided by greens

Money per resolved is optional. Attach it only after you freeze price_table_id. Do not mix a March rate card with a September run.

Step 2: Score the ledger locally

This script is a worked example. It does not call a model. Point it at any JSONL that matches the schema.

# ledger_stats.py — worked example, not a published benchmark
from __future__ import annotations

import argparse
import json
import random
from collections import defaultdict
from pathlib import Path


def load_rows(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text().splitlines():
        line = line.strip()
        if line:
            rows.append(json.loads(line))
    return rows


def group(rows: list[dict]) -> dict[str, list[dict]]:
    by_agent = defaultdict(list)
    for row in rows:
        by_agent[row["agent_id"]].append(row)
    return by_agent


def point_estimates(rows: list[dict]) -> dict:
    n = len(rows)
    greens = [r for r in rows if r["resolved"]]
    tokens = sum(r["prompt_tokens"] + r["completion_tokens"] for r in rows)
    wall_s = sum(r["wall_ms"] for r in rows) / 1000.0
    resolved = len(greens)
    return {
        "n": n,
        "resolved": resolved,
        "resolved_rate": resolved / n if n else 0.0,
        "tpr": tokens / resolved if resolved else None,
        "spr": wall_s / resolved if resolved else None,
        "tokens": tokens,
    }


def bootstrap_rate(rows: list[dict], rounds: int, seed: int) -> tuple[float, float]:
    rng = random.Random(seed)
    n = len(rows)
    stats = []
    for _ in range(rounds):
        sample = [rows[rng.randrange(n)] for _ in range(n)]
        stats.append(point_estimates(sample)["resolved_rate"])
    stats.sort()
    lo = stats[int(0.025 * rounds)]
    hi = stats[int(0.975 * rounds)]
    return lo, hi


def intervals_overlap(a: tuple[float, float], b: tuple[float, float]) -> bool:
    return not (a[1] < b[0] or b[1] < a[0])


def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--path", type=Path, required=True)
    p.add_argument("--bootstrap", type=int, default=2000)
    p.add_argument("--seed", type=int, default=7)
    args = p.parse_args()

    by_agent = group(load_rows(args.path))
    summary = {}
    for agent, rows in sorted(by_agent.items()):
        est = point_estimates(rows)
        lo, hi = bootstrap_rate(rows, args.bootstrap, args.seed)
        est["ci95"] = (round(lo, 3), round(hi, 3))
        summary[agent] = est
        print(agent, est)

    ids = list(summary)
    if len(ids) == 2:
        a, b = ids
        overlap = intervals_overlap(summary[a]["ci95"], summary[b]["ci95"])
        print("ci_overlap", overlap)
        if overlap:
            tpr = {k: summary[k]["tpr"] for k in ids}
            print("tie_break_tpr", tpr)
            print("no_winner_on_rate")


if __name__ == "__main__":
    main()

Run it:

python ledger_stats.py --path sample_ledger.jsonl --bootstrap 2000 --seed 7

On this toy pack, A resolves 3/4 and B resolves 2/4. The intervals on four tasks are wide. They will overlap. That is the lesson, not a bug.

Step 3: Refuse a winner when the intervals overlap

A three-point gap on forty tasks can be noise. A fifteen-point gap on four tasks is still noise. Bootstrap the resolved bit, not the marketing headline.

Rule, in order:

1. If either agent has zero greens, do not compute TPR. Report failure, not a ranking.
2. If the 95 percent intervals on resolved rate overlap, you do not have a rate winner.
3. Only then compare TPR. Lower tokens per green wins the tie.
4. If TPR is within 10 percent, compare SPR. If both still tie, publish a draw.

The 10 percent band is a policy choice. Put it in the protocol. Do not tune it after you see the chart.

Step 4: Pin the rule with a test

If the tie-break lives only in a slide, it will drift. Lock it in a test you can run on a laptop.

# test_ledger_stats.py
from ledger_stats import intervals_overlap, point_estimates


def test_overlap_means_no_rate_winner():
    a = (0.40, 0.80)
    b = (0.35, 0.70)
    assert intervals_overlap(a, b)


def test_separated_intervals_are_not_a_tie():
    a = (0.60, 0.85)
    b = (0.20, 0.45)
    assert not intervals_overlap(a, b)


def test_tpr_uses_greens_not_attempts():
    rows = [
        {"resolved": True, "prompt_tokens": 100, "completion_tokens": 100, "wall_ms": 1000},
        {"resolved": False, "prompt_tokens": 900, "completion_tokens": 900, "wall_ms": 9000},
    ]
    est = point_estimates(rows)
    assert est["resolved_rate"] == 0.5
    assert est["tpr"] == 200  # failed tokens still sit in the numerator? change this if you prefer

Read that last assertion twice. The sample puts all tokens in the numerator, including failed attempts. That punishes thrash. If you only divide tokens from successful tasks, a noisy agent looks cheap. Pick one definition. Write it on the protocol. Do not switch after the ranking is ugly.

python -m pytest test_ledger_stats.py -q

Failed-attempt accounting is the whole fight. Include the waste, or you are ranking luck.

Decision table: what you may print

Situation	You may print	You may not print
Intervals overlap, TPR gap > 10%	Draw on rate; B cheaper per fix	"B is worse"
Intervals overlap, TPR gap ≤ 10%	Draw	Any crown
Intervals separated, higher rate also higher TPR	Rate winner, with the cost warning	"Strictly better"
Intervals separated, higher rate also lower TPR	Rate and cost winner	Human-level claims
N < 30	Exploratory notes	Vendor ranking
Price table not frozen	Tokens and seconds only	USD rankings
Checker not frozen	Nothing	Anything

Print the table next to the chart. If a cell says no, delete the sentence.

Where a free lane belongs

Run this harness off the production key. Eval traffic that shares a paid quota with users will contaminate both the bill and the latency column.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option are one way to keep a small ledger off paid production traffic. Point the agent at whatever OpenAI-compatible endpoint you already trust, write JSONL, and score it with the script above. That is the only product role here: a place to burn eval tokens without mixing them into customer traffic. No model names, no quota theater, no claimed speedup.

A free server is enough for this scale. Forty tasks, one process, a disk for JSONL. If you need a multi-GB sandbox per task, this lane is the wrong hardware story. Say so and stop.

Who should not use this

Do not use a four-task pack to rank vendors. Do not use bootstrap-on-tasks as a substitute for a held-out human study. Do not turn TPR into a hiring rubric. Do not compare a local 7-second checker against a cloud agent that waits on a queue and then claim a latency win.

Skip this method if you cannot freeze the checker. Skip it if tool schemas change between agents. Skip it if one agent may call the network and the other may not. Those are different tasks. They do not belong on one leaderboard.

Wall-clock on a free server will jitter. Treat SPR as a coarse filter, not a microbenchmark. If you need tail latency, provision dedicated hardware and say so in the protocol.

What you still may not claim

You may claim that, on this frozen pack, with this seed, with this checker, agent B spent fewer tokens per green while the rate intervals overlapped. You may not claim B is the better engineer. You may not claim the model is “better than most developers.” That sentence needs a human protocol you have not run.

Ship the ledger with the post. Ship the script. Ship the overlap rule. If a reader cannot replay the ranking from the JSONL, you do not have a number. You have a slide.