Tokens per Accepted Patch: The Only AI Metric I Keep

Everyone is arguing about whether AI writes good code. I stopped arguing. The number that actually changed my workflow is boring: tokens per patch I keep.

That is not latency. It is not a benchmark score. It is total spend divided by accepted results. One division sign, and half the hype falls apart.

Two loud conversations, one missing number

This week the feed is full of the same two threads. One says AI is coming for our jobs. The other hands out benchmark trophies. Both ask the same question: is the model good?

That question is unanswerable. Good at what? On whose task list? At what price per result you actually ship?

A benchmark score is a rate. Your bill is a product. Rates flatter; products cost.

I have watched a model feel twice as fast while costing three times as much. Warm cache, small prompts, one lucky run. My dashboard was green. My review queue was not.

So I stopped graphing speed. I graph efficiency now.

Define the metric like you mean it

tokens per accepted patch =
  (prompt tokens + completion tokens, every attempt)
  ---------------------------------------------------
  (patches the test suite actually accepted)

The denominator is the honest part. Only kept work counts. A model that needs three swings and lands one is expensive, even when each swing looks instant.

Rejected attempts are never free. They burn tokens, reviewer patience, and the twenty minutes of context you rebuild afterward. A free tier does not erase that. Free is a price, not a budget.

Ask yourself a blunt question. If your provider charged you per accepted patch instead of per token, would your prompt library survive the invoice?

The harness, in about forty lines

No framework. No SDK. Just urllib, a frozen task list, and a test runner as the judge. This is a template — read it, run it, then trust it.

# tokens_per_patch.py — template, adapt before you quote numbers
import collections, csv, json, pathlib, subprocess, time, urllib.request

ENDPOINT = "https://your-endpoint.example/v1/chat/completions"  # OpenAI-compatible
MODEL    = "your-model-id"
KEY      = pathlib.Path(".key").read_text().strip()

def ask(prompt):
    body = json.dumps({
        "model": MODEL,
        "temperature": 0,           # freeze it, or you are measuring dice
        "messages": [{"role": "user", "content": prompt}],
    }).encode()
    req = urllib.request.Request(ENDPOINT, body, {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {KEY}",
    })
    with urllib.request.urlopen(req, timeout=120) as r:
        payload = json.load(r)
    text = payload["choices"][0]["message"]["content"]
    return text, payload.get("usage", {})   # <- the ledger lives here

def accepted(patch, task):
    """No oracle, no metric. Your tests are the only judge."""
    f = pathlib.Path(task["file"])
    backup = f.read_text()
    try:
        f.write_text(patch)
        return subprocess.run(task["test"], shell=True, timeout=60).returncode == 0
    finally:
        f.write_text(backup)

Now the loop. One variable per variant, same task order every run, and log the rejects instead of deleting them.

rows = []
for variant in VARIANTS:
    for task in TASKS:
        t0 = time.time()
        out, usage = ask(variant["template"].format(task=task["prompt"]))
        ok = accepted(extract_patch(out), task)
        rows.append({
            "variant": variant["id"], "task": task["id"],
            "ok": int(ok),
            "in":  usage.get("prompt_tokens", 0),
            "out": usage.get("completion_tokens", 0),
            "ms":  int((time.time() - t0) * 1000),
        })

spend, won = collections.Counter(), collections.Counter()
for r in rows:
    spend[r["variant"]] += r["in"] + r["out"]
    won[r["variant"]]   += r["ok"]

for v in sorted(spend):
    ratio = spend[v] / max(won[v], 1)
    print(f"{v:>12}  {ratio:8.0f} tok/patch  kept {won[v]}/{len(TASKS)}")

with open("ledger.csv", "w", newline="") as fh:
    csv.DictWriter(fh, fieldnames=rows[0].keys()).writerows(rows)

Two mistakes bit me early. First, I summed tokens from the winning run only. My numbers looked superb and were fiction. Count every attempt, including the ones you rage-quit.

Second, I changed three prompt things at once and learned nothing. One variable per variant. Otherwise you are paying to measure noise.

The experiment I actually keep

Twelve tasks from your own suite. Frozen list, frozen order, temperature zero. Three prompt variants: baseline, plus a diff hint, plus two worked examples.

Then one chart. Horizontal bars by variant. Bar length is tokens per accepted patch. A second column is kept count, so a cheap variant that solves nothing cannot win.

The shape matters more than the absolute number. If your bar is tall and your kept column is short, you are buying attempts, not results. If shortening the prompt cuts tokens but drops kept count, you just moved the cost into your review queue.

Variant	Tokens spent	Kept	Tokens per kept patch
baseline	your CSV	your CSV	your CSV
+ diff hint	your CSV	your CSV	your CSV
+ 2 examples	your CSV	your CSV	your CSV

I keep the CSV, not the screenshot. Screenshots age badly and cannot be re-aggregated when someone asks about retries.

Where free capacity changes the plan

Here is the awkward part. A metered API discourages you from running this harness properly. You trim the task list to save money, and a trimmed task list is a biased one.

That is the one real argument for free model access. This article was prepared as part of MonkeyCode's product outreach, and the project advertises ten million free tokens plus a free server option. Treat that figure as a starting condition, not a guarantee — quotas change, so check the current program page before you plan around it.

What I would use it for is narrow and honest. Run the full task list instead of a trimmed one. Run all three variants instead of the cheapest. Park the harness on the free server so the sweep does not compete with your laptop for CPU during the run.

What I would not do is confuse free tokens with free attention. The denominator still costs you. Reviewers still read the rejects. Your test suite still needs to be good enough to be a judge.

Limitations, stated plainly

Small sample means small confidence. Twelve tasks is a smoke test, not a paper.

Your tests are a harsh oracle. They reject correct patches that fail one assertion, which inflates your ratio.

Usage fields vary between providers. If usage is missing, you are measuring nothing, so stop.

And free capacity is a moving target. Rebuild the chart monthly or discard it.

Who should skip this

Anyone without a test suite. Without an oracle you have opinions, not measurements.

Anyone shipping latency-critical paths. Tokens per patch says nothing about p99, and I have the scars to prove those are different problems.

Anyone whose real bottleneck is review. If humans are the constraint, cheaper tokens just queue faster.

So before you add another model to your stack, ask the boring question. What does one patch you keep actually cost you?

If you want a zero-cost place to run the first sweep, MonkeyCode's free tier is a reasonable starting line — bring your own task list and your own tests.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.