The smaller token bill can buy fewer working builds

A failed build can be wonderfully economical. It just has the small inconvenience of not working. 🫠

That sounds obvious until a chart sorts coding agents by tokens per attempt. A workflow can win that chart while producing fewer usable results.

I made a tiny ledger to show the accounting change. Every number below is synthetic. These are invented workflows named lean and steady, not measurements of models or frameworks.

What I Built

The starting point was the renewed discussion around Nolan Lawson's JavaScript framework benchmark. His setup asks agents to build an app, checks conformance, and reports several dimensions, including tokens and resulting app quality. It also reports variation and failed runs.[1]

That's a useful direction: inspect the finished thing as well as the effort spent making it. My example isn't a rerun of his benchmark or a correction to his framework rankings. It isolates one question for anyone building their own evaluation: what happens to the tokens spent on failed attempts?

I wrote a Python summary that retains those attempts and reports both the per-attempt median and the total tokens divided by accepted results. 🧮

Computed illustration from the synthetic records below. No model calls or framework comparison were run.

Demo

The fixture contains the same number of attempts for each workflow:

Synthetic workflow	Token units per attempt	Accepted results
lean	1,000; 1,000; 1,000	1 of 3
steady	1,600; 1,600; 1,600	3 of 3

For this illustration, each row represents a complete attempt, including its repair work. All rows use the same token-unit definition. ✅

Run this with Python:

from collections import defaultdict
from statistics import median
import json
# Synthetic attempts: (workflow, token units, acceptance passed).
# Each row includes the complete attempt, including repair work.
runs = [
    ("lean", 1000, True),
    ("lean", 1000, False),
    ("lean", 1000, False),
    ("steady", 1600, True),
    ("steady", 1600, True),
    ("steady", 1600, True),
]
def summarize(rows):
    groups = defaultdict(list)
    for workflow, tokens, passed in rows:
        groups[workflow].append((tokens, passed))
    result = {}
    for workflow, attempts in groups.items():
        total = sum(tokens for tokens, _ in attempts)
        accepted = sum(passed for _, passed in attempts)
        result[workflow] = {
            "attempts": len(attempts),
            "accepted": accepted,
            "median_tokens_per_attempt": median(t for t, _ in attempts),
            "total_tokens": total,
            "tokens_per_accepted_result": total / accepted if accepted else None,
        }
    return result
if __name__ == "__main__":
    print(json.dumps(summarize(runs), indent=2))

The useful part of the output is:

lean:   median per attempt = 1000; total / accepted = 3000
steady: median per attempt = 1600; total / accepted = 1600

The arithmetic is plain:

lean:   (1000 + 1000 + 1000) / 1 = 3000
steady: (1600 + 1600 + 1600) / 3 = 1600

If I silently drop the failed rows, lean looks like it spent only 1,000 tokens per accepted result. That discards the effort that produced no accepted output. The ledger keeps it.

💡 These summaries answer different questions. The median describes a typical attempt. Total divided by accepted results describes the observed aggregate token effort per acceptance in this fixture. Neither replaces the pass count.

What if nothing passes?

I also checked a group containing a failed attempt and no accepted result. The summary returns None, serialized as null, for tokens per accepted result.

Zero would be a misleading result: it would make total failure look exceptionally efficient. There was no accepted result to divide by.

The acceptance check is part of the measurement

Before collecting your own records, decide what “accepted” means for the task. Does the app boot? Does the requested behavior work? Are important error cases checked? Keep that grader outside the agent's editable output.

The DEV write-up from the Kumiko team is a useful companion: their isolated agent evaluation first exposed problems in their own grader and documentation.[2] A pass count only helps if the acceptance check itself is doing the intended job. 🔍

For real comparisons, keep the task set, model settings, versions and stopping rules matched. Save failed and timed-out attempts, and record whether setup or repair effort is included. If an attempt has missing usage, mark it missing rather than turning it into zero.

Partner Technologies

This example uses Python's standard library. It doesn't use a model provider or sponsor integration.

The limit is deliberate: this is an accounting demonstration, not evidence that one coding workflow beats another. Token counts also aren't a billing total. Cached input, uncached input and output can have different prices; preserve those buckets when accounting for actual usage.

Hackathon Experience

I built this as a standalone local example. It isn't an MLH event project or competition entry.

References

1. Nolan Lawson — The AI cost of JavaScript frameworks
2. @marc_kumiko — Our framework docs, three models, isolated agents: 99 seconds, 50 minutes, and a timeout

Asset credits

Original chart computed from the synthetic ledger. No external benchmark figure is reused.

AI disclosure

AI was used to draft the article.