DEV Community

Cover image for The tokenomics of software development: checking AI-written code now costs as much as writing it
Samuel Cozannet for Cloud ex Machina

Posted on Originally published at blog.cloudexmachina.io

The tokenomics of software development: checking AI-written code now costs as much as writing it

Originally published on the Cloud ex Machina engineering blog.

Our AI coding agents now spend about as much checking and shipping code as writing it. We got there in six steps past the review wall, the point where agents write code faster than people can review it. Each step was paid for in tokens, and each one handed us the next problem. Here they are, with our own numbers.

How agentic review shaped our agent bill: monthly coding-agent spend split into subscriptions and pay-per-use overage, with overage growing as agents took over more of the review, ultrareview formalising it, Codex seats for all, and Tach. Below, merged PRs across our repositories as a multiple of October 2025: product PRs about 8x, agent-management PRs about 26x

1. We decided to fix forward with agentic code review

We started with a stance. Every pull request exists for a good reason, so review stopped being a gate that bounces it back with complaints and became the place where we take it to the best version of itself.

At agent speed, that only works if the team's expertise already sits in the review, so we handed it to agents and formalised it as ultrareview, our own multi-agent review. (Anthropic ships a cloud review with the same name. Great minds. Ours is now called cxm-pr-review, which is less catchy and entirely ours.) Nobody asks a teammate to look at a pull request until its author thinks it's as good as it gets. Before it opens, the review runs in separate agents that start without the author's context, and the change has to clear gate after gate. Does it do what the ticket asked, and do the acceptance checks prove it? What could break? Is it secure? Would the tests notice if a changed line were mutated? Does it duplicate something we already have? What a gate finds gets fixed in the PR, and only a change that reaches a quality score of 85 goes to a teammate. None of this is exotic. Tessl describes the same pairing of adversarial reviewer agents and gates an agent can't bypass, and Chainguard's Checks lets an agent loop require a passing verdict before a pull request opens.

In the month after it went on every pull request, our product PRs went up by more than 40%. The models got better at the same time, so review can't take all the credit, but without it we couldn't have trusted that volume.

It cost money, because a review burns around a million tokens whatever the size of the diff, and Anthropic prices its own Code Review on tokens too, and its ultrareview at $5 to $25 a run. We ran ours on pay-per-use overage on purpose, to learn the real price, and in May and June the agent bill sat at more than three times its usual level, with a couple of thousand dollars of overage at the peak. Cheap tuition, as these things go.

Fixing forward also assumes the reason behind the PR was a good one. We'll come back to that.

2. We bought the tokens wholesale

Once we knew the number, the fix was boring, which is how you want fixes. Every developer got a Codex seat on top of Claude Code, and the heaviest users got more capacity. Overage fell to zero the next month, came back smaller in August, and has been zero since September, while agent spend settled 44% below the peak and we shipped more. The real inflexion came in September, when the top-tier flat-rate plan became the default for everyone. Velocity followed straight away, and merged product PRs nearly doubled from August to September.

At list price, our usage would have cost at least six times more, and that gap shaped everything we built next. From October 2025 to September 2026, agent spend per merged pull request fell 4.6x, and by October merged product PRs had grown about 8x. Agent-management PRs, the ones that change our agents' rules and skills or build Tach itself, grew about 26x from a much smaller starting point. That work is what lets us ship product faster, because every rule an agent stops breaking is a review round we never pay for again.

3. We built a meter for agent token spend

With the bill under control, our attention shifted to the return on all those tokens. So we built Tach, short for tachymeter, with an off-the-shelf product-analytics tool for the events and our own layer for everything it wasn't built for. Version one simply recorded every agent session with its tokens, its tools and its work, and it paid for itself fast by finding a tool bug our agents had worked around for weeks and memories that kept steering agents after the problem was fixed.

Once we had all that data, we needed to classify it. We tagged every trace with the repo, branch and ticket the agent was working on, and went looking for a simple classifier we could use off the shelf. TypeSafe's Jev matched our criteria: send it some text and a few typed questions, get back a label and a probability for each. We didn't want to send our sessions to a third party, and we loved the idea, so we built our own. Our JEV speaks the same request shape as the open-source OpenJev, puts a small, fast model behind three votes, and lives next to Tach. It now classifies our pull requests and grades our tickets before the factory takes them, and it drew the chart below by reading the prompts of every piece of work Tach carved out of a session.

Share of agent spend by activity, per week, over eight weeks: writing code 39% overall, review 23%, shipping (CI, rebases, merge conflicts) 13%, planning 10%, investigating 7%, other 8%. Review went from 14% to 32% in the week of September 21 and stayed between 18% and 34% after that, while writing code fell from 47% in the first week

The interesting line is where writing code meets everything that comes after it. In mid-September, writing code took almost half the spend and reviewing and shipping it about a quarter. Since late September the two have run neck and neck, and in the first week of November review and shipping took 46% against 25% for writing. Writing the code is turning into the cheap part, and making sure it's right is where the tokens go now.

4. The meter became our software factory

Plenty of people now argue that every team should build its own orchestrator, and Eric Zakariasson's talk on building your own software factory makes the case well. For us it happened on its own. Two of us started separate efforts, and they converged into rewriting our whole development lifecycle around a software factory. The idea was to use Tach across every repository as a shared store of knowledge and know-how, with each repository adding its own pieces as plugins, the same model Claude Code uses. It turned out to be easy and it worked, so we moved our SDLC harness into Tach piece by piece, and Tach quietly became our factory. It runs loops, agent tasks that fire on events like a review request or a failed CI run, on the machine of whoever owns the work, and it walks pull requests from draft to merge. Agents managing agents, or middle management for software, if you prefer. The 13% we spend on shipping is the first number we want it to shrink.

The design rule comes from step 2. Tach runs on our own subscriptions and our own machines, while hosted options mostly price the other way (Anthropic bills its managed review on usage credits and suggests an API key for GitHub Actions). Whoever owns the harness decides which subscription pays, so we kept ours.

5. We're moving the scrutiny to the ticket

Fixing forward makes whatever a PR was trying to do technically excellent, which is also the catch. If the ticket asked for the wrong thing, review polishes the wrong thing faster than any of us could. It's very easy to write a great spec for a bad idea, and an agent will build it beautifully, so our tickets had to get close to perfect in their intent. The how, we can leave to the agents.

Fixes merged for every feature merged, by month in 2026: 1.9 in February, falling to 1.1 in June, then 1.5 to 1.6 from July to October, against a reference line of one fix per feature

The number we watch most closely now is fixes per feature. In September and October we merged about 1.6 fixes for every feature, and every one of those fixes was a trip back to something we'd already shipped. We want that number down, and it's why we now spend more time and more tokens at the start of the pipeline, on the ticket and on the review, so a change comes out right the first time. Re-reviewing what we shipped after release, replaying our review on the PRs that introduced a bug to learn what it missed, is moving into Tach too, so the next ticket starts from what the last one got wrong.

A study titled, fittingly, Tokenomics found that an automated agent pipeline building small programs from short task descriptions spent 59% of its tokens on review and 2% on design. We spend 23% on review and 10% on planning, five times their share for design. The setups are too different for a fair comparison, but the shape fits. When the intent is thin, agents spend their tokens in review working out what was wanted.

xkcd 1205, Is It Worth the Time?: a table of how long you can spend making a routine task more efficient before you spend more time than you save, by how often you do it and how much time you shave off

xkcd 1205: Is It Worth the Time? by Randall Munroe, CC BY-NC 2.5.

Agents broke that table. Building a change is now so cheap that almost anything clears the bar, so the question is whether it's worth anything at all. Our Linear tickets now state the outcome and why it matters, a "done when" an agent can check before work starts, and a number to watch after release, and the factory refuses tickets without them.

Refining a ticket works like reviewing a PR. Can it be delivered? Is it a duplicate, because two tickets for one problem means paying twice? Has it been tried before? Every ticket those gates stop is work we never plan, build, review or maintain, which is how we stay lean with more agents than people. The six-line fix from our first post is the model, with almost nothing to build, about $20,000 a year saved and a measurement to prove it.

This step is still a work in progress, and the open question is value. A "done when" tells an agent when a change works. Whether the change was worth building is a separate question. What we're after is a way to estimate a feature's value early, before anyone specs it in detail, so we can rank ideas while they're still cheap and spend the expensive tokens only on the ones that win.

6. What's on the R&D bench

Two more ideas are still in the lab, in the same spirit of paying less to get it right.

The first one comes from caveman, the skill that gets agents to talk like cavemen to save tokens. It got us thinking about the words our agents use for the work itself. We already keep a company glossary, so what if we were strict about it and forced agents to write tickets, specs and reviews in a compact dictionary of agreed terms? We suspect it would cut tokens and ambiguity at the same time, since two words for one thing is how a spec ends up meaning two things.

The second is drift. An agent takes a rule for granted, applies it where it doesn't belong, and before long the mistake is spreading through other agents' work as if it were a rule itself. We've already written about memory drift. What we don't have yet is a metric that catches this kind of drift early and tells us when a fix has held for good.

We'll write about both once we have numbers.

If you're about to hit the review wall

  1. Spend tokens on review, but buy them flat. Pay-per-use is the fastest way to regret it.

  2. Meter before you optimise. Groundbreaking advice, we know, right up there with "measure twice, cut once". The catch is that metering your developers' agent sessions is harder than it sounds, because every harness records a session its own way and sessions hop between worktrees. We ended up writing our own plugin for Claude Code, Codex and the rest, and it was worth it.

  3. Own your harness of harnesses. Your processes and your orchestration are your development culture written down, and when they live in one place the whole team gets better with them.

  4. Check the intent, then fix forward. Review makes a PR better, but only the ticket makes sure it was worth building.

None of this is finished, and right now we're working on intake, the path a request takes before an agent builds anything, because every fix we avoid there gives the factory time back for features.

It's also the work Cloud ex Machina does for other teams. We run and price the loops of a software factory like ours, and we turn production cost data into application-level FinOps recommendations, whether or not a factory picks them up. If you want one check to start with, pull last month's merged pull requests, count the fix: titles against the feat: titles, and see where your fixes per feature sit.

We'd love to hear how other teams are building their factories, what they meter and what caught them off guard. DM me on LinkedIn. I never say no to a friendly tech talk.

Sources: GitHub merge data, Anthropic and OpenAI invoices, GitHub's billing API, error-tracking receipts, our AWS cost and usage report, and Tach. Spend shares come from eight weeks of sessions, September 14 to November 8, with each piece of work classified from its prompts; cached tokens aren't priced, so read them as shares, not amounts.

Top comments (1)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The 'a review burns around a million tokens whatever the size of the diff' number is the one that reframes the whole trade — checking now costs as much as writing, and the review wall just moved from humans to the bill. The 85 quality-score gate before a teammate sees a PR is the honest part: the agents earn their keep by clearing gates, not by volume. Do the merged-PR multiples (8x product, 26x agent-management) hold up as real throughput, or is some of it review-loop churn the gates themselves generate?