Run A uses the cheaper model. It edits the right files, fails a test, retries with a wider change, burns through more tool calls, and stops with a confident completion message. An engineer spends the next hour figuring out which half of the patch can be kept.
Run B uses a more expensive model. It works from the same repository state, stays inside scope, runs the named checks, and returns a patch the reviewer accepts with one small correction.
The model line item can make Run A look cheaper. The engineering outcome says otherwise.
Token price is an input price. It does not tell you what a verified patch cost. If coding agents are doing repository-level work, the useful internal metric is cost per accepted task, with every retry, failed check, recovery step, execution environment, and human intervention left in the ledger.
Your invoice measures consumption, not completion
Tokens matter. Tool calls matter. Model time and sandbox time matter. You should track all of them.
They still stop short of the result.
A provider invoice cannot tell you whether a migration was safe, a visual regression was caught, or the reviewer rejected the patch. It usually cannot associate an abandoned worktree with the accepted patch that replaced it. If the agent switched models halfway through, the bill may split one engineering task across several unrelated line items.
Current model releases are starting to frame efficiency around the whole agent loop. Meta's Muse Spark 1.3 announcement, for example, claims fewer tokens and tool calls alongside better long-horizon coding behavior. Those are vendor claims, not independently reproduced results here. More importantly, fewer calls are only useful if the run still reaches the acceptance boundary.
A loop that stops early because it gave up is not efficient. Neither is a loop that says "done" before the test suite disagrees.
The accounting needs three layers:
- consumption: model usage, tool calls, execution time, and isolated compute
- failure load: retries, failed checks, abandoned branches, and recovery work
- outcome: accepted, rejected, or incomplete under named checks
The third layer keeps the first two honest.
Define the task before measuring the agent
Cost per accepted task becomes nonsense if every run gets a different task.
A typo fix with a deterministic test is not comparable to a dependency migration across six packages. A generated frontend with visual and interaction requirements is not comparable to renaming a private function. Averaging them into one number creates a tidy dashboard and a bad purchasing decision.
Define a task envelope before the agent starts:
- repository revision and working state
- task class and goal
- allowed file and command scope
- authority and time limits
- expected artifact
- acceptance checks
- review decision required
Then compare runs inside the same task class. Use the same starting revision when practical. Keep the validation contract stable. If one model gets a precise brief and another gets "fix the app," you are measuring prompt quality as much as model performance.
The acceptance boundary also has to be explicit. A zero exit code may be one check, but it is not a general definition of success. The boundary could include unit tests, type checks, a migration dry run, browser screenshots, an accessibility scan, or human review of a risky diff. Different tasks need different evidence.
This metric is local by design. Your repository, harness, and review policy shape the result. Cost per accepted task is useful for comparing choices inside one engineering system, not for declaring a universal winner.
Put the receipt in the harness
Most teams already have enough raw data to build a useful receipt. The problem is that it lives in separate places: provider usage, terminal logs, worktrees, CI, review comments, and somebody's memory of the failed first attempt.
The harness is where those records can meet.
Alera's README describes separate terminals, Git worktrees, lifecycle hooks, quota tracking, and resource attribution in one agentic development environment. Mivia documents isolated worktrees, durable step records, lifecycle hooks, provider choice, and local workflow files. These are project-authored descriptions, not proof that either tool measures every cost correctly. They do show where task-level accounting can attach to real execution.
A compact receipt could look like this:
task:
id: auth-callback-fix-017
repository_revision: 84c19e2
class: bounded-bug-fix
goal: fix duplicate callback processing
agent:
provider: example-provider
model: example-model
limits:
scope:
- src/auth/callback.ts
- tests/auth/callback.test.ts
time_minutes: 45
allowed_effects:
- edit_worktree
- run_local_tests
forbidden_effects:
- deploy
acceptance:
checks:
- pnpm test --filter auth-callback
- pnpm typecheck
reviewer_decision_required: true
consumption:
input_tokens: recorded_by_provider
output_tokens: recorded_by_provider
tool_calls: recorded_by_harness
execution_duration: recorded_by_harness
isolated_compute_duration: recorded_by_runtime
failure_load:
retries: 1
failed_checks:
- first auth-callback test run
abandoned_worktrees: 0
recovery_steps:
- reverted out-of-scope config edit
human_load:
approvals: 0
intervention: reviewer requested one scope correction
result:
artifact: worktrees/auth-callback-fix-017.diff
status: accepted
The example does not force every field into dollars. Human review may stay qualitative unless the team records time consistently. The point is traceability: one task, its spend, its failures, its evidence, and its final disposition.
Store rejected and incomplete receipts too. If the ledger keeps only successful runs, the apparent cost will improve every time the agent fails.
Failed loops are part of the price
Coding-agent cost discussions often drift toward routing: use a cheaper model for easy work, fall back to a stronger model when needed. That can work. It can also hide a pile of waste behind the word "fallback."
Suppose the first model makes three attempts, expands the diff, and leaves the worktree in an uncertain state. The second model then spends time reconstructing the task and cleaning up before producing the accepted patch. The accepted result belongs to the full chain, not just the final model call.
The same rule applies to parallel agents. More workers can finish independent tasks faster, but they can also duplicate investigation, collide on shared state, or produce several patches that compete for one review slot. Per-session resource attribution matters because concurrency can raise throughput and waste in the same afternoon.
This is where durable steps and lifecycle hooks earn their keep. A step boundary gives the harness somewhere to record spend and check outcomes. A validation hook can stop a broken loop before it starts another expensive revision. An abandoned worktree remains associated with the task instead of disappearing from the success story.
Human intervention belongs in the receipt as well, but do not fake precision. "Reviewer resolved an out-of-scope config edit" is better data than a guessed dollar amount. Over time, repeated interventions reveal where the task contract or harness needs work.
Segment by task class before comparing models
A useful evaluation set should look like the work your team wants agents to do.
For small deterministic edits, acceptance may be cheap: one focused test plus diff review. A migration needs stronger state checks and rollback evidence. Debugging needs a reproduced failure before the patch and a clear signal afterward. Generated frontend work needs browser behavior and visual evidence, not merely code that compiles.
Keep those classes separate. Within each class, select representative tasks, fix the envelope, and compare accepted outcomes. For a narrow frontend study, a collection of Generative UI research papers can help identify domain-specific systems and evaluation ideas worth turning into a task set. The directory is a research surface, not a coding-agent benchmark and not evidence that any model will save money.
Track the distribution, not only the average. One model may be cheap and predictable on bounded edits while occasionally exploding on migrations. Another may cost more per call but fail less often on long tasks. A single blended number hides the difference that should drive routing.
Vendor efficiency claims can feed this process as hypotheses. If a release claims fewer tokens or tool calls, run representative local tasks under the same envelope. Check whether accepted-task cost falls, whether the failure rate changes, and whether review gets easier or harder. The local receipt decides whether the claim matters to your workflow.
Optimize the harness before the token
The cheapest improvement may have nothing to do with model price.
Add stop rules so an agent cannot repeat the same failed approach indefinitely. Bound concurrency so every session has an owner and a budget. Persist step records so a fallback does not need to reconstruct the run from chat history. Trigger validation at meaningful boundaries, not only after a large patch has accumulated.
Execution belongs in the accounting too. Cloudflare's documented Cursor integration separates the agent loop from customer-controlled isolated execution. That architecture does not prove lower cost or better security. It does make the operating boundary visible: commands, repositories, tools, and compute can live outside the model provider, and their cost does not vanish because the token invoice ignores them.
Once the harness can associate consumption, failures, checks, and acceptance with one task, model pricing becomes useful again. You can see where a cheaper model is genuinely economical, where a stronger model reduces recovery, and where the task contract is causing both to waste time.
Until then, a low token bill can be an expensive way to produce work nobody should merge.
Source notes
Top comments (1)
Cost per accepted task is exactly the metric I ended up with after running a fleet of coding agents on a VPS for a few months. The trap I hit first: the invoice splits one logical task across several line items when a run retries or switches models, so per-image and per-request averages look fine while the real number is much worse. Once I started tagging every retry, cold start and human fix-up to the same task id, the 'cheap model' runs I thought were saving money turned out to cost more end-to-end than the pricier ones that finished in one pass.
One thing I'd add: the acceptance boundary has to be automated or the metric collapses. If a human still has to read the patch to decide if it's good, the labour cost dominates the token cost by an order of magnitude and the model choice stops mattering. Named checks first, cheap verification gate, then compare models — in that order.