Method article. The example uses a synthetic log. It is not a benchmark, and nothing here is a report of my own measurements.
Dashboards report what a call or a run costs. That answers a question nobody actually asks. What you pay for is finished work, and finished work is paid for by the attempts that went nowhere as well.
So here is a second number: total measured cost of all attempts divided by the number of tasks that met your success criterion. Call it cost per successful task. It puts failed attempts back into the arithmetic, and it lets you compare two workflows on the same fixed set of tasks. It will not tell you whether the work was worth doing or whether the agent saved anyone time. Different questions.
Fix the unit before you start
- Task. A distinct piece of work with a defined scope and a success criterion. Keep the boundaries consistent if you plan to compare workflows, because most of the disagreement about this metric comes from where you cut tasks.
- Attempt. One agent run, from the moment it starts until it stops or you stop it. A task can take several.
- Successful task. A distinct task that meets the criterion by the end of your measurement window. Count it once, however many attempts it took. The criterion might be a test suite plus a review against the requirements, because passing tests is not proof that the work is right.
- Measured attempt cost. Whatever you are including: API usage, separately billed tool calls, and so on. State what you leave out.
Settle the task set, the criteria, the retry policy and the window in advance. Failed, stopped and abandoned attempts all cost money, so they belong in the numerator. An attempt that produced useful context for the next run is still unsuccessful by this definition. That does not make it worthless, and it is worth saying so out loud, because someone will accuse the metric of calling all retries waste.
The formula
cost per successful task = total measured cost of all attempts
/ number of distinct successful tasks
Two cases where people go wrong.
Zero successful tasks means the metric is undefined. Report the total cost and the zero. Do not report a cost of zero, which reads like the work came free.
Unknown cost on some attempts means you do not have a complete numerator. Report coverage and, if you want a figure anyway, label it partial. A parser that treats missing cost as zero will produce a confident number that is not true.
A worked example
The log below is synthetic. It shows the arithmetic and nothing about real prices. The same file ships with the companion repo as sample.csv, so you can reproduce every figure:
python3 parse.py sample.csv
Twelve tasks, sixteen attempts, ten of them passing the criterion. Costs vary per attempt, as they do in any real log.
| Task | Attempts | Attempt costs | Result by end of window |
|---|---|---|---|
| 1 | 1 | $0.2410 | Success |
| 2 | 2 | $0.2860 failed + $0.2600 success | Success |
| 3 | 1 | $0.1980 | Success |
| 4 | 3 | $0.4120 + $0.3890 failed + $0.2940 success | Success |
| 5 | 1 | $0.1620 | Success |
| 6 | 1 | $0.4780 failed | Not successful |
| 7 | 1 | $0.2140 | Success |
| 8 | 1 | $0.1830 | Success |
| 9 | 2 | $0.3410 failed + $0.2380 success | Success |
| 10 | 1 | $0.1760 | Success |
| 11 | 1 | $0.4310 failed | Not successful |
| 12 | 1 | $0.2260 | Success |
| Measure | Calculation | Result |
|---|---|---|
| Total measured cost | sum of all 16 attempts | $4.5290 |
| Average cost per attempt | $4.5290 / 16 | $0.2831 |
| Cost per successful task | $4.5290 / 10 | $0.4529 |
| Share of spend on attempts that failed | $2.3370 / $4.5290 | 51.6% |
| Retry tax: spend on tasks that never succeeded | $0.9090 / $4.5290 | 20.1% |
| Cache share of input tokens | cached / total input | 84.8% |
One counting detail worth spelling out in your own write-up, because two people quoting "attempts per success" will otherwise disagree: you can divide all attempts by successful tasks (16 / 10 = 1.6), or only the attempts spent on tasks that eventually succeeded (14 / 10 = 1.4). Pick one and say which.
The two cost figures answer different questions. Cost per attempt describes an average run. Cost per successful task spreads everything, including the spend on tasks that never got solved, across the ones that did. It is useful across a task set, and it is not the bill for any individual success.
Of the two failure readings in the table, the larger one is the more honest. Half the spend went to attempts that failed, even though several of those tasks were solved later by another attempt. The retry tax is the sharper question: what did outright failure cost me. Neither is a measure of waste. Any of those failed attempts may have handed the next one the context it needed, and both numbers move depending on where you draw the attempt boundary.
What to log
A minimal attempt-level record:
task_id, attempt_no, attempt_outcome, task_final_outcome,
input_tokens, cached_input_tokens, output_tokens,
tool_cost_usd, duration_min, measured_cost_usd
One row per attempt, same task_id for attempts on the same task. Record the final task outcome consistently and count it once in the denominator.
Say whether cached tokens are included in the input token total, because provider fields and billing rules differ and a reader will not know which convention you used. Alongside the log, record the model, the agent and versions, the date, the criteria, the pricing basis, the retry policy and any usage limits you hit.
Start with tasks you can describe and evaluate consistently. A hand-picked set of a dozen tells you how your own workflow behaves. It cannot rank models, so do not write as if it does, and report how many tasks of which type you used rather than only the final ratio.
Costs that stay outside the headline
A monthly subscription is not a per-token price. Report it separately, or allocate a stated share of it to the task set and explain what else shared that fee. The result is an estimate, not the marginal cost of a run.
Human time is the part most write-ups quietly drop. Track supervision, review and corrections. Include it in the monetary total only with a stated hourly rate; otherwise report minutes, and compare against a sensible alternative before claiming anything about time saved.
Infrastructure and other tooling: include the charges when you know them, or list them as excluded. An API-only total is not the cost of delivering the work.
Caching deserves a warning of its own. Cached input can be billed differently from uncached input, and providers may also charge for cache writes, so a cache-hit percentage alone does not let you reconstruct the bill. For context, a Stanford Digital Economy Lab working paper looked at token consumption across eight frontier models on SWE-bench Verified and found input tokens rather than output tokens driving cost, with runs on the same task differing by up to 30x in total tokens. That is one evaluation on one benchmark, so do not assume it transfers to your setup. It is a decent reason to measure your own.
What this number cannot tell you
- A passed criterion is a proxy. Tests and review miss defects, and requirements are often incomplete.
- Tasks differ in difficulty and value, so a lower aggregate can just mean an easier task mix rather than a better workflow.
- Prices, agent versions, prompts, retry limits and environments change. Write them down before comparing two periods.
- Small samples wander. Report the counts and the costs, not only a precise-looking ratio.
- Cost says nothing about quality, productivity, security or business value. Put time and quality next to it.
If you want a comparison worth anything, hold the task definitions, criteria, cost coverage and window fixed across the alternatives. Even then, a dozen tasks and run-to-run variation keep the conclusion narrow.
Further reading
- METR: measuring AI ability to complete long software tasks on task difficulty and completion.
- NIST CAISI: examples of cheating in agent evaluations on how access to solutions distorts an evaluation.
- OpenAI: separating signal from noise in coding evaluations, an audit of problems in SWE-bench Pro tasks. A vendor analysing a benchmark, not a validation of anything above.
- Stanford Digital Economy Lab: how do AI agents spend your money? on token consumption in agentic coding tasks.
Over to you
If you already measure agent work, which part is hardest to account for: failed runs, human review, subscription costs, or deciding whether a task is genuinely finished? I would also take a counterexample, or a different accounting method, and if you have a rough cost per successful task from your own setup, I want to see how it compares.
Top comments (0)