Teams are cutting AI bills by switching to cheaper models, shrinking prompts, and watching token dashboards. For chatbots, that works. For agents, it often makes the real cost go up. An agent's cost includes retries, tool calls, sandboxes, human review, rework, and the failures nobody counted. Here is the cost model I would use instead, a worked example of a cheaper model that costs more, and the telemetry you need to measure what an agent outcome actually costs.
Research checked November 7, 2026.
The Token Bill Fell 75 Percent. The Cost Went Up.
Imagine a platform team running a customer-support agent.
It resolves refund and billing tickets end to end: reads the ticket, checks the order system, applies policy, issues the refund or explains why not, and closes the ticket. It runs on a premium model.
Finance asks the team to cut AI spend. The team tests a budget model that is five times cheaper per token. In a demo, it handles the common tickets well. They switch.
The next month, the token bill falls by three quarters. The team posts the dashboard in the leadership channel.
A quarter later, the support operations lead shows up with a different spreadsheet.
Escalations to human agents have roughly quadrupled. Reopened tickets are up. A handful of refunds were issued that should not have been, and a handful were denied that should have been approved. Each of those turned into a manual review, a customer apology, and sometimes a goodwill credit.
When finance adds human handling time and rework to the token bill, the cost to resolve one ticket has more than doubled.
Every AI dashboard showed savings. The business paid more.
Nothing in this story requires a bad model or a careless team. The budget model was fine on easy tickets. The team optimized the number they could see, and the number they could see was the wrong one.
Token price tells you what a model call costs.
It does not tell you what an outcome costs.
Token price measures the cost of a model call. The number that matters for agents is the fully loaded cost per verified successful task, and it often ranks models in the opposite order from the price sheet.
TL;DR
- Tokens are a cost input, not a cost metric. An agent task includes model calls, tool fees, sandbox time, retrieval, retries, human review, rework, and incident costs. Tokens are one line of that.
- Agents multiply token usage. Anthropic reported that agents typically use about 4x more tokens than chat, and multi-agent systems about 15x more. The multiplier is not waste by definition; on its research evaluation, token usage alone explained 80 percent of the performance variance.
-
Failed attempts cost money too. If an attempt costs
cand succeeds with probabilityp, the expected spend per success is roughlyc / p, before you pay for the failures that escape. - Escaped failures dominate. The most expensive outcome is not a failed run. It is a run that reports success, is accepted, and turns out to be wrong.
- A cheaper model can produce a more expensive outcome. In the worked example below, a model five times cheaper per token cuts the token bill by 75 percent and more than doubles the cost per resolved ticket.
- Cheap models win when failure is cheap and detectable. For low-stakes routing and classification, the budget option is often the right one. For consequential work, verification quality decides the economics.
- Routing to cheaper models is a verification problem. A cascade that tries a cheap model first only pays off if you can reliably detect when it failed.
- More reasoning is not automatically better. Princeton's Holistic Agent Leaderboard ran 21,730 agent rollouts across 9 models and 9 benchmarks and found higher reasoning effort reduced accuracy in the majority of runs.
- Measure per task, not per call. Every cost event needs a task ID, and every task needs a verified outcome. Without that join, you are reporting spend, not unit economics.
- Watch the tail, not the average. A few runaway loops can cost more than thousands of normal tasks. Track p95 and p99 cost per task and cap them.
- Optimize first-attempt success before token price. Better context, better tools, and early failure detection usually reduce cost per success more than switching models does.
First, Separate Price, Spend, and Cost per Outcome
AI cost conversations mix several different numbers. They answer different questions.
| Metric | What it measures | Who it helps | What it hides |
|---|---|---|---|
| Token price | Vendor list price per million tokens | Procurement | Everything about how many tokens a task needs |
| Cost per call | One model request | Developers tuning prompts | Retries, tools, and multi-step loops |
| Cost per attempt | One full agent run | Engineers comparing designs | Whether the run worked |
| Cost per task | All attempts for one piece of work | Platform owners | Quality and downstream cost |
| Cost per successful task | Total cost divided by tasks that succeeded | Product and engineering leads | Errors that looked like successes |
| Cost per verified outcome | Fully loaded cost divided by outcomes confirmed correct | Finance and business owners | Very little, if verification is real |
These are different things, and the distinction matters.
Token price is the easiest number to get and the least informative. It is set by the vendor and does not depend on your workload.
Cost per verified outcome is the hardest number to get and the most informative. It depends on your tasks, your tools, your verification, and your people.
Most AI cost dashboards stop at the second row.
Why Agents Break Token Accounting
A chatbot turn is roughly one request and one response. Token cost per conversation is a reasonable proxy for cost.
Agents are different in five ways.
1. They use many more tokens per unit of work
Anthropic's engineering write-up on its multi-agent research system, published in June 2025, reported that "agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats."
That is not necessarily bad. In the same analysis of its research evaluation, Anthropic found that "token usage by itself explains 80% of the variance" in performance. Spending more tokens often bought better results.
The post's conclusion is the right economic framing: multi-agent systems require "tasks where the value of the task is high enough" to justify the cost.
That sentence contains the entire argument of this article. The question is never "how many tokens?" It is "what was the outcome worth, and what did it cost to get?"
2. Context grows on every step
An agent loop typically re-sends the conversation, tool results, and instructions on every step. Without caching or compaction, total input tokens for a task grow much faster than the number of steps. A 40-step task is not 4 times as expensive as a 10-step task. It can be far more.
Prompt caching and context compaction change this curve dramatically, which is why they belong in any agent cost model.
3. Reasoning tokens are real tokens
Reasoning models spend tokens thinking before they answer. Providers typically bill those tokens, even when you do not see them. Two runs with identical visible output can cost very different amounts.
4. Most of the cost is not in the model
A realistic agent task can include:
- model calls, including retries and sub-agents;
- paid tool and API calls;
- sandbox or browser compute time;
- retrieval and vector search;
- verification runs, such as tests or a second model;
- human review and escalation;
- rework when the output is wrong; and
- incident costs when a wrong output reaches a customer.
Only the first line appears on a token dashboard.
5. Failures cost money and produce nothing
A run that loops for 30 steps and gives up spent real money. A run that produces a wrong answer that a human has to fix spent real money twice. Token dashboards count both as usage. Neither is counted against the outcomes it failed to produce.
Tokens are a cost input.
They are not a cost metric.
The Equation That Matters
The metric I would standardize on is fully loaded cost per verified successful task:
model + tools + compute + retrieval + review + rework + incidents
cost per success = -------------------------------------------------------------------
number of tasks verified as successful
Two parts of this definition do most of the work.
"Fully loaded" means the numerator includes everything the task consumed, including human time and the cost of fixing mistakes.
"Verified" means the denominator only counts tasks confirmed correct by something other than the agent's own claim: tests passing, a reviewer approving, a customer not reopening, a reconciliation matching.
Why c / p is the first thing to check
Suppose one attempt costs c and succeeds with probability p. If you retry until success, the expected number of attempts is 1 / p, so the expected spend per success is:
expected cost per success ~ c / p
A $0.20 attempt that succeeds 40 percent of the time costs about $0.50 per success. A $0.40 attempt that succeeds 90 percent of the time costs about $0.44 per success.
The model with double the price is cheaper per outcome.
Why escaped failures matter even more
The c / p formula assumes you can tell when an attempt failed. In practice, some failures look like successes.
Call the rate of accepted-but-wrong outputs the escape rate. Each escape costs whatever it takes to discover and fix the problem downstream: a reopened ticket, a reverted deployment, a corrected invoice, an apology.
Escapes are usually far more expensive than attempts. A model with a slightly higher escape rate can lose all of its token savings on a few bad outcomes.
Worked Example: The Cheaper Model That Cost More
Here is the opening scenario with numbers. These are illustrative assumptions, not vendor prices, chosen to be realistic for a support workflow. Use your own numbers; the structure is what matters.
The workflow. Each ticket gets up to two automated attempts. Automated checks catch most failures. If both attempts fail, a human resolves the ticket. Some wrong outputs pass the checks and are discovered later.
Shared assumptions:
- tools and compute cost $0.05 per attempt;
- a human resolving an escalated ticket costs $8.00; and
- an escaped wrong outcome costs $40.00 to discover, fix, and compensate.
Model-specific assumptions:
| Premium model | Budget model | |
|---|---|---|
| Tokens per attempt | $0.60 | $0.12 |
| Success rate per attempt | 85% | 55% |
| Escape rate among accepted outputs | 1% | 4% |
Now compute the expected cost per ticket.
| Cost component | Premium model | Budget model |
|---|---|---|
| Expected attempts per ticket | 1.15 | 1.45 |
| Token cost | $0.69 | $0.17 |
| Tools and compute | $0.06 | $0.07 |
| Escalation to a human (both attempts fail) | 2.25% x $8 = $0.18 | 20.25% x $8 = $1.62 |
| Escaped wrong outcomes | ~0.98% x $40 = $0.39 | ~3.19% x $40 = $1.28 |
| Fully loaded cost per resolved ticket | $1.32 | $3.14 |
The token line fell from $0.69 to $0.17, a 75 percent reduction.
The cost per resolved ticket rose from $1.32 to $3.14, an increase of about 138 percent.
Every ticket still gets resolved, by the agent or a human, so the comparison is fair. The budget model simply moves cost out of the token bill and into human escalation and rework, where AI dashboards do not look.
What would make the budget model win?
Change the assumptions and the answer changes:
- If escalations were cheap, for example a $0.50 automated fallback, the budget model would look much better.
- If escaped errors cost $1 instead of $40, as in internal tagging or low-stakes routing, the budget model would win.
- If the budget model's success rate were close to the premium model's, the price difference would dominate.
That is the real lesson. The cheaper model wins when failure is cheap and detectable. It loses when failure is expensive or hidden.
What about a cascade?
A popular design is to try the cheap model first, check the result, and escalate to the premium model only on failure.
Using the same numbers, with a $0.01 verification check per attempt, a cascade costs about $1.17 per ticket before counting the budget model's escaped errors. To beat the premium model's $1.32, the budget model's escape rate must fall from 4 percent to under about 0.7 percent.
In other words, the cascade only works if verification catches almost every budget-model failure before it is accepted.
Routing to cheaper models is not a pricing decision.
It is a verification problem.
This is where decision models such as Jev, which I covered in a previous article, become interesting. A fast, cheap, calibrated check on "did this output satisfy the criteria?" is exactly the component that makes cascades economical. The check has to be good enough to cut the escape rate, and you have to measure that on your own data.
What the Research Says About Cost and Accuracy
The idea that agent evaluation should include cost is not new. It has been building in the research community for two years.
"AI Agents That Matter" (2024)
Sayash Kapoor and colleagues at Princeton argued in 2024 that agent benchmarks had a "narrow focus on accuracy without attention to other metrics," which encouraged needlessly complex and costly agents. Their proposal: "Our focus on cost in addition to accuracy motivates the new goal of jointly optimizing the two metrics."
That is the research version of this article's thesis. Accuracy alone is not a sufficient objective. Neither is cost alone.
The Holistic Agent Leaderboard (2025, ICLR 2026)
The same group built the Holistic Agent Leaderboard (HAL) to report cost alongside accuracy. Its paper describes "21,730 agent rollouts across 9 models and 9 benchmarks" across coding, web navigation, science, and customer service, "with a total cost of about $40,000."
One finding is directly relevant to anyone turning up reasoning settings to improve quality: the authors report "higher reasoning effort reducing accuracy in the majority of runs."
That does not mean reasoning never helps. It means reasoning effort is a parameter to measure per task, not a dial to turn up by default. More thinking costs more and, on many tasks, did not buy better results.
The business consequence
Gartner predicted in June 2025 that over 40 percent of agentic AI projects will be canceled by the end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. That is a forecast, not a count, and it is broader than cost alone.
It still describes the failure mode in this article. A project that cannot show its cost per outcome cannot show its value, and a project that cannot show its value is the first one cut.
What Token Dashboards Hide
Even teams that track AI spend carefully tend to miss the same things.
The tail
Average cost per task looks fine while a small number of runs loop for hundreds of steps. One runaway agent can cost more than thousands of normal tasks. Track p95 and p99 cost per task, and set hard caps.
Unattributed spend
If a model call is not tagged with a task ID, it cannot be assigned to an outcome. Spend without attribution can only be reported, not managed.
Failures counted as zero
Dashboards count the tokens a failed run used. They rarely charge that spend against the outcomes it failed to produce. A system with a 50 percent failure rate effectively doubles its cost per success, and the dashboard shows the same token total.
Human time
Review queues, escalations, and rework usually live in a different budget. Moving work from the model to people looks like savings on the AI line.
Self-reported success
If the agent decides whether it succeeded, your denominator is inflated by every confident mistake. Success must be verified by something independent of the agent.
Cost shifted to other teams
A support agent that issues wrong refunds creates cost in finance. A coding agent that ships subtle bugs creates cost in on-call. Those costs belong in the agent's unit economics, even if they land in someone else's budget.
Goodhart's law
When token count becomes the target, teams cut context, remove verification steps, and lower reasoning effort. Token usage falls. Success rate falls faster.
The Production Architecture: Cost per Outcome Telemetry
You cannot manage cost per outcome without a way to join costs to outcomes. The architecture is simple; the discipline is in applying it everywhere.
+--------------------+ every event tagged with taskId, workflow,
| Agent Runtime | agentRelease, model, step
| model calls |-------------------------------+
| tool calls | |
| sandbox / browser | v
| retrieval | +---------+----------+
+--------------------+ | Cost Event Stream |
| tokens (in, out, |
+--------------------+ | cached, reasoning)|
| Human Review Queue |-------------------->| tool fees, compute|
| minutes, decisions | | review minutes |
+--------------------+ +---------+----------+
|
+--------------------+ v
| Outcome Verifier | +---------+----------+
| tests, approvals, |-------------------->| Task Cost Ledger |
| reopen checks, | outcome + method | join by taskId |
| reconciliation | +---------+----------+
+--------------------+ |
v
cost per verified outcome by workflow,
model, release, step, and customer tier
1. Give every task an ID and tag every cost
Every model call, tool call, sandbox session, retrieval query, and human review action should carry the same taskId. Include the workflow, agent release, model, and step number.
If you already use distributed tracing, the task ID can be the trace ID or a stable attribute on the root span.
2. Record tokens in their billing categories
Input, output, cached input, cache writes, and reasoning tokens are often billed at different rates. Store them separately, and compute cost from the price table that was in effect at the time.
3. Record outcomes with a verification method
An outcome without a verification method is a claim.
{
"taskId": "task_01JD2M...",
"workflow": "support.refund_resolution",
"agentRelease": "support-agent@4.2.0",
"attempts": 2,
"models": ["premium-model-2026-09"],
"cost": {
"tokensUsd": 1.14,
"toolsUsd": 0.04,
"computeUsd": 0.06,
"humanReviewMinutes": 0,
"humanReviewUsd": 0.00,
"totalUsd": 1.24
},
"outcome": {
"status": "resolved_by_agent",
"verifiedBy": "policy_check+no_reopen_7d",
"verifiedAt": "2026-11-14T09:00:00Z",
"escaped": false
}
}
Note no_reopen_7d. Some outcomes can only be verified after time passes. Your cost-per-outcome report should be finalized only when verification windows close.
4. Cap the tail per task
Give each task a budget: maximum steps, maximum tokens, maximum spend, and maximum wall-clock time. When a cap is hit, stop and escalate, rather than letting the agent keep trying. The same task-scoped budget idea I described for agent payments applies to model spend.
5. Report cost per verified outcome as the headline metric
Put it on the same dashboard as success rate and escape rate, broken down by workflow, model, and release. Token totals become a supporting chart.
The Levers That Actually Reduce Cost per Success
Once you measure the right number, the optimization priorities change. I would work through these levers in roughly this order.
1. Raise first-attempt success
Better context, better tools, clearer instructions, and fewer ambiguous steps reduce retries and escalations. This is often the largest lever, and it rarely shows up as a token saving.
2. Detect failure early
A failure caught at step 3 is cheaper than a failure caught at step 30, and far cheaper than one caught by a customer. Add checks between steps, not just at the end.
3. Reduce the escape rate
Invest in verification for consequential outputs: tests, policy checks, reconciliations, and calibrated decision checks. Every escape avoided is worth many attempts.
4. Route by measured difficulty
Use a cheaper model where your own data shows it succeeds, and a stronger model where it does not. Route by task type and confidence, not by a global default.
5. Use caching and compaction
Cache stable prefixes such as system instructions and tool definitions. Compact long histories. These reduce cost without reducing quality, which makes them unusually safe optimizations.
6. Tune reasoning effort per step
Measure whether extra reasoning improves success for each step type. HAL's results suggest that, for many tasks, it does not.
7. Move small decisions to small models
Routing, classification, and pass-fail checks do not need a frontier model generating paragraphs. A decision model or a small model with structured output is often enough.
8. Then, and only then, negotiate token price
Price matters. It just matters last, after you know how many tokens a successful outcome needs.
A Practical 30-Day Plan
Week 1: Pick one workflow and define success
- Choose one agent workflow with clear volume and business value.
- Define what a verified successful outcome means and how it is checked.
- Define the verification window, such as "no reopen within seven days."
- List every cost category the workflow touches, including human time.
Week 2: Instrument the join
- Tag every model call, tool call, compute session, and review action with a task ID.
- Store tokens by billing category.
- Emit an outcome record with the verification method.
- Build a first report: cost per verified outcome, success rate, escape rate, and p95 cost per task.
Week 3: Find the expensive outcomes
- Rank tasks by total cost and inspect the top 1 percent.
- Classify failures: wrong context, tool errors, loops, ambiguous instructions, model limits.
- Estimate the cost of escapes using real rework data.
- Add per-task caps for steps, tokens, spend, and time.
Week 4: Run one controlled experiment
- Change one lever, such as a better tool, an earlier check, a cascade, or a different model for one step.
- Compare cost per verified outcome, not token totals.
- Keep the change only if cost per verified outcome falls without raising the escape rate.
At the end of 30 days, you will have one workflow where you can say what a successful outcome costs, where that cost goes, and which change actually lowered it.
That is worth more than any token dashboard.
Seven Questions to Ask About Any Agent's Cost
- What does one verified successful outcome cost, fully loaded?
- How is success verified, and by what other than the agent itself?
- What is the escape rate, and what does one escape cost?
- What share of total cost is human review and rework?
- What are the p95 and p99 costs per task, and what caps them?
- If we switched to a cheaper model, which line items would rise?
- Which single change would most improve first-attempt success?
If the answer to the first question is a token total, the team is measuring spend, not economics.
Final Take: The Cheapest Token Is Not the Cheapest Outcome
Token prices will keep falling. That is good news, and it will make many agent workflows economical that are not today.
It will also tempt teams to keep optimizing the wrong number.
An agent is not a sequence of model calls. It is a worker that attempts tasks, sometimes fails, sometimes asks for help, and sometimes gets things wrong in ways nobody notices until later. Its cost includes all of that.
The teams that win on agent economics will not be the ones with the lowest token bill. They will be the ones who know exactly what a successful outcome costs, where that cost comes from, and which change makes it cheaper without making it worse.
Count tokens if you like.
Just do not confuse them with cost.
The unit of AI economics is not the token.
It is the outcome you can prove was done right.
Sources and Further Reading
- Anthropic Engineering, How we built our multi-agent research system (June 13, 2025): https://www.anthropic.com/engineering/multi-agent-research-system
- Kapoor, Stroebl, Siegel, Nadgir, and Narayanan, AI Agents That Matter (July 2024): https://arxiv.org/abs/2407.01502
- Kapoor, Stroebl, Kirgis, et al., Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (October 2025; ICLR 2026): https://arxiv.org/abs/2510.11977
- Holistic Agent Leaderboard (live results): https://hal.cs.princeton.edu/
- Gartner, Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (June 25, 2025): https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
- Bain & Company, FinOps for AI: From Managing Costs to Maximizing Value: https://www.bain.com/insights/finops-for-ai-from-managing-costs-to-maximizing-value/
- TechTarget, How to apply FinOps to optimize agentic AI costs: https://www.techtarget.com/searchenterpriseai/tip/How-to-apply-FinOps-to-optimize-agentic-AI-costs
About the Author
Suraj Khaitan - Gen AI Architect | Building scalable platforms and secure cloud-native systems
Connect on LinkedIn | Follow for more engineering and architecture write-ups
How does your team measure AI cost today: token totals, cost per call, or cost per outcome? And has a "cheaper" model ever ended up costing you more?
Top comments (0)