The unit economics that break between 100 users and 10,000
One documented enterprise deployment ran its proof of concept for about $1,500 a month in API usage. The results looked strong. Leadership approved full production, and the monthly bill at real-world volume landed just over $1 million. That is a 700X jump from pilot to production, and no business case survives a multiplier like that.
The case comes from an analysis of enterprise LLM deployments, and while the number represents a worst case, the mechanics behind it are ordinary. At pilot scale, a cost of $0.10 to $0.50 per agent request is easy to absorb and even easier to present as a savings story. At 10,000 users, the same request rate produces a monthly infrastructure bill that makes the original spreadsheet unrecognizable. Most teams run those numbers after the architecture decision is locked, which happens to be the most expensive possible time to learn them.
The Multiplier Hiding Inside Every Agent Request
A chatbot query triggers one inference call. An AI agent working through a task plans, calls tools, evaluates results, and loops back when something fails. Gartner’s analysis from earlier this year found that agentic workflows consume between 5 and 30 times more tokens per task than a standard chatbot, with a single user request often triggering 10 to 20 separate model calls behind the scenes.
Four mechanics drive that multiplier.
Reasoning loops sit at the center. Every pass through plan, act, and evaluate fires at least one model call, and complex tasks can take dozens of passes before the agent settles on an answer.
Context accumulates. Agents carry system prompts, tool definitions, and step history into every subsequent call. All of it gets re-sent each time, so the token cost of step twelve includes the freight of steps one through eleven.
Tool calls stack their own costs on top. Web searches, database queries, and code execution each add latency and expense beyond the model call that triggered them.
Retries compound everything above. When a tool returns an unexpected schema or an output fails validation, the agent tries again. Each retry is a fresh trip through the loop, and you pay for the attempt whether the task succeeds or fails.
One documented example makes the point better than any abstraction. A coding agent assigned to fix a one-character typo in a README consumed over 21,000 input tokens listing issues, branching, committing, and opening a pull request. A trivial fix, wrapped in an expensive workflow.
Cheaper Tokens, Bigger Bills
Per-token pricing has collapsed. Inference for a GPT-3.5-level model fell from $20 per million tokens in late 2022 to $0.07 by October 2024, roughly a 280x drop in two years, and Gartner projects inference on trillion-parameter models will cost 90 percent less by 2030. Enterprise AI bills keep rising anyway, because total token consumption is growing faster than prices are falling. More capable agents run more reasoning loops, call more tools, and burn more tokens per completed task. Capability and cost move together by design, since quality in these systems comes from iteration rather than single-pass generation.
Uber ran into the same problem at scale. The company rolled out agentic coding tools to roughly 5,000 engineers, and heavy users racked up between $500 and $2,000 per month each, burning through the annual AI budget in about four months. The pilot had only ever tested one engineer, and nobody had modeled what concurrency at that scale would cost.
Run the Production Math Before the Architecture Locks
The forecast that prevents all of this takes about an afternoon to build. Start with cost per completed task from your pilot data, repriced at full production rates rather than free-tier or discounted credits. Multiply by the ratio of production users to pilot users. Then apply a burstiness factor of 3 to 5x, because production traffic spikes and runs in parallel in ways a pilot never exercises. If the resulting number breaks the business case, a spreadsheet is a far cheaper place to find out than an invoice.
Cost per successful task is the metric worth anchoring on. Cost per prompt and cost per session both hide failure. An agent that completes tasks cheaply but fails half the time and requires human cleanup costs far more than its dashboard suggests, and that gap stays invisible until you measure completion rather than activity.
In my work with client teams, the forecast conversation almost never happens at this stage. The pilot generates momentum, the demo impresses the steering committee, and the architecture gets approved on pilot economics. Everything downstream inherits that assumption.
Decide Which Steps Actually Need an Agentic Loop
Autonomous reasoning is the most expensive pattern in the stack, and most workflows only need it in a few places. Research on enterprise deployments suggests small language models can handle 60 to 80 percent of agent tasks at 10 to 30 times lower inference cost, with frontier models reserved for the steps that require genuinely complex reasoning.
A routing layer that classifies each step and sends classification, formatting, and retrieval work to smaller models can cut costs by 60 percent or more without touching quality where it matters. Caching does similar work on the input side. If the agent starts every task with the same system prompt and knowledge base, prompt caching can reduce input costs by roughly 90 percent. Retry caps close the remaining leak. Classify errors so some warrant a retry, some escalate to a human, and some fail gracefully before the loop becomes a line item.
The work is unglamorous. Walk the workflow step by step and decide where iteration earns its cost.
Build Cost Governance Before the Quarterly Surprise
Gartner predicts that more than 40 percent of agentic AI projects will be cancelled by the end of 2027, with escalating costs and unreliable outputs as leading causes. Most of those cancellations will trace back to the same sequence. The pilot succeeds, the deployment scales, the bill arrives, and the cost conversation happens under pressure with a CFO reading line items aloud.
Governance moves that conversation earlier, where it costs almost nothing. Track spend per task, per step, and per tool call, since aggregate metrics hide the one workflow that’s eating the budget. Set alerts on cost per successful task so anomalies surface in days instead of at quarter close. And assign an owner, because a cost that’s technically everyone’s job doesn’t get caught by anyone.
…
Nick Talwar is a CTO, ex-Microsoft, and a hands-on AI engineer who supports executives in navigating AI adoption. He shares insights on AI-first strategies to drive bottom-line impact.
→ Follow him on LinkedIn to catch his latest thoughts.
→ Subscribe to his free Substack for in-depth articles delivered straight to your inbox.
→ Watch the live session to see how leaders in highly regulated industries leverage AI to cut manual work and drive ROI.
Top comments (0)