An architectural guide to reducing AI costs without sacrificing quality, reliability, or business value.
Your AI feature is working. Users are adopting it, response quality looks good, and the team is preparing to scale.
Then the bill arrives.
You inspect the model pricing. Perhaps the team is using a more expensive model than necessary. You switch to a cheaper one, shorten the prompts, and deploy the changes.
The bill comes down, but so does answer quality. Retries increase. Some tasks that previously completed automatically now require human intervention. The team spends the next two weeks recovering the quality it lost.
You optimized the cost of a model call, but not the cost of running the system.
This is the distinction that matters in production AI: the objective is not to minimize token consumption. It is to minimize the cost of delivering a successful outcome at an acceptable level of quality and reliability.
That requires three different kinds of engineering work: measuring where money goes, eliminating unnecessary computation, and controlling costs once the system is in production.
1. Measure before optimizing
Imagine an AI support agent whose monthly API bill has doubled.
Where would you start?
You could compare model prices, reduce the maximum output tokens, or introduce prompt caching. But none of those actions tells you why the bill doubled in the first place.
Perhaps traffic doubled. Perhaps a new feature introduced five additional model calls per conversation. Perhaps a tool now returns thousands of rows when the agent needs only a handful. Or perhaps a retry loop is repeatedly processing the same context.
These problems require different solutions. Without visibility, optimization becomes guesswork.
Start with cost per successful outcome
Consider two implementations of a support agent:
| Metric | Agent A | Agent B |
|---|---|---|
| Cost per attempt | $0.10 | $0.14 |
| Success rate | 70% | 95% |
| Human intervention after failure | $4.00 | $4.00 |
Assume a failed attempt requires human intervention, and the human intervention cost is $4.
For Agent A, the expected cost per task is:
$0.10 + (0.30 x $4.00) = $1.30
Agent B
$0.14 + (0.05 x $4.00) = $0.34
Agent B's model call costs 40% more, yet its expected cost per task is substantially lower because it avoids expensive human intervention.
The exact economics depend on the workload, but the principle is general: a model's price per token does not tell you whether it is the cheaper option for your business.
Measure cost per successful task alongside success rate, quality, latency, retries, and human intervention.
Make the bill explain itself
To identify the source of cost, every model call should be traceable to the work it performs.
At minimum, capture:
- Model, input tokens, output tokens, and cached tokens.
- Feature, tenant, workflow, and environment.
- Number of model calls, tool calls, and retries per task.
- End-to-end latency and task outcome.
- Estimated human intervention and infrastructure costs where applicable.
This turns a vague question like "Why did our AI bill increase?" into a much more useful investigation: "Why did the cost per successful task for this workflow increase by 35% after the last release?"
It also establishes the baseline against which future optimizations can be evaluated.
The architectural decision: Instrument the system before changing it. If you cannot attribute costs and measure outcomes, you cannot confidently determine whether an optimization worked.
2. Reduce unnecessary work
Once you know where the money goes, the next question is what work the system actually needs to perform.
This is where model selection, context engineering, caching, and workflow design come into play. They are not interchangeable techniques; each addresses a different source of waste.
First, question whether the task needs an LLM
Not every task that can be performed by an LLM should be.
Suppose an application receives a request to retrieve an order's delivery status. The system already has the order ID and the delivery status in a database.
An LLM could interpret the request, generate a database query, inspect the result, and formulate an answer. But a deterministic workflow can retrieve the record and format the response without requiring the model to reason through those steps.
Now consider a request to investigate why a delivery is delayed, reconcile conflicting updates, and explain the likely cause. That may genuinely benefit from language understanding and reasoning.
The architectural boundary matters.
Use conventional code for deterministic operations. Introduce an LLM where ambiguity, unstructured information, or reasoning justifies its cost.
A hybrid workflow often gives you the best of both: predictable execution for known operations and model-driven reasoning where necessary.
Next, inspect what the model receives
Agentic systems introduce a cost pattern that is easy to overlook.
An agent makes a model call, executes a tool, receives the result, and calls the model again. The next request often includes previous instructions, conversation history, and tool results.
As the workflow progresses, the context can grow significantly.
Consider a coding agent that retrieves a 20,000-token tool response. The next model call includes that response, and subsequent calls may carry it forward again. If the agent performs ten more steps without pruning the context, the same information can be billed repeatedly as input.
The problem is not necessarily the model's price. It is the amount of unnecessary information being sent to it.
The first fix is often at the tool boundary:
- Return the relevant database rows, not the entire table.
- Extract useful sections from retrieved documents instead of passing every page.
- Truncate verbose logs while preserving the information needed for debugging.
- Summarize older conversation history when its details are no longer required.
- Limit agent iterations and stop workflows when further reasoning is unlikely to improve the outcome.
There is an important trade-off here. Aggressive context reduction can remove information the model needs. Evaluate the changes against representative tasks and monitor quality, not just token counts.
The goal is not the smallest possible context. It is the smallest context that reliably supports the task.
Reuse computation before paying to repeat it
Prompt caching is particularly useful when requests share a large, stable prefix, such as system instructions, tool definitions, or repeated conversation context.
Providers can reuse previously processed input and charge less for eligible cached tokens. OpenAI documents cached-input discounts of up to 90% for supported models, depending on the model and pricing configuration. Actual savings depend on cache hits, prefix reuse, and the workload. OpenAI's prompt caching documentation explains the mechanics.
The architectural implication is that prompt construction matters.
Keep stable instructions and tool definitions consistent. Avoid unnecessary changes to reusable prefixes. Place request-specific information after the stable content where the provider's caching rules permit it.
But do not assume caching is automatically beneficial. A workload with little repeated context may see limited savings, and cache-write costs or retention constraints can affect the economics.
Semantic caching solves a different problem. It reuses answers for sufficiently similar requests, which can work well for repetitive support questions or stable policy information.
However, similar questions do not always have identical answers. Account-specific requests, changing policies, and time-sensitive information need careful handling. Incorrect cache hits can cost more in business impact than the model calls they eliminate.
Choose the model for the workload, not the price list
Once unnecessary calls and context have been addressed, model routing becomes a more meaningful optimization.
A small model may handle extraction, classification, and routine support questions. A more capable model can be reserved for ambiguous requests and complex reasoning.
You can route requests using task categories, deterministic rules, a lightweight classifier, or a measured escalation strategy. But the routing decision needs an evaluation set that reflects real production traffic.
Test whether the cheaper route maintains acceptable quality. Measure retries, escalation frequency, latency, and cost per successful outcome.
A smaller model that fails frequently may erase its per-token savings. Conversely, using a frontier model for every trivial operation may be unnecessary expenditure.
For workloads that do not require immediate responses, batch processing can also reduce costs. Offline evaluations, document classification, and bulk enrichment are common candidates. OpenAI's cost optimization guide covers request reduction, token minimization, model selection, and batch processing.
The architectural decision: Remove unnecessary work first, reuse computation where appropriate, and then choose the least expensive model or execution strategy that meets your quality requirements.
3. Control costs in production
Suppose your team has implemented these optimizations. Cost per task is lower, evaluations pass, and the feature is ready to scale.
Are you ready to roll it out to ten times the traffic?
Not necessarily.
A system can be efficient at its current volume and still become financially unpredictable as usage grows. Traffic composition changes, new features introduce different workloads, retries spike, and a prompt change silently increases token consumption.
Optimization without cost governance is temporary.
Forecast before scaling
A useful starting point for forecasting is:
Monthly AI cost = Number of tasks per month × Average cost per task
The difficult part is estimating the second variable accurately.
Averages can hide substantial differences between simple and complex requests. An agent that performs two model calls on a routine task may perform fifteen on a difficult one.
Break the forecast down by workload type, expected traffic mix, model route, and observed token usage. Include retries and other material costs. Account for caching only when the assumed hit rate is supported by measurements or a defensible pilot.
Then produce best-case, expected, and worst-case scenarios.
A small instrumented pilot helps validate these assumptions before the team commits to a larger rollout. As traffic grows, compare actual spending against the forecast and update the model.
The purpose is not to predict every token perfectly. It is to understand which assumptions drive the bill and how much the result changes when those assumptions change.
Attribute spending and enforce limits
Forecasting tells you what you expect to spend. Attribution tells you who or what is consuming the budget.
Attach metadata to model calls so spending can be grouped by feature, tenant, workflow, model, and environment. This makes it possible to distinguish a product-wide increase from a single tenant's unusually expensive workload.
Next, define controls appropriate to the system:
- Budget alerts at meaningful spending thresholds.
- Per-tenant or per-workflow limits where isolation matters.
- Maximum iteration counts and retry budgets for agent loops.
- Rate limits and concurrency controls to protect provider quotas and infrastructure.
- A mechanism to stop or degrade noncritical workloads when spending exceeds defined limits.
A dashboard alone does not prevent runaway spending. Hard limits must be enforced in the execution path, with safeguards against concurrent requests bypassing a shared budget.
There is also a product decision to make: what happens when a budget is reached? Some workloads can be paused. Others may switch to a cheaper model, defer processing, or request human review. That behavior should be designed in advance rather than improvised during an incident.
Treat cost regressions as engineering regressions
Consider a seemingly harmless change to an agent's system prompt. The evaluation suite still passes, but the prompt now causes the agent to make two additional tool calls on average.
At low traffic, the difference may be invisible. At scale, it can become a significant recurring expense.
Cost should therefore be measured during evaluation and monitored after deployment, just like latency and quality.
Track cost per task and cost per successful outcome across releases. Investigate material changes. Use regression thresholds that reflect the workload, and require review when a change increases cost without a corresponding improvement in quality or business value.
This does not mean every cost increase should block a release. A more capable model may be worth the additional expense if it materially improves outcomes. The purpose is to make the trade-off explicit.
The architectural decision: Make spending attributable, bounded, forecastable, and part of your release criteria before scaling traffic.
The decision that matters most
There is one final question that sits above all three principles:
Does the value of the outcome justify the total cost of delivering it?
The answer must account for more than API usage. Depending on the architecture, it may include infrastructure, observability, human review, maintenance, operational failures, and the cost of incorrect decisions.
For some workloads, a third-party model API remains the best choice because it provides strong capabilities without requiring the team to operate inference infrastructure.
For others, sustained high volume and a sufficiently narrow task may justify evaluating a smaller self-hosted model. But GPU utilization, serving efficiency, reliability, and engineering overhead all influence that decision. There is no universal traffic threshold at which self-hosting becomes cheaper.
And sometimes, the right answer is a deterministic workflow with no LLM at all.
That is not a failure of AI engineering. It is an architectural decision made in the interest of the product.
Conclusion
The teams that manage AI costs well do not begin by asking which model is cheapest.
They establish what success costs, identify the work that contributes nothing to the outcome, and build controls that keep spending predictable as the system evolves.
Measure before optimizing. Reduce unnecessary work. Control costs in production.
Then evaluate the result against quality, reliability, and business value.
The goal is not to build the cheapest AI system. It is to build a system whose cost makes sense for the outcome it delivers, even as usage scales.
Top comments (1)
Cost per successful outcome is the framing I keep coming back to. Per-token pricing tells you nothing when a retry loop or a verbose tool response keeps getting rebilled every turn. Attribution by feature and tenant is where most teams are still flying blind.