Foundation model API spending hit $8.4 billion globally in 2025 and is tracking toward $15 billion in 2026, but the number that matters to any individual team is smaller and stranger: most production AI applications waste 40 to 70 percent of their token budget without anyone noticing. The waste is invisible in development, where conversations are three turns long and the retrieval index has forty documents in it. It only shows up when real usage arrives, and by then it is not a billing problem, it is an architecture problem.
The escalation pattern is consistent enough to set your watch by. Prototype costs under $50 a month. Pilot runs $500 to $2,000. Then production lands and the bill goes up 10x to 50x in one quarter. Nothing broke. The same design that was cheap at low volume is expensive at high volume, because the cost curve is steeper than any other infrastructure line most developers have dealt with.
Where The Token Budget Actually Goes
Language model APIs charge per token, and tokens accumulate in places that do not appear in any code review. Every character of the system prompt is sent with every single request. Every message in the conversation history is resent with every follow-up, so a twenty turn conversation pays for turn one twenty times. Every retrieved document chunk is included in full even when one paragraph of it was relevant. Every tool definition is repeated in every call even when the model uses none of them.
None of these are bugs. Each one is the obvious implementation, and each one is a multiplier. The useful exercise is to instrument a single real request and break it into four buckets: system prompt, history, retrieved context, and tool schemas. Most teams find that the part they were actually thinking about, the user's message, is a small fraction of what they paid for.
Caching And Model Routing Are The Quick Wins
Prompt caching pays first because the expensive part of most requests is also the most stable part. System prompts and tool schemas sit at the front of the request and rarely change between calls, which is exactly the shape caching wants. Moving stable content to the front of the prompt and keeping it byte identical across calls is usually an afternoon of work.
Model routing pays second and pays bigger. Pulling a date out of a receipt does not need the model that writes your hardest code, but most applications send everything to one endpoint because that is simpler to build. A small classifier in front, sending only genuinely hard requests to the expensive model, tends to cut more off a bill than any other single change. The recent harness benchmarks made this vivid: same model, same tasks, and cost per pass ranging from $1.05 to $18.34 depending entirely on what wrapped the model.
Batching is the third quick win, and it applies wherever work does not need to be interactive. Overnight enrichment, classification backfills and evaluation runs all tolerate latency, and most providers price that tolerance generously.
Memory Is The Architectural Fix
Caching and routing make each call cheaper. Memory changes how many tokens a call needs at all. If the things a model must remember across a conversation live in persistent storage and are retrieved by relevance, you stop replaying the entire transcript on every turn. Cost stops scaling with conversation length and starts scaling with how much is actually relevant right now, which is a much flatter curve.
This is also the fix that improves quality rather than trading against it. Resent history is not just expensive, it is noisy, and long contexts degrade attention on the parts that matter. Retrieving five relevant facts beats replaying two hundred turns on both axes. The full breakdown of these strategies, including the monitoring side, is written up in this AI cost optimization guide, and the caching mechanics specifically are covered in more depth under LLM caching.
What To Measure
A single monthly total tells you nothing actionable. Cost per feature, cost per user session and cost per successful outcome are the numbers that let you act, because they tell you which endpoint is eating the budget and whether the spend is producing anything. Teams that track cost per successful outcome rather than cost per call usually discover that their most expensive feature is not the one they assumed, and that a meaningful slice of spend goes to requests that fail and get retried.
The pattern across all of this is the same. AI costs do not spiral because models are expensive. They spiral because the redundancy is invisible until volume makes it loud, and every month you postpone measuring it, the architecture gets harder to change. Instrument one real request this week, split it into those four buckets, and the priority order will pick itself.
Top comments (0)