You deployed the feature. Usage picked up. Three months later someone opens the OpenAI dashboard and the monthly bill has quietly gone from €200 to €4,000. No single incident, no obvious mistake — just steady accumulation while nobody was watching.
This happens more often than it should, and usually for the same handful of reasons. This article covers where the money actually goes in production API deployments, and what I've found actually moves the number.
One scope note up front. I'm writing about direct API integrations in self-hosted applications — the kind of setup where you own the billing, the inference calls, and the optimization decisions. Not Vercel AI Gateway, not enterprise multi-region inference routing, not large internal ML platform deployments. The context is SMB and mid-market: a SaaS product, a support automation layer, an internal tool — something in the range of a few hundred to a few thousand API calls per day, where the bill matters but you're not yet at the scale where it dominates everything else.
The title says OpenAI because that's the most common starting stack and where most of my hands-on time has been. Most of the principles below apply to Anthropic, Google, and other commercial providers with similar billing models. Cross-provider routing — using DeepSeek for classification, Claude for nuanced generation, Gemini for long-context work, switching providers based on cost or latency — is a real lever in 2026, and I'll address it in its own section near the end. The early sections assume a single-provider stack because that's the situation most teams are starting from when this question becomes urgent.
Where the Money Actually Goes
slug="ai-integration"
text="If your OpenAI bill is growing faster than your traffic, I can audit your integration and identify where the tokens are going. Most overspend comes from three fixable patterns."
/>
The OpenAI bill is made of token counts multiplied by per-token rates. That sounds simple, but the token counts are the part people consistently underestimate until they measure them.
Verbose system prompts repeated on every request. This is usually the first thing I find when I look at someone's integration. A carefully written system prompt — a few hundred words of context, role definition, format instructions, constraints — gets sent with every API call. If you're making 10,000 calls a day, that preamble is sent 10,000 times. At gpt-4.1 input rates, a 500-token system prompt across 10,000 daily calls adds up to costs that aren't negligible. The fix here is prompt caching, which I'll cover below. But the prerequisite is recognizing that you're paying for the same text repeatedly.
Wrong model for the task. This is the most direct lever and also the most commonly overlooked. Routing everything through the most capable model because you're not sure which tasks need it is a straightforward way to multiply your bill by three to ten. In most cases I've seen, the production workload breaks down roughly like this: a large fraction of requests are classification, extraction, or structured data tasks that do not require frontier-model reasoning. They need consistency and reliability, not depth. GPT-4.1 mini handles these well; GPT-4.1 nano handles the simpler ones. Routing the right task to the right model is almost always the highest-ROI optimization you can make, and it's often the one that's deferred because it requires actually categorizing what the system does.
Streaming without tracking actual token usage. Streaming makes the user experience feel fast — responses appear progressively rather than after a long wait. But a common implementation mistake is to stop paying attention to token counts once streaming is enabled, because the usage block in a streaming response requires reassembly from stream_options: { include_usage: true } rather than arriving automatically the way it does in non-streaming responses. Teams that forget this step end up flying blind: they know the feature is used, but they have no per-request cost data, and billing surprises accumulate silently.
Retries that regenerate rather than replay. When a request to the OpenAI API fails with a transient error — a timeout, a 429, a 500 — the right behavior is to retry the same request with an idempotency key, not to generate a new completion. In practice, most simple retry implementations just call the API again, which costs the same amount as the original call. If your retry rate is 5% and you're making a lot of calls, you're paying for completions you may never use. Idempotency in API calls is a broader topic I'll cover separately, but the short version is: track whether you already have a successful completion for a given input before deciding to call the API again.
Embeddings rebuilt on minor content changes. If you're running a RAG pipeline and your indexing job rebuilds embeddings for every document on each run, you're paying for embeddings on content that hasn't changed. In the Pikkuna AI chatbot — 1,614 documents across 25 languages — the answer was to hash document content and skip the embedding call when the hash matches the stored value. I described the full ingestion pipeline in Building a i18n RAG Chatbot for E-commerce. It sounds obvious. It's also something I've seen omitted in a surprising number of pipelines, often because the initial implementation was built for a small document set and the per-run cost wasn't visible until the document set grew.
Background jobs without budget caps. A bug in a background processing job — an infinite loop, an off-by-one in a batch size, an unguarded retry in error handling — that triggers API calls can run for hours before anyone notices. The damage is proportional to how long it runs unchecked. A job that makes 10 API calls as intended can make 10,000 if something goes wrong. Project-level budget limits in the OpenAI dashboard exist precisely for this, and they're not enabled by default. More on this under Budget Controls.
Logging full conversation payloads to a database. This one is an indirect cost but it compounds. Storing complete message arrays — including the full system prompt and the entire conversation history — in a database or log aggregator at scale adds up to significant storage and egress costs that don't appear in the OpenAI bill but definitely appear somewhere. Log token counts, latency, and cost metadata. Don't log full message bodies unless you have a specific reason for a specific subset of requests.
Context Growth: The Cost Driver Nobody Inventories
Most discussions of AI cost frame the bill in terms of model selection, caching, and request volume. There's a parallel cost driver that's frequently larger than any of those and rarely inventoried explicitly: how much context each request carries.
Context grows in predictable ways once a system is live. A chat assistant that started with a 200-token system prompt and a single user message accumulates conversation history turn by turn. A RAG pipeline that originally retrieved three documents starts retrieving twelve when someone tunes recall up. An agent loop that originally passed scratchpad notes between steps starts passing the full reasoning trace because debugging was easier that way. None of these changes are wrong; each one is a reasonable local decision that increases per-request token consumption without anyone noticing the cumulative effect.
The patterns worth treating as design decisions rather than defaults:
Bounded conversation history. "Send the whole thread" is the default in most chat implementations, and it's the wrong default once the thread gets long. Cap context at the most recent N turns, or summarize older turns into a compact rolling summary, or do both. The right cap depends on the use case, but no cap is rarely the answer at any meaningful volume.
Retrieval budgets. A RAG pipeline that retrieves twelve documents at 800 tokens each is sending nearly 10,000 tokens of context with every query. If the model only uses two or three of them, you're paying for nine that didn't influence the answer. Set an explicit retrieval budget per query, and tune recall against that budget rather than against an abstract "more is better" intuition. Reranking is usually cheaper than over-retrieving.
Agent memory compaction. Agent loops that pass full reasoning traces between iterations grow context linearly with the number of steps. Compact older steps into terse summaries, drop intermediate observations once the relevant fact is extracted, and treat the scratchpad as a working surface, not a permanent log. The naïve "just append everything" architecture is fine for prototypes and dangerous in production.
Long-context overuse. Models with million-token context windows are useful for specific workloads. They are not a license to send 200,000 tokens of barely-relevant context to questions that would have been answered correctly from a 4,000-token retrieval. Long context is a tool; defaulting to it as a cost-control strategy is the opposite of one.
The frame to hold onto: every byte of context that doesn't influence the answer is overhead you're paying for. Many production systems are paying for a lot of it without anyone having made the decision to.
Model Routing: Which Task Gets Which Model
The model pricing gap between tiers is large enough that routing decisions matter materially.
A reasonable framing: use the smallest model that reliably handles the task. Increase to the next tier only when you've confirmed the smaller model isn't sufficient for your specific inputs — not because it might not be sufficient in general.
The tasks where a smaller model is almost always sufficient: binary classification, entity extraction from structured text, sentiment categorization, single-document summarization, keyword tagging, format conversion (JSON to text, structured field extraction), language detection.
The tasks where you typically do need a more capable model: multi-document synthesis, complex reasoning chains, code generation with non-trivial context, nuanced judgment calls, anything where the output will be shown directly to a user in a high-trust context and errors are expensive.
The tasks where you might need a reasoning model (o-series): problems that genuinely benefit from extended internal reasoning — complex planning, mathematical reasoning, multi-step logical deduction. These are real use cases, but they're not the majority of production workloads, and reasoning models are priced accordingly. If you're using an o-series model for a task that's essentially classification, that's not a performance decision — that's an unreviewed default.
I'm not going to give you precise cost comparisons by model here because they shift frequently and the relevant numbers for your workload depend on your input/output ratio and request volume. What I will say is that the relative gap between tiers is large enough that even approximate routing — "use mini for everything except the user-facing generation step" — produces meaningful savings.
One caveat worth being honest about: a routing layer is itself a product surface that has to be maintained. Provider model lineups change, the accuracy boundary between tiers shifts with each release, and a routing rule that was correct six months ago can quietly underperform today without producing any error you'd notice. Sophisticated routing systems need continuous evaluation against held-out test sets, regression suites when models update, and someone whose job includes watching the accuracy delta. For most SMB stacks, "approximate routing with periodic re-evaluation" beats "elaborate router with no maintenance plan." The cost of the router going stale is invisible until you spot-check it; build in a way that makes spot-checking cheap.
Cost Is Not Just Token Price
Most of this article frames cost as token count times per-token rate, because that's the part that shows up on the invoice. In production it isn't the whole story.
A few systemic costs that don't appear directly in OpenAI billing but dominate the real number:
Retry amplification. A 5% failure rate without idempotency means you paid for 1.05× the completions you needed. A 5% failure rate with aggressive retry policies — three attempts, exponential backoff — means worst case you paid for 3× during the failure window. If the failure was transient on the provider side, that math compounds across every dependent system retrying the same calls.
Accuracy feedback loops. Routing classification to a nano model looks like a 10× cost reduction on paper. If the nano model is wrong 8% of the time and your pipeline escalates failed classifications to a frontier model, your effective cost is somewhere between nano and frontier weighted by the error rate. If wrong classifications produce silent failures that are caught downstream by humans, your effective cost includes that human review time. The cheaper model is genuinely cheaper only when the accuracy delta is below your tolerance for that specific task.
Latency as a cost. A user-facing feature that calls a slower model holds connections open longer, consumes more concurrency budget on your application servers, and reduces overall throughput on the same hardware. A model that costs 20% less per token but adds 800ms of latency can be more expensive than the faster option once you account for the infrastructure it ties up at scale.
Orchestration overhead. Multi-step pipelines — a routing call to decide which model to use, a tool-call cycle that runs three to five iterations, an evaluator that grades each output — each add real token consumption. A naïvely composed agent loop can consume 4-8× the tokens of the underlying generation step, and the math only works if the orchestration genuinely improves outcomes enough to justify the additional spend.
The pattern: simple architectures are often cheaper than smart ones when you total the full cost. Most "AI cost optimization" advice optimizes the wrong layer — it tunes token counts on individual calls instead of asking whether the system architecture multiplies those calls unnecessarily.
Prompt Caching: How It Works and When It Wins
OpenAI's prompt caching reduces costs for repeated identical prefixes at the beginning of a request. If a request has a common prefix of at least 1,024 tokens that matches a recent request, the cached portion is billed at 50% of normal input token rate.
The key constraint is the word "prefix." The cached portion must be at the beginning of the prompt, must be identical (not similar — identical), and must meet the minimum token length. This means the optimization requires structuring your prompts to put stable content first.
The practical pattern: put your system prompt first, then any static context (documents, schemas, examples), then the dynamic per-request content. If your system prompt is 2,000 tokens and your static context is another 1,500 tokens, that's 3,500 tokens of cacheable prefix per request. At any meaningful request volume, this is the most direct path to meaningful cost reduction without changing any business logic.
What breaks caching: inserting dynamic content anywhere before the static portions. A timestamp in the system prompt. A request ID injected at the top. User-specific context placed before the role definition. Any of these prevents caching from activating for that prefix. Audit your prompt structure before assuming you're getting cache hits — OpenAI's usage API returns cached_tokens in the response, and it should be non-zero for your system prompts if caching is working.
The realistic savings when prompt caching is working correctly: 50–75% reduction on input token costs for the cacheable prefix, depending on the ratio of static to dynamic content in your prompts. The higher the static fraction, the better. This is not a small optimization. For systems with long system prompts or extensive static context, this is often the single largest lever available.
Response Caching: Useful in Narrow Cases
Caching API responses is a different technique, and it's worth being precise about when it's appropriate.
Exact-match response caching — storing the response to a given input and returning it without an API call on exact repeat — works reliably for deterministic use cases: FAQ lookups against a fixed document set, structured data extraction where the input is identical, content generation tasks where the input and expected output are stable. If you have a support workflow where the same three questions account for 30% of volume and the answers don't change, caching those responses is straightforward and safe.
Semantic caching — storing responses and returning them for "similar" inputs based on embedding similarity — is fundamentally more dangerous than it looks, and I'd treat it as experimental optimization rather than a recommended default.
The failure modes are not theoretical. Embedding similarity does not preserve intent — two questions that look semantically close can be asking for materially different information, and returning a cached answer to the "wrong" one produces silent errors that are almost impossible to detect from logs. The user sees a response, the request appears successful, no error is raised, and there's no observable signal that the answer doesn't match the question. The cache also degrades silently as your corpus evolves: an answer that was correct three months ago may be wrong now, but the embedding still hits.
I'd consider semantic caching for narrow internal use cases where I've validated that semantically similar queries in that specific domain reliably want the same answer, where the cost of a stale or near-miss answer is low, and where there's active monitoring of cache hit accuracy. For anything customer-facing, anything that touches personalization, or anything where the underlying answer can change over time, I'd default to not using it. The savings rarely justify the failure mode.
Budget Controls: Not Optional in Production
The OpenAI dashboard allows you to set monthly spending limits at the project level. Set them. Set an alert threshold before the hard limit so you get notified before the spend is stopped. These are not fine-tuning — they are the floor of responsible production deployment.
The two controls to configure before anything else touches production:
Project settings → Limits
Soft limit (email alert): ~70–80% of your comfortable monthly budget
Hard limit (API disabled): ~120% of your comfortable monthly budget
The gap between the two gives you time to react to an alert before the hard cap hits. The hard cap is your last line of defense against runaway jobs. It should be set above normal operating ceiling, not at it — you don't want production traffic blocked because you had a traffic spike — but it should be set.
Per-user spending controls, if your product involves users triggering API calls, are a separate concern. The right implementation depends on your billing model: pre-authorization against a credit balance, per-request cost accumulation with a ceiling, or rate limiting at the user level. Which of these is appropriate depends on your product design. The principle is the same: programmatic limits prevent the case where one misbehaving user (or a bug in one user's workflow) creates a disproportionate bill.
Observability: What to Log
The minimum observable state for any production AI integration:
interface AIRequestLog {
requestId: string;
model: string;
feature: string; // which product feature triggered this
promptTokens: number;
completionTokens: number;
cachedTokens: number; // from usage.prompt_tokens_details.cached_tokens
latencyMs: number;
costUsd: number; // computed from token counts × current rates
success: boolean;
errorCode?: string;
}
This is the minimum. It gives you: cost per request, cost per feature, cache hit rate per feature, error rate by model, and latency distribution. From these you can derive almost everything else you'd want to know about your API usage.
The derived view I find most useful: cost-per-feature-per-day. Not total spend, not average cost per request — cost per feature. This surfaces the cases where one feature is consuming a disproportionate fraction of the bill, which is often the signal that a model routing decision needs revisiting or a caching opportunity was missed.
What you don't need to log in structured form: full message payloads. If you need to debug a specific request, a request ID that maps to a logged trace is sufficient. Storing full conversation histories in your log aggregator at scale is expensive and creates GDPR exposure you probably don't want.
The tooling question is secondary. A cost_per_request table in Postgres with a timestamp and the fields above is sufficient to get started. You can add a dashboard layer (Grafana, Metabase, a simple SQL query) later. The important thing is that the data exists and is queryable before you need it — not after the first billing surprise.
Evaluation Has Its Own Bill
A point worth surfacing because it's invisible until it isn't: in any AI system that's actively evolving — model changes, prompt revisions, routing tuning — the evaluation infrastructure becomes its own cost line.
The shape of it: synthetic test generation calls the model. Regression suites that run on every prompt change call the model. LLM-as-judge evaluators call the model, often a more expensive one than the one being evaluated. Prompt experimentation runs hundreds of variations across hundreds of test cases. In aggregate, this can match or exceed the inference cost of production traffic, particularly during periods of active development.
This is not a reason to skip evaluation — running an AI system without it is more expensive in the long run, by larger margins. It's a reason to budget for evaluation as a real cost line and instrument it the same way you instrument production. Tag eval calls with a different feature label in your logs. Cap experimental runs against a separate budget. Don't conflate "the prompt change broke nothing in the regression suite" with "the prompt change was free."
When Optimization Is Not Worth the Time
This is worth saying explicitly because cost optimization is the kind of work that can consume engineering time indefinitely.
If your current spend is €50/month and the product is working well, optimizing from €50 to €25 is not a good use of a senior engineer's day. Do the math on your specific situation: if your product earns €0.10 in margin per transaction and you're paying €0.04 in AI costs per transaction, optimizing from €0.04 to €0.02 saves you €0.02/transaction. At 1,000 transactions per month, that's €20 in monthly savings. The engineering time to get there probably cost more than that.
The signal that optimization is worth prioritizing: the AI cost per transaction is high enough that it materially affects your unit economics, or the bill has grown faster than usage in a way that suggests waste rather than scale. If you're paying three times as much as six months ago and haven't tripled your traffic, that's a diagnosis worth doing.
The other signal: a single feature or use case is consuming a disproportionate fraction of total spend. That's almost always a routing decision that was never made explicitly, or a caching opportunity that was never implemented.
Cross-Provider Routing: When It's Worth the Complexity
Single-provider stacks are simple to operate; multi-provider stacks can be materially cheaper, but the savings have to clear an operational complexity tax that catches teams off guard.
The cost case is real. Different providers price input, output, and reasoning tokens differently. DeepSeek's pricing is dramatically below OpenAI's for general-purpose tasks where its accuracy is adequate. Anthropic prompt caching has different mechanics than OpenAI's. Gemini's long-context pricing changes the economics of large-document workloads. Provider arbitrage — routing each task to the cheapest provider that meets the quality bar for that specific task — is a genuine lever once your volume justifies it.
The operational case is also real. A multi-provider stack means: rate-limit handling across multiple APIs, separate observability per provider, separate billing reconciliation, different streaming semantics, provider-specific failure modes, and a routing layer that itself becomes a piece of infrastructure to maintain. Tool-calling semantics are not identical across providers. Output formats differ. Prompt techniques that work on one model can need rework on another. The complexity is not abstract.
The conditions under which it's worth doing: you have enough volume that the cost delta is material in absolute terms, you have engineering capacity to own the routing layer, and the workloads are heterogeneous enough that routing decisions have meaningful effect. Below those conditions, the simpler version usually wins — pick one provider, optimize within it, revisit when volume changes the math.
A middle path that I find genuinely useful: provider fallback rather than provider routing. Primary provider for normal traffic, secondary provider on rate limits or outages, both behind the same internal interface. This buys you resilience without taking on the full operational cost of cost-based routing, and it gives you observability on a second provider that you can later promote to active routing if the data justifies it.
Open-Source Models: Honest TCO
Self-hosting an open-source model is frequently suggested as the ultimate cost solution. The real economics are more complicated, and worth taking seriously in both directions — the dismissive "always more expensive than it looks" view is as wrong as the optimistic "always cheaper at scale" one.
You eliminate per-token API fees. You take on: GPU infrastructure costs, provisioning and scaling the serving layer, monitoring for model serving latency and availability, model versioning when you need to update, and someone's time to own all of this.
The economics shift heavily based on workload shape. The conditions under which self-hosting genuinely beats managed APIs at SMB/mid-market scale:
- Predictable, sustained throughput. A pipeline processing a steady stream of documents 24/7 amortizes fixed GPU costs in a way that bursty user-facing traffic does not. Idle GPUs are the silent killer of self-hosting economics.
- Batch tolerance over realtime. vLLM, SGLang, and TensorRT-LLM achieve dramatically higher throughput on batched inference than serial calls. If your workload can be batched — overnight document processing, large-volume embedding generation, analytics enrichment — the cost-per-token gap versus APIs can be an order of magnitude.
- Quantized inference is acceptable. INT8 or INT4 quantization roughly doubles or quadruples the tokens-per-GPU-hour at some accuracy cost. For classification, extraction, and structured tasks where the accuracy delta is below threshold, this changes the TCO meaningfully.
- Spot/preemptible GPU pricing is usable. For batch workloads where interruption is recoverable, spot pricing on cloud GPUs can be 50-80% below on-demand. Realtime serving generally can't tolerate this.
- Fine-tuning or LoRA adaptation is required. If your use case needs domain adaptation that managed APIs don't expose at acceptable cost, hosting your own model becomes the only option, not the cheap one.
The conditions under which it usually doesn't work: bursty user-facing traffic with strict latency SLAs, small absolute token volumes where fixed costs dominate, teams without anyone whose job includes GPU operations, and use cases that genuinely require frontier-model accuracy (the open-weight gap there is real even in 2026).
For most of the teams I work with — SMB SaaS products, support automation, mid-market integrations — the workloads are bursty and the volumes are low enough that managed APIs win the TCO comparison. For a different shape of company — predictable batch workloads at scale, in-house ML expertise — the conclusion can reverse. "Self-host to save money" is a workload question, not an aesthetic one.
If data residency is the driver rather than cost, that's a different analysis — Azure OpenAI, AWS Bedrock, and similar managed services may be a better fit than raw self-hosting.
The Pattern That Explains Most Bills
Across the various production deployments I've looked at when AI costs became a concern, most of the overspend comes from one of three things: everything routed to the largest model because nobody made routing decisions explicitly, prompt caching not configured because the prompts weren't structured to enable it, or no observability on per-feature costs so nobody knew which features were expensive.
None of these are exotic. None require specialized AI infrastructure expertise to fix. They're standard engineering hygiene — measurement, retries, budgets, caching — applied to a billing surface most teams haven't operated before. They're the natural result of building fast and instrumenting later, which is the normal order of operations for most products. All three are fixable in a focused sprint by the engineers who already own the system.
The deeper pattern, and the thesis the rest of this article was working toward: runaway AI bills are almost never caused by one bad architectural decision. They emerge from dozens of locally reasonable decisions — a slightly larger retrieval, a slightly more capable default model, a slightly longer system prompt, a slightly more permissive retry policy — that each look correct in isolation and compound silently across the system.
The mechanics aren't unusual. An AI bill grows the same way any infrastructure bill grows when nobody is watching the slope. Tokens instead of CPU-hours, model tiers instead of instance types, but the discipline is the same: instrument early, attribute by feature, set the limits before you need them, and revisit when the system shape changes. The first time you do this is the expensive lesson. After that, it stays in scope alongside every other production cost.
If you're building an AI-powered feature and want the cost surface treated as a first-class concern rather than discovered after the first billing surprise, that's part of what I cover in AI integration work. The patterns above are much easier to build in early than to untangle later.
For the broader integration picture — patterns, fallbacks, and production requirements — see How to Add AI to an Existing Product Without Rewriting It.
Top comments (0)