DEV Community: Hammad Malik

AI Subscriptions vs API Pricing: Where's the Crossover?

Hammad Malik — Sun, 01 Nov 2026 10:45:49 +0000

Originally published at API Cost Lab.

Last updated: October 2026

If you're building with AI, you eventually face a deceptively simple question: should I just pay for the subscription, or pay per token through the API? The answer isn't a preference — it's math. And the math has a crossover point most developers never calculate.

The $200/Month Rule of Thumb

Here's the heuristic, stated up front as what it is — a rule of thumb, not a law:

Under ~$200/month of usage: a flat subscription (ChatGPT Plus/Pro, Claude Pro/Max tier equivalents) usually wins.
Above ~$200/month: metered API pricing — with caching and routing — is typically cheaper and scales linearly.

Why does this threshold exist? Because subscriptions and APIs price fundamentally different things. A subscription sells you access at a flat fee: one price, generous-but-capped usage, no meter running. An API sells you tokens at unit prices: every token metered, no cap, cost scaling exactly with consumption.

At low volumes, the flat fee is subsidized — you're getting more value than you pay for, because the provider averages you against lighter users. At high volumes, you become the heavy user everyone else is averaged against, and the meter starts looking generous by comparison.

The exact crossover moves with prices and your usage shape, but ~$200/month is where the two curves typically cross for individual developers in 2026. Below it, don't overthink: subscribe. Above it, read on.

Why Subscriptions Win at Low Volume

Three reasons, all structural:

1. The flat fee subsidizes exploration. When you're prototyping, experimenting, or using AI as a personal productivity tool, your usage is bursty and unpredictable. A subscription absorbs that variance — a heavy week costs the same as a light one. On the API, every experiment has a meter running, which subtly discourages the tinkering that produces good results.

2. No engineering overhead. API usage needs keys, billing alerts, rate-limit handling, cost tracking. A subscription needs a login. For a solo developer or a small team just getting started, that simplicity is worth real money — your time spent on billing plumbing is time not spent building.

3. The caps are generous for humans. Subscription usage limits are designed around human-paced interaction. If your usage is "a developer chatting with a model through the day," you will rarely hit them. The meter only matters when usage stops being human-paced.

The honest version: if AI is a tool you use, subscribe. The subscription product is built for you, priced for you, and optimized for you.

Why APIs Win at Scale

The moment usage stops being human-paced — background jobs, user-facing features, agents, anything programmatic — the subscription model breaks down:

1. Subscriptions don't scale to machines. Usage caps assume a human clicking. An agent loop making hundreds of calls per hour blows through "generous" caps before lunch. And most subscription terms restrict or prohibit automated/API-style usage anyway — that's what the API product is for.

2. Per-token pricing rewards optimization. On the API, every efficiency gain drops your bill directly: prompt caching (~90% off repeated input on GPT-5.6 Sol: $0.40 vs $4.00 per 1M), batch processing (~50% off), routing simple tasks to cheaper models (GPT-6.1 Sol at $2/$10 vs $4/$20 — the reference workload drops from $205.92 to $97.68/month). Subscriptions give you none of these levers; the price is the price regardless of how clever your architecture is.

3. Costs scale linearly and predictably. API cost = tokens × rate. Double your users, double your tokens, double your bill — no cliffs, no "contact sales," no renegotiating tiers. For a business, that linearity is a feature: your unit economics are a formula, not a negotiation.

4. Teams multiply the gap. One subscription per human gets expensive fast at team scale — and still doesn't cover programmatic usage. API keys with per-key budgets and usage tracking are how teams actually manage AI spend.

The honest version: if AI is infrastructure you run, use the API. The metered product is built for machines, priced for machines, and gets cheaper the better you engineer.

How to Decide With Your Own Numbers

Forget the rule of thumb for a minute and do the actual comparison. It takes twenty minutes:

Step 1: Measure a month. Pull your real usage — or estimate it honestly. Count: how many tasks/requests per day, average input tokens (fresh vs. repeatable), average output tokens. If you're already on the API, the dashboard has this. If you're on a subscription, estimate from a heavy week and multiply.

Step 2: Price the API scenario. Use the workload formula: monthly cost = tasks × [(fresh_in × P_in) + (cached_in × P_cache) + (out × P_out)]. Use real rate-card prices — for example, GPT-5.6 Sol's promo at $4.00/$0.40/$20.00 per 1M, or GPT-6.1 Sol at $2.00/$0.10/$10.00. Apply the discounts you'd actually use: caching if your context repeats, batch if the work can wait.

Step 3: Compare against the subscription stack. Add up what you'd actually pay in subscriptions to cover the same usage: per-seat fees × team members, plus any overage or higher-tier costs when caps bind. Be honest about caps — "unlimited*" usually isn't.

Step 4: Check which side of the line you're on. If the API scenario comes in under the subscription stack, switch — and keep optimizing, because the API number only goes down as you tune. If subscriptions win, stay — and revisit when usage grows, because the crossover only moves in one direction as you scale.

Step 5: Consider the hybrid. This is what most serious teams end up with: subscriptions for humans (developers chatting, experimenting, doing support) and API for machines (features, agents, pipelines). They're different products for different consumers. Using both isn't indecision — it's the correct architecture.

The Trap: Subscription Thinking at API Scale

The most expensive mistake isn't picking the wrong option — it's keeping the right-for-last-quarter option too long. The pattern repeats constantly:

A solo developer subscribes, builds a side project, and wires the subscription's web UI into their workflow with copy-paste. It works. The project grows. They add a cron job that hits the API with their personal key "just for now." Then a second job. Then a teammate. Six months later they're paying for four subscriptions and an API bill, with no cost tracking, no caching, and no idea which dollar went where.

The subscription mindset — flat fee, don't think about usage — actively harms you at API scale, because the API rewards exactly the thinking the subscription trained you out of: measuring tokens, structuring prompts for cache hits, batching what can wait. Teams that migrate late discover their architecture was designed for a pricing model they no longer use: unbatched, uncached, flagship-everything.

The way out is a deliberate transition, not a flag day. Keep subscriptions for the humans. Move one machine workload to the API with proper tracking. Optimize it — caching, batch, model routing — until the per-task cost is dialed in. Then move the next. Each migration funds the engineering time for the one after, because the savings are immediate and measurable.

Watch for the signal that you're overdue: when your subscription tier's usage caps start binding, or when you're paying per-seat fees primarily to cover automated usage rather than humans. That's the market telling you the subscription product isn't yours anymore. Listen the first time — every month of delay is the most expensive month of AI spend you'll have.

The Bottom Line

Subscriptions vs. API isn't a loyalty question — it's a volume question with a crossover around ~$200/month. Below it, the flat fee subsidizes you; above it, the meter rewards your engineering. Measure your usage, run the formula, and let the numbers pick. And whatever you choose, recheck as you grow: the right answer at prototype scale is rarely the right answer at production scale.

Batch API vs Real-Time: When 50% Off Is Worth the Wait

Hammad Malik — Fri, 30 Oct 2026 10:35:42 +0000

Originally published at API Cost Lab.

Last updated: October 2026

Every major LLM provider sells the same model at two prices: full price for answers in seconds, roughly half price for answers in hours. The Batch API is the most underused discount in the industry — same model, same quality, ~50% off, in exchange for patience. Here's when it's worth it and when it isn't.

What Batch Mode Actually Is

A batch request is exactly what it sounds like: instead of sending one request and waiting for the response, you submit a batch of requests as a file, the provider processes them when it has spare capacity, and you collect the results later — typically within 24 hours.

Nothing about the model changes. It's the same weights, the same capabilities, the same output quality. The discount exists because you're letting the provider schedule your work during off-peak capacity instead of demanding instant compute. You're selling flexibility; they're buying it back at 50% off.

On GPT-5.6 Sol, the numbers are concrete: batch processing cuts the promo rates in half — $2.00 input / $10.00 output per 1M tokens instead of $4.00/$20.00. (Cached reads in batch presumably follow the same halving, though the headline discount is what providers publish.)

The mechanics are simple across providers: you upload a JSONL file of requests, poll for completion, download the results. Most providers return results well before the 24-hour SLA — but the SLA is what you're signing up for, so plan around the worst case.

The Worked Example: What 50% Off Looks Like

Take a concrete workload: a nightly eval run — 500 test cases, each with 8K input tokens and 2K output tokens, on GPT-5.6 Sol's promo pricing.

Real-time:

Input: 500 × 8,000 = 4M tokens × $4.00/1M = $16.00
Output: 500 × 2,000 = 1M tokens × $20.00/1M = $20.00
Total: $36.00 per nightly run → ~$1,080/month

Batch (50% off):

Input: 4M × $2.00/1M = $8.00
Output: 1M × $10.00/1M = $10.00
Total: $18.00 per run → ~$540/month

Same model, same evals, $540/month saved — for changing when the work runs, not what the work is. And this is a modest example. Teams running large eval suites, backfills, or bulk classification at 10x this volume save thousands monthly for a scheduling change.

Note this stacks with other optimizations. Batch requests with well-structured cached prefixes get both discounts. Batch on the cheaper model tier (say, GPT-6.1 Sol at $2/$10 → $1/$5 batched) compounds further. The teams with the lowest bills layer every applicable discount; batch is usually the easiest layer to add.

The Latency Tradeoff, Honestly

The catch is real: you don't control when results come back. The SLA is typically 24 hours. In practice, providers often return batch results in minutes to a few hours during off-peak times — but "often" isn't a guarantee, and you should architect for the SLA, not the typical.

This kills batch for anything user-facing or interactive. But it also rules out more than people think:

Fits batch: nightly eval suites, dataset labeling and backfills, bulk document processing, embedding generation, offline classification, regression test runs, periodic report generation, training-data preparation. Anything where the consumer of the output is a pipeline, a dashboard, or tomorrow's standup.

Doesn't fit: chat, interactive agents, anything with a human waiting, real-time moderation, request-response APIs, multi-step agent loops (each step would take hours — a 10-step agent becomes a 10-day agent).

The gray area is bigger than it looks. Many "real-time" workloads aren't: that Slack bot summarizing threads? Nobody needs it in seconds. The weekly analytics digest? The support-ticket triage that runs overnight anyway? Audit your async jobs — teams routinely find 30–50% of their API spend is on work that could wait hours with zero user impact.

The Migration Is Smaller Than You Think

The reason teams don't use batch isn't the tradeoff — it's inertia. The migration is genuinely small:

Collect requests instead of sending them. Where your code currently calls the API in a loop, write the requests to a JSONL file instead.
Submit and poll. One submit call, one status poll on a schedule (or a webhook where supported).
Process results the same way. The response format matches the real-time API — your result-handling code barely changes.

Most teams do this in a day for a single pipeline. The pattern generalizes: once one pipeline runs on batch, adding the next is mostly copy-paste.

Two operational notes. First, error handling differs: a malformed request in a batch of 10,000 doesn't fail fast — you find out hours later. Validate requests before submitting. Second, don't batch what you haven't measured: run the workload real-time first, confirm quality, then move it to batch. Debugging quality issues through a 24-hour feedback loop is miserable.

Stacking Discounts: Batch Plus Everything Else

The batch discount composes with every other optimization, and the combination is where the real money is:

Batch + caching: batch requests with well-structured cached prefixes get both the ~50% batch discount and the ~90% cached-input discount. On GPT-5.6 Sol's promo, a cached input token in a batch request costs a fraction of a cent per million — the input side of the bill effectively disappears.
Batch + smaller models: batching GPT-6.1 Sol ($2/$10 → $1/$5 batched) for bulk work gets you frontier-adjacent quality at budget-tier prices, halved again. For classification, extraction, and eval workloads, this combination is routinely 70–80% cheaper than the naive real-time flagship setup.
Batch + off-peak scheduling: submit batches in the evening, collect results by morning standup. The latency is invisible because nobody was waiting.

Work the full stack on the earlier eval example: 500 cases/night on GPT-6.1 Sol batched with decent caching, and the $36/run real-time flagship cost collapses to single digits. That's not a marginal optimization — it's an order-of-magnitude change in what the workload costs to run.

Finding Your Batch Candidates

If you're not sure what qualifies, run this audit over your API usage for a week:

List every automated caller — cron jobs, pipelines, background workers, scripts. Not user-facing features; machines.
For each, ask: who reads the output, and when? If the answer is "a dashboard tomorrow," "a data warehouse," or "an engineer next week," it's a batch candidate. If the answer is "a user staring at a spinner," it isn't.
Estimate the spend per candidate. Even rough numbers work: requests/day × average tokens × rate. Rank by spend, migrate the top two first.

Most teams find their top batch candidates in places they weren't thinking about: the nightly data-quality checks, the weekly report generation, the eval suite someone runs manually every Friday, the backfill script that runs "just this once" every month. The pattern is always the same — work scheduled by habit on the real-time API that never needed to be real-time.

The Bottom Line

The Batch API is a 50% discount for the cost of patience, and most teams are leaving it on the table because their async work is accidentally running on the real-time API. Audit your jobs, find everything that doesn't need an answer in seconds, and move it. It's the easiest large discount in LLM pricing — same model, half the price, one day of engineering.

How to Calculate Your AI API Bill (Developer's Guide)

Hammad Malik — Wed, 28 Oct 2026 10:44:39 +0000

Originally published at API Cost Lab.

How to Calculate Your AI API Bill (Developer's Guide)

Last updated: October 2026

Every AI API bill comes down to one thing: tokens in, tokens out, multiplied by the price list. But the gap between "I understand the pricing page" and "I can predict my monthly invoice" is where most teams get surprised.

This guide closes that gap. You'll learn what a token actually is, how the three prices on every rate card work, a formula for modeling any workload, and five practical ways to cut the bill — with real numbers throughout.

What Is a Token?

A token is the unit of text the model reads and writes. As a rough rule of thumb:

1 token ≈ 4 characters of English text
1 token ≈ 0.75 words

So 100 tokens is roughly 75 words. A typical page of text (~500 words) is about 650–700 tokens.

Why it matters: everything is metered in tokens, but you think in tasks, messages, and documents. The single most useful habit in API cost management is building an intuition for converting between the two. A few anchors:

A short chat message: 20–50 tokens
A system prompt for a coding agent: 2,000–8,000 tokens
A medium source file loaded into context: 3,000–10,000 tokens
A long document or small codebase: 50,000–200,000+ tokens

Tokenizers differ slightly between providers, but the 4-characters-per-token approximation holds well enough for budgeting. If you need precision, every major provider publishes a tokenizer tool — but for cost modeling, the approximation is fine.

The Three Prices on Every Rate Card

Nearly every LLM API prices three things separately. Understanding why they're priced differently is the key to the whole game.

Input tokens

What you send to the model: prompts, context, conversation history, documents. This is usually the cheapest of the three per-token rates — but in agentic workloads it's often the largest volume, which makes it the largest line item.

Cached input tokens

Input tokens the provider has seen before and kept in cache. When you send the same system prompt or the same repo context for the hundredth time, the provider doesn't recompute everything from scratch — and passes most of the savings to you.

Cached reads are typically priced at 5–10% of the input rate. On GPT-5.6 Sol, for example, cached input is $0.40/1M versus $4.00/1M fresh — a 90% discount. That single pricing detail reshapes how you should architect: anything stable and repeated (system prompts, few-shot examples, repo context) should be structured to hit the cache.

One caveat: writing to the cache usually costs extra (1.25x the input rate on GPT-5.6 Sol). The first time a prompt is cached you pay a small premium; every reuse after that is nearly free. For repeated workloads the math is overwhelmingly in your favor.

Output tokens

What the model generates. Almost always the most expensive per token — typically 5x the input rate. On GPT-5.6 Sol: $20.00/1M output versus $4.00/1M input.

This asymmetry drives real design decisions. Verbose chain-of-thought reasoning, long explanations, and rambling agent loops are the most expensive thing you can do with an API. Concise outputs, structured formats, and stopping the model when it has the answer aren't just good UX — they're cost control.

The Workload Formula

With those three prices, any workload reduces to one formula:

Monthly cost = tasks/month × [(fresh_in × P_in) + (cached_in × P_cache) + (out × P_out)]

Where P_in, P_cache, and P_out are the per-token prices from the rate card.

Let's run it with real numbers. Take GPT-5.6 Sol's promo pricing: $4.00/1M input, $0.40/1M cached, $20.00/1M output.

Assumptions:

A coding agent running 50 tasks/day × 22 days = 1,100 tasks/month
Per task: 12K fresh input tokens, 48K cached input tokens, 6K output tokens

Per task:

Fresh input: 12,000 × $4.00/1M = $0.0480
Cached input: 48,000 × $0.40/1M = $0.0192
Output: 6,000 × $20.00/1M = $0.1200
Total per task: $0.1872

Per month: 1,100 × $0.1872 = $205.92

Two things jump out. First, output tokens — only 6K of the 66K total — are 64% of the cost. That's the 5x price asymmetry at work. Second, the 48K cached tokens cost less than half the 12K fresh tokens. Caching is doing enormous work in this bill.

Now change one variable: if that promo pricing reverts to $5/$30 (the pre-promo list price), the same workload costs $290.40/month — a 41% jump with zero change in usage. This is why you model scenarios, not just today's rate card. (For the full breakdown of that deadline, see our GPT-5.6 pricing guide.)

5 Practical Ways to Cut the Bill

These are ordered by typical impact, not difficulty.

1. Maximize prompt caching

This is the highest-leverage move available. If your workload repeats context — and agentic loops almost always do — structure prompts so the stable prefix (system instructions, repo context, few-shot examples) is identical across calls and the variable part comes last. Cache hit rates above 80% are achievable, and at a 90% discount per cached token, that transforms the bill.

In our example workload, cached tokens are already 80% of input volume. Push that higher and the per-task cost drops fast.

2. Use batch APIs for anything that can wait

Most providers offer a batch mode at roughly 50% off — same model, slower turnaround (usually hours). Evals, bulk classification, backfills, nightly jobs: if it doesn't need an answer in seconds, it shouldn't pay the real-time rate. This is the easiest discount in the industry and the least used.

3. Route simple tasks to smaller models

Not every task needs the flagship. Classification, extraction, formatting, simple Q&A — a smaller model at a fraction of the price handles these fine. The pattern that works: a cheap model attempts the task, and only escalates to the expensive one when confidence is low or the task is flagged as hard. On GPT-6.1 Sol at $2/$10, for instance, our example workload drops to $97.68/month — less than half.

The key discipline is measuring: run the small model on a sample of your real tasks and check quality before committing. Don't assume, and don't guess — benchmark.

4. Watch context-window bloat

Input volume is the silent killer. Every file you stuff into context, every full conversation history you resend, every verbose tool result — all billed, every call. Audit what you're actually sending: teams routinely find 30–50% of their input tokens are context nobody reads.

Practical fixes: summarize conversation history instead of resending it raw, retrieve only the relevant files instead of the whole repo, truncate tool outputs to what the model needs. And remember the threshold trap — on some models, crossing a context size (272K input tokens on GPT-5.6 Sol) reprices the entire request at a higher rate. Know your provider's thresholds.

5. Use off-peak / flex tiers where offered

Some providers offer discounted tiers for flexible timing — your request runs when capacity is available, you pay less. If your workload is asynchronous by nature (and much of it is), flex pricing is nearly free money. Check whether your provider offers it and what the discount is; the tradeoff is latency variability, which batch-tolerant work doesn't care about.

A Checklist Before You Commit to a Model

Pricing pages change — November and December 2026 are proof, with two major models repricing within weeks of each other. Before you build on any model:

Measure your token mix — fresh vs. cached vs. output, from your actual usage, not guesses.
Model at least two price scenarios — today's rate and the plausible next rate.
Benchmark cheaper alternatives on your real tasks before you need them.
Check the thresholds — context-size pricing cliffs, cache write costs, minimum charges.
Build for portability — abstract the model call so a price change is a config edit, not a rewrite.

The developers with the lowest AI bills aren't the ones who found the cheapest model. They're the ones who measured their usage, cached aggressively, batched what could wait, and never let a promo price become a load-bearing assumption.

Argon vs Wonder: Which AI Should Developers Build On?

Hammad Malik — Mon, 26 Oct 2026 10:31:00 +0000

Originally published at API Cost Lab.

Argon vs Wonder: Which AI Should Developers Build On?

Last updated: October 2026

Two flagship models, two very different price tags — and one of the prices is about to double. If you're choosing a foundation model to build on right now, the decision isn't just about capability. It's about timing.

Here's the honest comparison: what each model costs, what the benchmarks actually say, and which one to pick depending on what you're optimizing for.

The Pricing: Read the Dates

Gemini 4 Argon is on introductory pricing: $2.00 per 1M input tokens / $10.00 per 1M output tokens. That intro rate runs through November 30, 2026. From December 1, the price doubles to $4.00 / $20.00.

Muse Wonder is priced at $4.00 / $20.00 per 1M tokens — no intro period, no scheduled change.

So today, Argon costs half what Wonder does. On December 1, they cost exactly the same.

That convergence matters more than it looks. Every cost comparison you run today has a 6-week expiry date. If you're doing the math — and you should be — do it twice: once at today's prices, once at December prices.

The December 1 Math

Take a straightforward workload: 1M input tokens and 500K output tokens in a month. (Small, but the ratios are what matter.)

Today:

Argon: (1M × $2) + (0.5M × $10) = $7.00
Wonder: (1M × $4) + (0.5M × $20) = $14.00
Argon is 50% cheaper.

From December 1:

Argon: (1M × $4) + (0.5M × $20) = $14.00
Wonder: (1M × $4) + (0.5M × $20) = $14.00
Identical.

The entire pricing advantage evaporates on December 1. Which means if you're choosing Argon for the price, you're choosing a 6-week discount, not a structural advantage. That's fine — six weeks of half-price inference is real money at scale — but don't architect around a price that won't exist in December.

There's a second-order effect worth noting: a lot of teams will do exactly this math in late November and migrate workloads at once. If you're planning a switch, don't leave it to the last week.

The Benchmarks: What the Scores Say

Pricing is only half the decision. Here's what the published benchmarks report — with the caveat that different indexes measure different things, and no single number captures "better."

Artificial Analysis Intelligence Index (a composite capability score):

Claude Opus 5.5: 58
Muse Wonder (Full): 56
Gemini 4 Argon: 53

On this index, Wonder leads Argon by 3 points, with Anthropic's Opus 5.5 ahead of both.

Vals AI Index:

Gemini 4 Argon: 68.9% — ranked #1

Argon tops a different leaderboard entirely.

What do you do with two indexes that disagree? You treat them as what they are: different test suites weighting different capabilities. The AA Index leans on a broad composite of reasoning, coding, and knowledge tasks. Vals AI's index has its own methodology and task mix. A model can genuinely be #1 on one and third on another without any contradiction — they're measuring different things.

The practical takeaway: the scores put Argon and Wonder in the same tier. Neither is dominating the other across the board. A 3-point gap on a composite index is not the kind of difference that decides a build — it's the kind of difference that gets washed out by prompt engineering, tooling, and how well the model fits your specific task.

What would decide it: benchmark them on your workload. Both models have APIs you can hit today. A weekend of evals on your actual tasks beats any public leaderboard.

Who Should Pick What

With pricing converging in December and benchmarks in the same tier, the decision comes down to your situation:

Pick Argon if you're cost-sensitive right now

At $2/$10, Argon is the cheapest frontier-class option available. If you're running high-volume inference — bulk processing, evals, user-facing features with thin margins — six weeks at half price is significant. Just go in with eyes open: re-run your cost model at $4/$20 and make sure the unit economics still work in December.

This is also the right pick if your workload is bursty and front-loaded — a migration, a backfill, a big eval run you can complete before December 1.

Pick Wonder if you value stability

Wonder's price isn't moving. What you model today is what you pay in December. If you're signing up users, promising SLAs, or just don't want to re-do your cost math in six weeks, that predictability has real value.

Reassess everything in December

This is the most important recommendation. Whatever you choose now, put a calendar reminder for late November. On December 1 the pricing landscape resets: Argon at $4/$20, Wonder at $4/$20, and the decision becomes purely about capability fit, ecosystem, and your eval results — which is how it should be.

When that reassessment comes, don't just compare the two headline rates. Re-run your full workload model: your actual token mix will have drifted since October, your cache hit rates may have improved, and both providers may have shipped new variants or pricing tiers by then. Six weeks is a long time in this market. Treat December 1 as a scheduled architecture review, not just a billing change — it's a natural checkpoint to ask whether your model choice still earns its place.

[EMBED: https://youtu.be/GI51xborhEg]

Don't Ignore Switching Costs

One more factor that rarely makes it into pricing posts: moving between models isn't free. Different providers mean different API shapes, different tool-calling conventions, different quirks in how each model handles structured output. If you've tuned prompts and eval harnesses for one provider, migrating takes real engineering days.

That cuts both ways. If you're on Wonder today and tempted by six weeks of cheap Argon, weigh the migration cost against the savings — at small scale, the engineering time costs more than the inference discount. Conversely, if you're starting greenfield right now, there's an argument for building provider-agnostic from day one: abstract the model call behind an interface, keep prompts portable, and you'll be able to chase whichever price is best in any given quarter without a rewrite.

The teams that treat the model as a swappable component — not a foundation to marry — are the ones that actually capture these pricing windows instead of just reading about them.

What About Opus 5.5?

You'll notice the AA Index has Claude Opus 5.5 at 58 — ahead of both. At $4/$20 with cache reads at $0.20/1M, its pricing is in the same band as Wonder's, with notably cheaper caching. If you're already in the Anthropic ecosystem, or your workload is cache-heavy, it belongs in your evaluation set. The two-horse framing of this post is about the models with moving prices; Opus 5.5 is the stable third option worth testing alongside them.

The Bottom Line

Today: Argon is half the price and roughly the same class. That makes it the rational default for cost-sensitive work — with a hard expiry date of December 1.

After December 1: price is no longer a differentiator between these two, and the choice comes down to your evals.

Either way, the developers who win are the ones who measured instead of guessed. Both APIs are live. The benchmarks are public. You've got six weeks. Use them.

Why Is My OpenAI API Bill So High?

Hammad Malik — Sat, 24 Oct 2026 17:41:25 +0000

Originally published at API Cost Lab.

Last updated: October 2026

Almost every developer hits the same moment: the first real OpenAI invoice. You expected tens of dollars. You got hundreds. And your first instinct — that something must be broken — is usually wrong. Nothing is broken. Your bill is high for predictable, fixable reasons, and they're the same reasons for nearly everyone.

This is a diagnostic checklist. Work through it in order, and you'll find where your money is going.

The Usual Culprits

1. Every call goes to the flagship model

This is the number one cause of bill shock, by a wide margin. Somewhere in your code there's a model name, and it's the biggest one — because that's the default, that's what the quickstart used, and nobody revisited it.

The problem: not every task needs the flagship. Classification, extraction, formatting, simple Q&A, summarization — these run fine on smaller, cheaper models. On OpenAI's current lineup, GPT-6.1 Sol is priced at $2.00 input / $10.00 output per 1M tokens versus GPT-5.6 Sol's promo $4.00 / $20.00. Same provider, same API shape, half the price. On our reference workload (1,100 tasks/month, 12K fresh + 48K cached input + 6K output per task), that's $97.68/month versus $205.92 — less than half, for tasks the smaller model handles fine.

The fix isn't "downgrade everything." It's routing: flagship for the hard stuff, smaller model for the routine stuff. Most teams find 60–80% of their calls don't need the flagship once they actually measure.

2. Uncapped output length

Output tokens are the most expensive tokens you buy — typically ~5x the input rate. On GPT-5.6 Sol's promo pricing, that's $20.00 per 1M output versus $4.00 per 1M input. Every token the model generates costs five times what it cost to feed it the prompt.

Now check: did you set max_tokens? If you didn't, the model decides how long its answer is. And models are verbose by default. Chain-of-thought reasoning, restated context, polite filler, long explanations nobody asked for — all billed at the 5x rate.

This is the easiest money to recover. Set explicit output caps per task type. A classification task doesn't need 2,000 output tokens. Neither does most extraction. Cap aggressively, then raise the cap only where quality actually demands it.

3. No prompt caching — or caching that never hits

If your workload repeats context — and agentic loops almost always do — prompt caching is the single biggest lever on your bill. On GPT-5.6 Sol, cached input reads cost $0.40 per 1M tokens versus $4.00 fresh. That's a 90% discount on every repeated token.

But there are two failure modes. The first is not having caching set up at all. The second, sneakier one: caching is "enabled" but your hit rate is near zero. Cache hits require the stable prefix of your prompt to be byte-identical across calls. If you're prepending timestamps, request IDs, or reshuffling prompt sections on every call, nothing matches the cache and you're paying full price while thinking you're optimized.

Check your actual cached-token ratio in the usage dashboard. If it's under 50% on a repetitive workload, your caching isn't working — the setup is the problem, not the concept.

4. Crossing the 272K threshold

This one is specific to GPT-5.6 Sol, and it bites hard. If a single request contains more than 272K input tokens, the entire request is repriced: 2x on input, 1.5x on output. Not just the tokens above the threshold — the whole call.

On promo rates, that turns $4.00/$20.00 into $8.00/$30.00 for the full request. A coding agent with a loaded repo plus a long conversation history can cross 272K without anyone noticing, and suddenly every one of those calls costs double on input.

If your bill has unexplained spikes, check your per-request token counts. A handful of threshold-crossing requests can dominate a monthly invoice.

5. Verbose outputs billed at 5x

Worth stating separately from the max_tokens point, because it's about design, not configuration. Every "explain your reasoning step by step" instruction, every request for the model to "be thorough," every agent loop that lets the model narrate its actions — that's output tokens at 5x input rates.

There's a second-order cost too: verbose outputs from one call often become input to the next call in an agent loop. You're paying 5x to generate tokens, then paying again to re-read them. Concise outputs compound their savings across the whole loop.

6. Reasoning tokens you never see

On reasoning models, there's a line item most developers don't know exists: the model's internal "thinking" tokens are billed as output tokens even though you never see them in the response. A request that returns 500 visible tokens may have burned 3,000 reasoning tokens behind the scenes — all at the 5x output rate.

This is by design and documented by providers, not a bug. But it means the true output cost of a reasoning-heavy task can be several multiples of what the visible response suggests. If your bill is high on an agentic workload and the visible outputs look modest, invisible reasoning tokens are a prime suspect. The mitigation is the same discipline as visible verbosity: match the reasoning effort to the task. Not every call needs maximum reasoning depth, and many providers let you set it per request.

How to Audit Your Bill

Before changing anything, measure. The OpenAI usage dashboard gives you what you need:

Split spend by model. This takes thirty seconds and answers the biggest question: what share of your bill is the flagship? If it's over 80%, culprit #1 is your main problem.

Split by endpoint or API key. If you use separate keys per service or environment, you can see which part of your system is expensive. Often it's one endpoint — a chat feature, a background job — doing most of the damage.

Measure your token mix. For your top endpoints, pull a sample of requests and compute: average fresh input tokens, cached input tokens, output tokens per call. Compare against the assumptions in your cost model. The gap between "what we assumed" and "what's actually happening" is where the money is hiding.

Check for threshold crossings. Look at the distribution of input tokens per request, not just the average. Averages hide the ten 300K-token requests that each cost double.

A concrete example of what the audit finds: a team running a coding agent assumed ~$150/month based on the $4/$20 promo rate card. The dashboard split showed 92% of spend on the flagship model, an average output of 4,200 tokens per task against no configured cap, and a 12% cache hit rate on a workload that re-sent the same 40K-token repo context every call. Three findings, three fixes — the bill dropped by more than half without changing what the agent could do. Your numbers will differ, but the shape of the discovery is almost always the same: the model mix, the output length, and the cache ratio.

Do this audit before optimizing. Otherwise you're guessing, and guessing at API costs is how you got the surprising invoice in the first place.

The 3 Fastest Fixes

If you want the biggest savings this week, in order:

1. Cap your outputs. Set max_tokens per task type today. This is a config change, not a project. Same-day savings on every call.

2. Route simple tasks to GPT-6.1 Sol. Pick your highest-volume, lowest-complexity endpoint, run a quality sample against the smaller model, and switch it if it clears your bar. On the reference workload this cuts the monthly cost from $205.92 to $97.68 — and most teams have at least one endpoint where the smaller model is fine.

3. Fix your cache hit rate. Structure prompts with the stable content first and the variable content last, keep the prefix byte-identical across calls, and verify the hit rate in the dashboard. Going from 0% to 80% cached on a repetitive workload cuts the input portion of your bill by roughly 70%.

The Bottom Line

High OpenAI bills are rarely a mystery and never a bug. They're the predictable result of flagship-by-default, uncapped outputs, and unmeasured usage. The audit takes an hour. The three fixes take a day. And the developers who do this work stop being surprised by invoices — because they finally know what each request actually costs.

Cheapest LLM API in 2026: What to Actually Pick

Hammad Malik — Thu, 22 Oct 2026 10:39:22 +0000

Originally published at API Cost Lab.

Last updated: October 2026

"What's the cheapest LLM API?" is the wrong question — or rather, it's a question with a trap in it. The model with the lowest per-token price is rarely the cheapest one on your invoice. Your real bill is decided by three things the headline price doesn't show: cache rates, output ratios, and pricing thresholds.

Here's the honest comparison, using only verified current prices.

The Verified Price Table

These are the models with confirmed pricing as of October 2026. No rumors, no "reportedly" — just published rate cards:

Model	Input / 1M	Cached input / 1M	Output / 1M
GPT-5.6 Sol (promo)	$4.00	$0.40	$20.00
GPT-6.1 Sol	$2.00	$0.10	$10.00
Claude Opus 5.5	$4.00	$0.20	$20.00
Claude Sonnet 5	$2.00	—	$10.00
Gemini 4 Argon (intro)	$2.00	—	$10.00
Gemini 4 Argon (from Dec 1)	$4.00	—	$20.00
Muse Wonder	$4.00	—	$20.00

Two deadlines matter here. GPT-5.6 Sol's promo is "available at least through November 21, 2026" — after that, pricing is unpublished, but the pre-promo list price was $5/$30. Gemini 4 Argon's intro rate runs through November 30, 2026, then doubles to $4/$20 on December 1.

If you stopped at the table, you'd conclude GPT-6.1 Sol and Sonnet 5 and Argon (today) tie at $2/$10. But the table isn't the bill. Here's why.

Why Headline Price Doesn't Decide Your Bill

Cache rates change everything

Look at the cached-input column. GPT-5.6 Sol charges $0.40 for cached reads — 10% of its input rate. GPT-6.1 Sol charges $0.10 — just 5%. Claude Opus 5.5 charges $0.20.

On a cache-heavy workload, these differences dominate. Take the reference workload: 1,100 tasks/month, each with 12K fresh input, 48K cached input, and 6K output tokens:

GPT-5.6 Sol (promo): $205.92/month
GPT-6.1 Sol: $97.68/month
Claude Opus 5.5: $195.36/month — 5.1% cheaper than the Sol promo, despite identical $4/$20 headline rates, purely because its cache reads cost half as much

Two models with the same sticker price can differ 5% or more on real workloads. And a model with a higher sticker price but better caching can beat a cheaper-looking one. Always model with your actual cache ratio, not the headline rate.

Output ratios punish verbosity

Output tokens cost roughly 5x input tokens across the board. That means the model that generates concise answers is cheaper than the model that rambles — at identical per-token prices.

This is where per-task cost beats per-token cost as a metric. A $2/$10 model that writes 12K output tokens per task costs more than a $4/$20 model that writes 4K. You can't know this from the price table; you learn it from measuring your actual usage per model. When evaluating, benchmark cost per completed task, not cost per million tokens.

Thresholds and deadlines are hidden price hikes

GPT-5.6 Sol reprices the entire request at 2x input / 1.5x output past 272K input tokens. A workload that regularly crosses that line isn't paying $4/$20 — it's paying $8/$30. Meanwhile, Argon's price doubles on December 1 and Sol's promo has an expiry date with no published follow-on price. The "cheapest" model today can be the most expensive one next quarter if you're not watching the dates.

Quality-Per-Dollar Verdicts by Use Case

Price without capability context is meaningless. Here's how the verified options stack up by workload — with the honest caveat that you should benchmark on your own tasks:

Bulk processing and evals (cost-first): GPT-6.1 Sol at $2/$10 with $0.10 cached reads is the value pick among verified options. For high-volume, low-complexity work — classification, extraction, bulk summarization — it's less than half the cost of the flagship tier. Pair it with batch APIs (50% off) and the economics get hard to beat.

Coding agents (balanced): This is where cache economics matter most, because agentic loops re-send context constantly. Claude Opus 5.5 at $4/$20 with $0.20 cache reads is 5% cheaper than GPT-5.6 Sol's promo on the reference workload — and on the AA Intelligence Index it scores 58 versus Wonder Full's 56 and Argon 53. GPT-5.6 Sol's promo is competitive until November 21; after that, re-model. The right answer here is genuinely workload-dependent: measure your cache ratio, then pick.

Chat and interactive products (latency-sensitive): Batch discounts don't apply, so you're paying full rate-card prices. Here the decision is mostly about the $2/$10 tier versus the $4/$20 tier — Sonnet 5 (whose $2/$10 intro rate was made permanent), GPT-6.1 Sol, or Argon before December 1. All three are half the flagship price; pick on capability fit and ecosystem, not on fractions of a cent.

One more consideration: don't ignore switching costs. Different providers mean different API shapes, tool-calling conventions, and prompt quirks. Chasing the cheapest rate every quarter only pays if your stack is provider-agnostic. Otherwise the migration engineering costs more than the inference savings.

A practical way to stay portable without over-engineering: wrap model calls behind a thin internal interface from day one — one function that takes a prompt and returns a completion, with the provider details behind it. You don't need a full abstraction framework; you need the freedom to change a config value instead of rewriting call sites. Teams that do this capture pricing windows as they open. Teams that hard-code the provider into every module read about the discounts instead.

The Re-Check Calendar: Prices Move

One more thing the table doesn't show: every price on it has a shelf life. In 2026, LLM pricing moves constantly — providers reprice quarterly, run promos with expiry dates, and reverse direction without much warning. Two dates are already on the calendar: November 21, when GPT-5.6 Sol's promo period reaches its published "at least through" date with no follow-on pricing announced, and December 1, when Gemini 4 Argon's intro rate doubles.

Build a habit around this: re-verify any price table older than 30 days before budgeting against it, and put the known deadlines in your calendar now. Better yet, don't hard-code rates into your cost models — parameterize them, so a price change is a config edit rather than a rewrite. The developers who get burned are the ones who picked the cheapest model in October and never looked again.

There's a second-order effect to the deadlines too. When a popular model's price jumps, a lot of teams migrate workloads at once — which means the weeks around November 21 and December 1 are exactly when you don't want to be scrambling. Do your fallback benchmarking now, while the current prices still hold, so the deadline is a non-event instead of a fire drill.

The Bottom Line

The cheapest LLM API in 2026 isn't a model — it's a method. Model your actual token mix (fresh vs. cached vs. output), watch the pricing deadlines (November 21 for Sol's promo, December 1 for Argon's doubling), benchmark cost per task instead of cost per token, and re-check quarterly. Prices in this market move constantly; the developers with the lowest bills are the ones who measure instead of assuming.

GPT-5.6 API Pricing: The November 21 Deadline

Hammad Malik — Tue, 20 Oct 2026 12:04:42 +0000

Originally published at API Cost Lab.

GPT-5.6 API Pricing: The November 21 Deadline

Last updated: October 2026

If you're running GPT-5.6 through the API right now, you're on a promotional price. And that promotion has an expiry date: November 21, 2026.

OpenAI's pricing page states the current rate is "available at least through November 21, 2026." Read that carefully. "At least through" — not "until." The promo could extend. It could also end exactly on that date. OpenAI has published no pricing for what comes after.

This post lays out exactly what the pricing is today, what it costs on a realistic workload, and how to budget for the unknown that follows November 21.

The Current Pricing, Exactly

GPT-5.6 is available in several variants. The one most developers are using — the flagship reasoning model, referred to here as GPT-5.6 Sol — is currently priced at:

Input: $4.00 per 1M tokens
Cached input: $0.40 per 1M tokens
Output: $20.00 per 1M tokens

That is promotional pricing. Before the promo, announced August 21, 2026, the list price was $5.00 input / $30.00 output per 1M tokens.

So the promo is a 20% cut on input and a 33% cut on output. Meaningful — but temporary, or at least not guaranteed past November 21.

The Fine Print That Actually Matters

Three pricing details that bite people:

1. The 272K token threshold. If a single request contains more than 272K input tokens, the entire request is billed at a higher rate: 2x on input, 1.5x on output. Not just the tokens above the threshold — the full request. On the promo rate, that means $8.00/1M input and $30.00/1M output for the whole call.

This matters most for coding agents with large context windows. A repo loaded into context plus a long conversation history can cross 272K faster than you'd expect. If your agent regularly runs hot on context, model your costs at the 2x/1.5x rate, not the headline rate.

2. Cache writes cost 1.25x. Prompt caching is the single biggest cost lever on this model (more on that below), but writing to the cache isn't free: cached tokens you write are billed at 1.25x the input rate. Reads are the cheap part at $0.40/1M. The math still overwhelmingly favors caching — a token read from cache costs a tenth of a fresh token — but don't model cache writes as free.

3. Batch API and Flex processing: 50% off. If your workload isn't latency-sensitive — evals, bulk processing, overnight jobs — the Batch API halves your bill. Same model, same quality, just slower turnaround. Flex processing offers a similar discount tier for eligible workloads. If you're not using batch for anything that can wait, you're leaving money on the table.

The Workload Math

Headline rates don't tell you what you'll actually pay. Let's run a realistic coding-agent workload.

Assumptions (stated up front so you can adjust):

50 tasks per day, 22 working days per month = 1,100 tasks/month
Per task: 12K fresh input tokens, 48K cached input tokens, 6K output tokens

That token mix is typical for an agentic coding loop: a stable system prompt and repo context served from cache, plus fresh tokens for the current task and conversation, with moderate output per step.

At promo pricing ($4.00 / $0.40 / $20.00):

Fresh input: 12,000 × $4.00/1M = $0.0480
Cached input: 48,000 × $0.40/1M = $0.0192
Output: 6,000 × $20.00/1M = $0.1200
Per task: $0.1872 → per month: $205.92

If pricing reverts to the pre-promo $5/$30 (the only concrete number we have for "after"):

Fresh input: 12,000 × $5.00/1M = $0.0600
Cached input: 48,000 × $0.50/1M = $0.0240 (cache scales proportionally)
Output: 6,000 × $30.00/1M = $0.1800
Per task: $0.2640 → per month: $290.40

That's a 41% increase overnight, on identical usage.

To be clear: nobody has said the price reverts to $5/$30. It might stay at $4/$20. It might land somewhere in between. There is no official post-November-21 pricing — I've checked, and as of this writing OpenAI has published nothing. The $290.40 figure is a scenario, not a prediction. But it's the only scenario with a real number attached, which makes it the one to budget against.

[EMBED: https://youtu.be/6xS9yEQpiLE]

The Fallbacks Worth Knowing

If the promo ends and the price jumps, you have options. Two worth modeling on the same workload:

GPT-6.1 Sol: $2/$10

The newer, smaller sibling is priced at $2.00 input / $10.00 output per 1M tokens, with cached reads at just $0.10 — a twentieth of the input rate. On our workload:

Fresh input: 12,000 × $2.00/1M = $0.0240
Cached input: 48,000 × $0.10/1M = $0.0048
Output: 6,000 × $10.00/1M = $0.0600
Per task: $0.0888 → per month: $97.68

Less than half the promo cost of GPT-5.6 Sol. The tradeoff is capability — it's a smaller model. For routine coding tasks it may be plenty; for the hardest reasoning work, maybe not. The honest answer is to benchmark it on your tasks before November 21, while you still have the promo as a safety net.

Claude Opus 5.5: $4/$20

Anthropic's flagship is priced at $4.00 input / $20.00 output — identical headline rates to the GPT-5.6 Sol promo — but with cheaper cache reads at $0.20 (0.05x). On our workload:

Fresh input: 12,000 × $4.00/1M = $0.0480
Cached input: 48,000 × $0.20/1M = $0.0096
Output: 6,000 × $20.00/1M = $0.1200
Per task: $0.1776 → per month: $195.36

That's 5.1% cheaper than the Sol promo on a cache-heavy workload, and the gap widens the more you lean on caching. If your workload is cache-dominated — and most agentic loops are — Opus 5.5's cache economics deserve a serious look regardless of what happens on November 21.

What To Do Before November 21

Three concrete moves:

1. Measure your actual token mix. The math above uses assumed numbers. Pull your real usage from the OpenAI dashboard — fresh vs. cached input ratio, average output per request, how often you cross 272K. Your numbers will differ; the method is what matters.

2. Benchmark the fallbacks now. Run GPT-6.1 Sol and Claude Opus 5.5 against a sample of your real tasks while the promo is still active. You want to know before the deadline whether the cheaper option clears your quality bar — not scramble after it.

3. Budget on the later price, not the promo. This is the core discipline. If your unit economics only work at $4/$20, then November 21 is a 41% cost shock waiting to happen. Model the $5/$30 scenario (or higher), and make sure the business still works. If it doesn't, you have a month to fix it — via caching, batching, or a cheaper model — instead of discovering it on the invoice.

The Bottom Line

GPT-5.6 Sol at $4/$20 is a good price with an expiry date nobody can see past. OpenAI's page says "at least through November 21, 2026" and nothing more. The developers who'll be fine are the ones who measured their usage, tested the alternatives, and budgeted for the worst case while the promo was still running.

November 21 is a month away. That's enough time — if you start now.

Prompt Caching: The 90% Discount You're Missing

Hammad Malik — Sun, 18 Oct 2026 10:16:37 +0000

Originally published at API Cost Lab.

There is a 90% discount sitting in your API bill, and most developers are either not using it or using it in a way that doesn't work. Prompt caching — paying a tenth of the input price for tokens the provider has already seen — is the highest-leverage cost optimization available on modern LLM APIs. Here's how it actually works and how to get the savings.

How Prompt Caching Actually Works

The idea is simple. When you send a request, the provider processes your input tokens through the model — that's the expensive computation you're paying for. But if your next request starts with the exact same tokens — the same system prompt, the same repo context, the same few-shot examples — the provider doesn't need to recompute all of that. It kept the intermediate results cached. So it charges you a fraction of the price for those repeated tokens.

The key phrase is stable prefix. Caching works from the start of your prompt forward: the provider matches your prompt's beginning against what it has cached, and everything that matches gets the discounted rate. The moment the tokens diverge — your new user message, the latest tool result — matching stops and the rest is billed at full price.

This has one big architectural implication: put stable content first, variable content last. System instructions, repo context, few-shot examples, reference documents — all of that goes at the top, byte-identical every time. The thing that changes per request goes at the end. Get this ordering right and the cache does the rest.

The Math: What 90% Off Actually Means

On GPT-5.6 Sol, the numbers are published and stark:

Fresh input: $4.00 per 1M tokens
Cached input reads: $0.40 per 1M tokens

That's the 90% discount. Now run it through the reference workload — 1,100 tasks/month, each with 12K fresh input, 48K cached input, and 6K output tokens:

Fresh input: 12,000 x $4.00/1M = $0.0480 per task
Cached input: 48,000 x $0.40/1M = $0.0192 per task
Output: 6,000 x $20.00/1M = $0.1200 per task

Notice: the 48K cached tokens — four times the volume of fresh tokens — cost less than half as much. Without caching, those 48K tokens at full price would cost $0.1920 per task instead of $0.0192. Across 1,100 tasks a month, caching saves roughly $190/month on this workload alone. That's most of the input bill, gone.

One cost people miss: cache writes. The first time a prompt prefix is cached, the provider charges a premium — on GPT-5.6 Sol, cache writes cost 1.25x the input rate. So the first call with a new prefix costs slightly more than an uncached call. Every reuse after that is nearly free. For anything you call more than a handful of times, the math is overwhelmingly in your favor — but don't model the first call as discounted.

Structuring Prompts for Cache Hits

Caching fails more often from prompt structure than from provider issues. The rules:

1. Fixed order, every time. The stable prefix must be byte-identical across calls — not just semantically the same, but the exact same tokens in the exact same order. Build your prompts programmatically from fixed templates, not by string-concatenating in whatever order the code happens to run.

2. Nothing dynamic in the prefix. Timestamps, request IDs, user names, "today's date" — if any of it sits inside the cached region, every request has a different prefix and nothing ever hits the cache. Move all dynamic content after the stable block, or accept that the cache starts after it.

3. Longest stable content first. Order the stable blocks by how rarely they change: system prompt first (never changes), then reference documents (rarely change), then conversation summaries (change slowly), then the live request (changes every time). This maximizes the cached portion even when later blocks rotate.

4. Don't reshuffle tool definitions. In agentic loops, the tool schemas are often a large, stable block — perfect cache material. But if your framework reorders them or regenerates them per call, the cache breaks. Pin them.

5. Verify, don't assume. After restructuring, check the cached-token ratio in your usage dashboard. You're aiming for 80%+ of input tokens served from cache on repetitive workloads. If you're at 20%, something in the prefix is still varying — diff two consecutive prompts and find it.

Realistic Hit Rates

What's actually achievable? It depends on the workload shape:

Coding agents with stable repo context: 70–90% of input tokens cached is realistic. The system prompt and codebase context barely change between calls; only the task description and tool results vary.
Chatbots with fixed system prompts: 50–80%. The system prompt caches well, but conversation history grows and shifts, which erodes the hit rate over long sessions. Summarizing history periodically instead of resending it raw helps.
One-shot or highly variable tasks: 0–30%. If every request is genuinely different input — document processing, unique queries — there's little repeated prefix to cache. Don't force it; caching is for repetition.

The pattern: the more your workload looks like "same setup, different question," the higher the hit rate. The more it looks like "different everything," the lower.

Which Workloads Benefit Most

Biggest winners: agentic coding loops (huge stable context, repeated thousands of times), RAG systems with fixed document sets, chatbots with long system prompts, eval harnesses running the same prompts across many inputs. These are exactly the workloads where input volume is highest — which is why caching matters so much.

Modest benefit: mixed workloads with partial repetition. You'll get the system prompt cached but not much else. Still worth doing — it's nearly free to set up.

Skip it: truly one-off inference, bulk jobs over unique documents, anything where the input is never repeated. The cache-write premium means you'd pay more for zero benefit.

Debugging a Cache That Won't Hit

When the hit rate stays low after restructuring, the culprit is almost always something small and dynamic hiding in the prefix. The debugging method is unglamorous but effective: capture two consecutive prompts, diff them, and look at exactly which tokens differ. Common offenders:

Timestamps and dates injected by a template ("Current date: ..." at the top of the system prompt is a classic cache-killer)
Session or request IDs interpolated into instructions
Tool definitions regenerated per call with non-deterministic ordering (sort them, or better, serialize once and reuse)
Conversation history prepended before the system prompt instead of appended after it — every new message shifts the entire prefix
Whitespace and formatting differences from string templates that render slightly differently per code path

Fix them one at a time and watch the dashboard after each change. The hit rate should climb in steps; when it plateaus, you've found everything that was varying.

One more subtlety: minimum cacheable lengths. Providers typically require the stable prefix to exceed a minimum token count before caching engages at all — short prompts may never qualify regardless of structure. If your prompts are small (a few hundred tokens), caching may simply not apply, and that's fine: there's little to save on small inputs anyway. Caching earns its keep on large, repeated context.

The Bottom Line

Prompt caching is the closest thing to free money in LLM APIs: a ~90% discount on your largest token category, available for the cost of ordering your prompts correctly. The developers paying full price for repeated context aren't being overcharged — they're just sending the same tokens in a slightly different order every time. Fix the order, check the dashboard, and watch the input line of your invoice collapse.