DEV Community: Andrey Altrouter

Prompt caching has a break-even point, and it's 22%

Andrey Altrouter — Sun, 16 Aug 2026 08:46:58 +0000

Every guide to cutting LLM costs eventually says the same thing: turn on prompt caching. Almost none of them mention that caching can make your bill larger.

It has a break-even point, it is computable, and most teams never check which side of it they're on.

Two words first, because the whole argument lives in them. A token is roughly ¾ of a word; models bill per million of them. Prompt caching stores the front part of your prompt — the system prompt, the tool definitions, the documents you resend every time — so the model doesn't reprocess it on the next request. It matches on an exact prefix: change one byte anywhere near the start and everything after it is a miss.

Cache writes cost more than plain tokens

Here is the part that gets skipped. Caching does not have one price, it has two, and one of them is a penalty.

On Anthropic's API, relative to the normal input price per token:

Operation	Multiplier
Normal input token	1.0×
Cache write, 5-minute TTL	1.25×
Cache write, 1-hour TTL	2.0×
Cache read	0.1×

TTL is time-to-live: how long the entry survives after it was last touched. Five minutes by default.

So a cache hit is a 90% discount, and a cache miss is a 25% surcharge. Every request that doesn't find a warm entry writes a new one and pays extra for the privilege. Caching is a bet: you're wagering that the entry you just paid 1.25× to create will be read at least a couple of times before it expires.

The break-even hit rate

Your hit rate is the share of cacheable tokens served from cache rather than written to it. Call it h, the write multiplier W, and the read multiplier R. Caching pays off when the blended cost drops below plain input:

(1 − h)·W + h·R  <  1
h  >  (W − 1) / (W − R)

Plug the numbers in:

5-minute cache: (1.25 − 1) / (1.25 − 0.1) = 21.7%
1-hour cache: (2 − 1) / (2 − 0.1) = 52.6%

Below roughly a 22% hit rate, the five-minute cache is costing you money. The one-hour TTL doubles the write cost, so it needs more than half your requests to hit before it earns its keep — it exists for bursty traffic with gaps longer than five minutes, not as a default upgrade.

Make it concrete. A support bot on claude-sonnet-5 ($2.00 per 1M input tokens at list price) with a 20K-token prefix and 200 requests a day. Uncached input: $8.00/day. At a 90% hit rate: $1.72/day. At a 15% hit rate — requests spread far enough apart that most entries expire unread — $8.62/day. Same code, same feature flag, a bill that moved 8% in the wrong direction.

"But my traffic is steady" is a reasonable objection, and if it's true you're fine. The teams that get burned are the ones with low-volume production traffic, a per-user prefix that's never shared, or a nightly batch job spread thin across an hour.

Measure your actual hit rate

Don't estimate it. Every response reports it. Run your real traffic, in its real order, through this:

import anthropic
client = anthropic.Anthropic()

created = read = fresh = 0
for prompt in prompts:          # your real requests, in real order
    r = client.messages.create(
        model="claude-sonnet-5", max_tokens=512,
        system=[{"type": "text", "text": SYSTEM,
                 "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": prompt}],
    )
    u = r.usage
    created += u.cache_creation_input_tokens
    read += u.cache_read_input_tokens
    fresh += u.input_tokens

hit = read / (read + created)
billed = created * 1.25 + read * 0.10 + fresh
print(f"hit rate {hit:.0%} | billed {billed / (created + read + fresh):.2f}x uncached")

If hit comes back at 0% across repeated requests with an identical prefix, something is silently invalidating it — a timestamp in the system prompt, a UUID, an unsorted json.dumps, a tool list assembled in a different order. The cache key is the exact bytes.

If cache_creation_input_tokens is 0 too, your prefix is below the minimum cacheable length. That threshold is per-model and it is not monotonic across generations: 512 tokens on Claude Opus 5, 1024 on Opus 4.8 and Sonnet 5, 4096 on Opus 4.6 and Haiku 4.5. A 3K-token prompt caches on one and silently doesn't on the other, with no error either way.

What this doesn't fix

A good hit rate cuts the token count. It does nothing to the other multiplier — the price per token — and your bill is the product of both. That's where a gateway comes in: altrouter.ai resells the same models at 10–25% below the vendors' own list prices (claude-sonnet-5 at $1.69 per 1M input against Anthropic's $2.00), through the OpenAI-compatible API, so switching is a base_url change. Its honest gap on this exact topic: our usage records don't break out cache-read tokens yet, so the measurement above has to come from the provider's response body rather than our dashboard.

The one number to take away

Caching is not free money, it's a bet at fixed odds: 22% hit rate on the five-minute cache, 53% on the one-hour one. Print your hit rate before you argue about breakpoint placement. If it's under the line, the cheapest change you can make today is turning caching off.

The cheap model is only cheap for half your tasks

Andrey Altrouter — Sat, 15 Aug 2026 04:55:32 +0000

Most of us pick a model the same way: read a leaderboard, pick the best one we can afford, ship it. Then the bill arrives and the "cheap" model turns out not to be the cheap one.

The reason is that there is no such thing as a cheap model. There is only a model that is cheap for the shape of your traffic — and the ranking reorders when the shape changes.

Quick vocabulary, because the whole argument lives in two words. A token is roughly ¾ of a word; models bill per million of them. Input tokens are what you send (prompt, files, chat history); output tokens are what the model writes back. They have different prices, and the gap between them is not the same for every model.

The number you don't have yet

Every model's price is two numbers, and every vendor publishes the ratio between them without commenting on it. Here it is, list prices per 1M tokens, snapshot taken 12 Aug 2026:

Model	Input $/1M	Output $/1M	Output is
gemini-3.1-flash-lite	0.25	1.50	6× input
grok-4.3	1.25	2.50	2× input
claude-haiku-4-5	1.00	5.00	5× input
gemini-3.5-flash	1.50	9.00	6× input
claude-sonnet-5	2.00	10.00	5× input

Look at row two. Grok 4.3 charges only twice as much for writing as for reading, where everyone else charges five or six times. That single number decides whether it is expensive or a bargain — and which one it is depends entirely on you.

So measure your own ratio before you compare anything. Every OpenAI-compatible response already carries it:

r = client.chat.completions.create(model=MODEL, messages=msgs)
u = r.usage
print(u.prompt_tokens, u.completion_tokens)

# add it up over a few hundred real requests, then:
def cost(price_in, price_out, tok_in, tok_out):
    return tok_in / 1e6 * price_in + tok_out / 1e6 * price_out

print(cost(1.00, 5.00, total_in, total_out))   # claude-haiku-4-5
print(cost(1.25, 2.50, total_in, total_out))   # grok-4.3

Same two models, opposite verdict

Take Haiku 4.5 and Grok 4.3 and run them through two ordinary jobs, per 1,000 requests, at the list prices above.

Classification — you send 4,000 tokens of document and get back one word (50 tokens). Almost all of the bill is reading.

claude-haiku-4-5: 4M × $1.00 + 0.05M × $5.00 = $4.25
grok-4.3: 4M × $1.25 + 0.05M × $2.50 = $5.12

Haiku wins by 17%. Now the same two models on code generation — 1,500 tokens in, 2,500 tokens out. Now almost all of the bill is writing.

claude-haiku-4-5: 1.5M × $1.00 + 2.5M × $5.00 = $14.00
grok-4.3: 1.5M × $1.25 + 2.5M × $2.50 = $8.12

Grok wins by 42%. Nothing changed except the shape of the traffic. Any blog post that tells you which model is cheapest, without asking what your job looks like, is guessing.

When the upgrade is nearly free

The same arithmetic works in the other direction, and this is the part that surprised me. Compare gemini-3.5-flash ($1.50 / $9.00) with claude-sonnet-5 ($2.00 / $10.00). On that RAG workload — 8,000 in, 700 out, per 1,000 requests — Flash costs $18.30 and Sonnet costs $23.00. You are one quarter away from a model in a completely different class, while the word "flash" in the name suggests you are saving a fortune.

Then add the multiplier nobody puts in the spreadsheet: retries. If the cheap model fails one call in five and you re-run those on the expensive one, you pay for the cheap attempt and the expensive one. At a 20% failure rate that Flash job is really $18.30 + 20% × $23.00 = $22.90 — the discount is gone, and you also shipped worse latency.

Prices below list change the same arithmetic without changing its shape. I work on altrouter.ai, which bills the same vendor models under list — Claude Sonnet 5 at $8.50 per 1M output against the official $10.00, Grok 4.3 at $2.12 against $2.50 — so the crossover points move, but you still have to know your own ratio to find them. It also does not host embedding models, and no discount will rescue a model that keeps failing your task.

The actual decision order

Measure prompt_tokens / completion_tokens on a few hundred real requests. This takes ten minutes and everything else depends on it.
If input dominates (10:1 or more), rank candidates by input price only. Output price is noise; ignore the headline number everyone quotes.
If output dominates (below 2:1), rank by output price only — and specifically look for models with a low output-to-input multiple.
Price the next tier up on your own mix before assuming you can't afford it. If it lands under ~1.5× and it removes a retry, it is cheaper, not dearer.
Re-run this per endpoint, not per app. Your classifier and your code generator are different workloads and deserve different models.

What this doesn't solve

This is arithmetic about price, not about quality. It will not tell you whether Grok 4.3 writes code you'd merge — only you can judge that, on your tasks. It ignores latency, rate limits, and prompt caching, which can each move the answer more than the price gap does. And the numbers are a snapshot from 12 Aug 2026; every one of them will be stale within a quarter, though the method won't be.

Pick the model for the task, not for the app. And before you argue about which one is cheaper, go print your two usage numbers — the argument is usually already settled by them.

Your LLM gateway takes a cut. Seventeen lines of Python tell you how big.

Andrey Altrouter — Fri, 14 Aug 2026 05:18:57 +0000

You wanted to try three models from three vendors, so you did the sensible thing: pointed one client at a gateway, put one key in the environment, and stopped thinking about it. A gateway here is a service that speaks the OpenAI API and forwards your requests to Anthropic, Google, OpenAI, xAI and the rest, so switching models is a string change instead of a new SDK.

What almost nobody checks after that: what the gateway charges for the exact same tokens the vendor would have sold you directly.

What you actually buy from a gateway

Three things, and they are worth very different amounts.

One key and one invoice. Real, and boring. It saves you three signups and three billing pages.

Routing and fallback. If a provider returns 529 or a region goes dark, the gateway retries elsewhere. Worth something on production traffic, nothing on a weekend project.

A price per token. This is the one that varies by an order of magnitude between services, and the one nobody puts in a comparison table — because there is no single "gateway price". There is a price per model, and a service can sit below vendor list on one model and far above it on another.

Vocabulary, in case it is new: models are billed per million tokens (a token is roughly ¾ of a word), input and output priced separately, with output typically 3–10× more expensive. The list price is what the vendor publishes on its own pricing page. That list price is your baseline — a gateway is either below it, at it, or taking a markup on top.

Seventeen lines that answer it

Many gateways publish per-model pricing in their models endpoint. If yours does, this reads it and prices out your token mix, not a benchmark's:

import json, urllib.request

GATEWAY = "https://api.altrouter.ai/v1/models/public"   # your gateway's models endpoint
LIST = {"claude-sonnet-5": (2.00, 10.00), "gpt-5.6": (2.50, 15.00)}  # vendor list $/1M in, out
MIX = (200, 40)  # your monthly millions of tokens: input, output

req = urllib.request.Request(GATEWAY, headers={"User-Agent": "price-check"})
models = json.load(urllib.request.urlopen(req))["data"]

for m in models:
    inp, out = m["pricing"].get("prompt_per_1m_usd"), m["pricing"].get("completion_per_1m_usd")
    if m["id"] not in LIST or inp is None:
        continue
    li, lo = LIST[m["id"]]
    mine = MIX[0] * inp + MIX[1] * out
    vendor = MIX[0] * li + MIX[1] * lo
    print(f"{m['id']:<18} gateway ${mine:>8.2f}  list ${vendor:>8.2f}  {mine / vendor - 1:+.1%}")

Run on 2026-08-14 against the endpoint in the snippet, it prints:

gpt-5.6            gateway $  935.09  list $ 1100.00  -15.0%
claude-sonnet-5    gateway $  677.91  list $  800.00  -15.3%

If your gateway does not publish prices in that endpoint — plenty don't — use the invoice method instead: take last month's charge, divide by the input and output tokens the gateway logged, and you have your real effective $/1M. That number is the only one that matters, and it is the one you compare to the vendor's page.

The obvious objection: the mix is made up. It is — replace MIX with your own two numbers before you believe any of the output. The ratio between input and output tokens is workload-specific, and since output costs several times more, a chat workload and a document-summarizing workload rank gateways differently.

The four things I'd check before switching

Price per model, on your mix. The script above. A gateway that is cheap on a small model can be expensive on the large one you actually run.
Whether base_url is genuinely the only change. Test streaming, tool calls, and the usage object in the response. Those are where compatibility usually breaks.
What happens on a provider outage. Ask for the retry and fallback behaviour explicitly, then unplug a model and watch.
Your exit. If leaving means rewriting anything beyond a URL and a key, that is not a gateway, that is a platform.

Disclosure, since the URL in the snippet is mine: I work on altrouter.ai, which is why I could hardcode an endpoint that publishes its prices. It resells vendor models below list — Claude Sonnet 5 at $1.69 / $8.50 per 1M against the official $2.00 / $10.00, Opus 4.5 at $3.74 / $18.75 against $5.00 / $25.00, discounts running 10–25% by model, prices as of August 2026. Point the script at whatever you use now; the arithmetic does not care.

What this does not tell you

Price, and only price. The script says nothing about latency, uptime, rate limits, or whether prompt caching is passed through at the vendor's discount — check that separately, it can outweigh a 15% price gap. And the honest gaps on my side: no embedding models, no data residency options, no SLA of our own beyond the upstream vendor's. If you are on one vendor and never plan to leave, buy direct — you save exactly the middleman's margin.

Run the seventeen lines on your own gateway this week. Whatever the number is, it is better to know it than to assume the convenience was free.

Your LLM bill has two levers. Your team only pulls one.

Andrey Altrouter — Thu, 13 Aug 2026 07:19:53 +0000

Every guide to cutting LLM costs is about the same thing: send fewer tokens. Trim the system prompt, cache the prefix, retrieve less, route easy requests to a small model. All of it works. And all of it is one half of the equation.

Your bill is:

bill = tokens × price_per_token

Two factors. Multiplication. Every cost-cutting article I have read attacks the left one and treats the right one as a law of physics. It is not — it is a number on a pricing page, and pricing pages are not the only place those models are sold.

The two levers, in the same units

A quick vocabulary check, because the whole argument lives in these words. A token is roughly ¾ of a word — models are billed per million of them. Input tokens are what you send (prompt, files, chat history), output tokens are what the model writes back. Output costs 3–10× more per token, which is why it usually dominates the bill. Prompt caching means the provider stores your unchanged prefix and re-reads it at a large discount, often around 10% of the input price — you pay a small premium the first time to write the cache, then read it cheaply.

Now, one workload. A team runs a coding agent, and last month it burned 200M input and 40M output tokens on Claude Sonnet 5. At list price ($2.00 in / $10.00 out per 1M):

200 × $2.00 + 40 × $10.00 = $400 + $400 = $800

Lever one — fewer tokens. Say 70% of that input is a stable prefix (the same repo files, the same instructions) and you make it cacheable. That 140M drops to roughly $0.20 per 1M instead of $2.00. Input becomes 140 × $0.20 + 60 × $2.00 = $148. Output does not cache — it is generated fresh every time. New total: $548, a 31% cut. That is a good month of engineering work.

Lever two — cheaper tokens. Same tokens, same model, price reduced 15%. $800 becomes $680. Zero engineering work.

Neither is the interesting part. The interesting part is that they multiply:

Configuration	Monthly bill
List price, no caching	$800.00
Caching only	$548.00
Cheaper price only	$680.00
Both	$465.80

Pulling both levers is a 42% cut. Pulling the one everybody writes about is 31%.

Why the second lever gets ignored

Because it does not feel like engineering. Caching is a diff you can review, benchmark, and put in a changelog. "We pay less per token" is a procurement decision, so it falls between the team that owns the code and the team that owns the invoice, and nobody optimizes it.

It is also the lever with no quality tradeoff. Every token-side optimization trades something: caching adds cache-invalidation bugs, retrieval trims context the model might have needed, routing to a smaller model risks a worse answer that costs more in retries. Changing the price per token changes exactly one thing — the price per token. The weights, the context window, and the output are identical.

Where the price is actually negotiable

Disclosure before the numbers: I work on altrouter.ai, a gateway that resells the same vendor models below their list price — Claude Sonnet 5 at $1.69 / $8.50 per 1M against the official $2.00 / $10.00, Opus 4.5 at $3.74 / $18.75 against $5.00 / $25.00, GPT-5.6 at $2.13 / $12.75 against $2.50 / $15.00. Discounts run 10–25% depending on the model, the API is OpenAI-compatible, so switching is a base_url change. Prices as of August 2026.

The honest gaps, so you hear them from me and not from the comments: no embedding models, no data residency options, no SLA of our own beyond what the upstream vendor delivers, and no invoicing for legal entities yet. If any of those is a requirement, this lever is closed for you and lever one is all you have.

What to do this week

Pull last month's spend and split it into input, output, and cached input. If you cannot produce that split, you cannot price anything — in/out prices differ by 3–10×, and cached input by another 10×.
Multiply your actual token counts by two or three candidate prices. Not benchmark tokens — yours. The ratio is workload-specific.
Do the caching work anyway. The levers multiply; skipping one to take the other is leaving the smaller half on the table.

The point is not that one lever beats the other. It is that you are running an optimization with two variables and holding one constant for no reason other than habit.

Your model benchmark measures everything except the bill

Andrey Altrouter — Wed, 12 Aug 2026 11:30:17 +0000

Every model benchmark I read ranks the same two things: how good the output was, and sometimes how many tokens it took. Almost none of them print the number you actually get charged for.

That gap matters, because the two rankings are not the same list. A model can win on quality, lose on tokens, and still be the cheapest way to finish your task — or the opposite. You cannot tell which without doing the multiplication yourself.

Here is the multiplication.

The formula

cost = (input_tokens / 1M) × price_in + (output_tokens / 1M) × price_out

That is the whole thing. Two numbers from the benchmark, two numbers from the vendor's pricing page. The reason it is worth writing down is that every term in it moves independently, and benchmarks only ever report the first two.

A worked example

Take one coding task. Say it consumes 40K input tokens and 12K output tokens — the shape of a real agent run with a few files in context and a couple of iterations. These numbers are an assumption for the example, not a measurement; substitute your own from your logs.

Vendor list prices per 1M tokens, as of August 2026:

Model	in $	out $	Cost of one task	100 tasks
Claude Opus 4.5	5.00	25.00	$0.500	$50.00
GPT-5.6	2.50	15.00	$0.280	$28.00
Claude Sonnet 5	2.00	10.00	$0.200	$20.00
Grok 4.5	2.00	6.00	$0.152	$15.20
Claude Haiku 4.5	1.00	5.00	$0.100	$10.00
Gemini 3 Flash	0.50	3.00	$0.056	$5.60

Nine times between the top and the bottom row. No leaderboard shows you that spread, because no leaderboard multiplies.

The trap in the cheap column

Now the part that makes the table dangerous if you read it naively.

Token counts are not a property of the task. They are a property of the model on that task. A weaker model that misunderstands the requirement, retries twice, re-reads the whole file and asks a clarifying question burns three or four times the tokens of a model that gets it in one pass.

Run the arithmetic again. Haiku 4.5 is five times cheaper per token than Sonnet 5. If it needs four times the tokens to converge, one task costs $0.40 instead of $0.10 — and Sonnet, at $0.20, is now twice as cheap as the "cheap" model.

This is why "just use a smaller model" is advice, not a strategy. The only way to know is to log input and output tokens per task, per model, and multiply. Both numbers, separately: output usually dominates the bill, and it is the one that explodes when a model starts thinking out loud.

Same reason time-to-finish deserves a column. A model that produces correct code after forty minutes of tool calls is not free just because each call is small.

What to actually do

Log tokens per task, split into input and output. If your framework does not expose them, most gateways do. Without the split you cannot price anything, because in/out prices differ by 3–10×.
Exclude cache reads or price them separately. Cached input is usually an order of magnitude cheaper; mixing it in makes your numbers look better than reality.
Price your top three candidates on your own tasks, not on a public benchmark's tasks. The token ratio is workload-specific — long-context refactoring and short algorithmic prompts produce completely different bills.
Recompute quarterly. Model prices move, and new models land every few weeks at a different point on the curve.

Do this once and the model choice stops being a debate about vibes. It becomes a number with a currency symbol on it.

Where the price itself is negotiable

One more term in that formula is worth attention, and it is the one everybody treats as fixed: price_in and price_out.

Vendor list price is not the only price those models sell at. I work on altrouter.ai, a gateway that resells the same models below the vendor's list — Claude Opus 4.5 at $3.74 / $18.75 per 1M against the official $5.00 / $25.00, Sonnet 5 at $1.69 / $8.50 against $2.00 / $10.00, and so on across 44 models on an OpenAI-compatible API. The discount is a standing price, shown next to the official one on every model page, not an intro offer.

Plug it into the same example: the 100-task run on Opus 4.5 costs $37.44 instead of $50.00. Same weights, same tokens, same code — only the third and fourth term of the formula changed.

I am obviously not a neutral party on that last paragraph, so treat it the way you should treat the rest of this post: as arithmetic you can check. The formula is public, the list prices are public, and your token counts are in your own logs.