DEV Community: Andrey Altrouter

You typed 6,000 tokens and got billed for 282,000

Andrey Altrouter — Fri, 11 Sep 2026 06:03:44 +0000

The first LLM bill that surprises you is almost never the one for a big job. It's the one for a chat feature that "barely gets used."

The reason is a single line most of us write on autopilot: messages.append(...). It looks like you're adding one message. You're actually re-buying every message that came before it.

Two words first, since the whole argument lives in them. A token is roughly ¾ of a word; models bill per million of them, at two different prices — input tokens (everything you send) and output tokens (what the model writes back), with output usually 5–6× the price of input. And the API is stateless: it remembers nothing between calls. The "memory" in your chat bot is you, resending the entire transcript on every single turn.

The cost grows with the square of the turn count

Say a system prompt of 500 tokens, user messages of 200, and replies of 400. On turn k, what you send is the system prompt, plus everything already said, plus the new question:

input(k) = 500 + 600·(k − 1) + 200

That 600·(k − 1) is the killer. It grows every turn, and you pay it again every turn. Sum it across a conversation of n turns and the total input is roughly 300·n² — quadratic, not linear.

Turns	Input tokens billed	Output tokens billed
10	34,000	4,000
30	282,000	12,000
60	1,104,000	24,000

Doubling the conversation from 30 turns to 60 didn't double the cost. It nearly quadrupled it. And the user in that 30-turn conversation typed 6,000 tokens of questions — you were billed for 282,000, a factor of 47.

What that costs in actual dollars

Numbers as of 2026-09-03, on claude-sonnet-5 at Anthropic's list price of $2.00 per 1M input and $10.00 per 1M output.

One 30-turn conversation: 282,000 input = $0.564, 12,000 output = $0.12. $0.68 a conversation — and 82% of it is history you already paid for.

Ten thousand of those a month is $6,840. That's the bill people describe as "coming out of nowhere," because the intuitive estimate — count the questions and answers, 18,000 tokens a conversation — lands at about $0.09. Off by 7×.

Note which side the money is on. Everyone tunes max_tokens and trims the model's replies; output is $0.12 of that $0.68. The bill is in the input, and the input is your own transcript.

Cap the history and the curve goes flat

The fix is a sliding window: keep the system prompt, keep the last N exchanges, drop the rest. Once the window is full, input(k) stops growing, and quadratic becomes linear.

SYSTEM = {"role": "system", "content": "..."}

def window(history, keep_turns=8):
    """System prompt + the last N user/assistant pairs."""
    return [SYSTEM] + history[-keep_turns * 2:]

# Before sending, see what you're actually paying for:
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
sent = window(history)
print(sum(len(enc.encode(m["content"])) for m in sent), "input tokens this turn")

On the same 30-turn conversation, an 8-turn window bills 143,400 input tokens instead of 282,000 — 49% off, from four lines of code. Print that number on every request for one afternoon and you will know whether you have this problem, which is more than most teams can say.

If the tail of the conversation genuinely matters, summarize it instead of dropping it: fold turns 1–20 into a 150-token summary with a cheap model and prepend that. Same shape, less amnesia.

What this doesn't fix

A window shrinks the token count. It does nothing to the other multiplier — the price per token — and your bill is the product of both. That's the case for a gateway: altrouter.ai resells the same models 10–25% below the vendors' own list prices (claude-sonnet-5 at $1.69 per 1M input against Anthropic's $2.00), over the OpenAI-compatible API, so it's a base_url change. The 30-turn conversation above comes to $0.58 instead of $0.68. Its honest gap on this exact topic: our usage log stores prompt tokens per request, with no conversation id, so it can't point at the conversation that went quadratic. That grouping has to happen in your app.

Prompt caching is the other half of the answer, and it works well here — the growing prefix is stable — but it has its own break-even point and is not automatic.

The one number to take away

Input tokens in a chat scale as n², and the user's typing is a rounding error inside them. Before you optimize anything else, log the input-token count of every request and sort descending. The top of that list is your bill.

622 of our 5,087 LLM API calls never returned an answer

Andrey Altrouter — Fri, 11 Sep 2026 06:01:29 +0000

Every tutorial about LLM pricing counts calls. You send a request, you get an answer, you multiply. The arithmetic is clean because it assumes something that isn't true: that a call returns an answer.

Between 28 June and 10 September 2026 our gateway logged 5,087 chat completions. 622 of them — 12.2% — ended with no answer for the caller. That is not an outage. That is the normal weather of a production LLM client, and nobody puts it in the cost model.

Two words before the numbers. A token is roughly ¾ of a word; you are billed per million, separately for input (what you send) and output (what the model writes back). And every failure arrives as an HTTP status code — a three-digit number where 4xx means "your request was wrong" and 5xx means "something on our side broke". That distinction turns out to be the whole article.

What actually fails

Outcome	Status	Share of all calls
no provider available	503	6.88%
invalid request	400	2.40%
client closed connection	499	1.51%
timeout	504	0.94%
out of credit	402	0.28%

The biggest slice isn't your code. It's 503 — the upstream vendor had no capacity for that model at that second. The second biggest is your code: a malformed request, an unsupported parameter, a role the endpoint doesn't accept.

That split matters because most retry wrappers don't make it. The default shape everyone copies — catch the exception, sleep, try again, three times — treats a 400 exactly like a 503. A 503 has a real chance of succeeding on the next attempt. A 400 will fail identically until the heat death of the universe. Of our 622 failures, 137 were terminal in that way: 400, 402, 404. Retried three times each, that is 411 requests that could never have produced anything.

The one failure that costs real money

Here is the part that surprised me. A rejected request is cheap. A 400 never reaches the model, so no tokens are generated and nothing is billed. A 503 means the request never started. You lose latency, not dollars.

Timeouts and cancellations are different. A 504 means the model was generating — you just stopped waiting. A 499 means your own user hit stop, or your HTTP client's deadline fired, after the answer had already started coming back. The tokens exist. Somebody generated them. In our logs that's 2.45% of all calls, and it is the only category where the meter was actually running.

Then you retry, and the second attempt pays for the entire input again. Not the remainder — the whole prompt, from the system message down. This is why the number to track is attempts per answer, not calls. At gpt-5.6's official $2.50 per million input tokens, our median prompt of 1,290 tokens costs about a third of a cent per attempt. Trivial, until a job with a 40,000-token context retries twice under load and you've bought that context three times for one answer.

What to change today

Two lines of policy, and they're both about classification rather than volume.

RETRYABLE = {429, 500, 502, 503, 504}

def call_with_budget(send, payload, max_input_tokens=60_000):
    spent = 0
    for attempt in range(3):
        try:
            return send(payload)
        except APIError as e:
            spent += payload["approx_input_tokens"]
            if e.status not in RETRYABLE or spent > max_input_tokens:
                raise
            time.sleep(2 ** attempt)
    raise RuntimeError("retry budget exhausted")

First: retry only 429 and 5xx. Everything in the 4xx family except 429 is a bug report addressed to you, and sleeping on it won't fix your JSON. Second: cap the retry budget in tokens, not attempts. Three attempts on a 500-token prompt and three attempts on a 40,000-token prompt are the same line of code and an eighty-fold difference in what you bought. The budget is the thing that scales; the attempt count isn't.

One more, cheaper still: turn on streaming, where tokens arrive as they're produced instead of all at once at the end. A timeout on a streamed response leaves you holding a partial answer you can show or salvage. A timeout on a blocking call leaves you holding nothing, having paid the same.

What this doesn't tell you

Our own gateway records zero tokens and zero charge for all 622 of those events, because we only meter what the upstream returns in its usage block — and a request that died mid-generation doesn't return one. So I can show you exactly how often calls fail, and I cannot show you what an abandoned generation cost upstream. That's a real gap in our metering, not a clean bill of health, and it's the number I'd most like to have.

Your own logs probably have the same hole. Fill it before you trust any cost model built on call counts. Count attempts.

Your model reads 1M tokens and writes back 8K — and the gap lands on your input bill

Andrey Altrouter — Fri, 04 Sep 2026 06:50:20 +0000

The first number you see on a model page is the context window: 200K, 400K, a million. It reads like capacity — "I can feed it my whole repo." True, and beside the point.

The number that decides your bill is the other one, printed smaller: how much the model is allowed to write in a single reply. It's usually 10 to 30 times smaller than the context window, and it quietly turns one job into a loop.

Three words first, because the whole argument lives in them. A token is roughly ¾ of a word — models bill per million of them, at one price for input (what you send) and a higher one for output (what the model writes). The context window is how many tokens one request may contain. And max output tokens — max_tokens in most APIs — is the ceiling on a single reply. When the reply hits it, the API doesn't crash: it stops mid-sentence and sets finish_reason: "length". You are billed in full for the truncated half.

The gap is real, and it changes with the route

Here is the same family of models, seen from two places — the numbers on the left are what one OpenAI-compatible gateway exposes, the ones on the right are Anthropic's own published limits:

Model / route	Context window	Max output	Ratio
Gemini 3 Pro, via gateway	1,000,000	32,000	31×
Claude Sonnet 5, via gateway	200,000	8,192	24×
Claude Sonnet 5, first-party API	1,000,000	128,000	8×

Same model name, a 15× difference in how much it will write per call. The cap is not a property of the model alone — it's a property of the model plus the road you reach it by, and nobody puts it in the marketing copy. Check it yourself: the Models API returns it, and any gateway's /v1/models should too.

What the cap actually costs

I ran the arithmetic on a real corpus: the 46 Russian posts in our blog's content/ directory, 138,108 tokens counted with o200k_base, the tokenizer behind the current GPT models. The job: produce an English version of all of it.

Reading it is free of drama — 138K fits in every context window in the table above, once. Writing it back is the problem. Output is roughly as long as input, so with an 8,192-token cap the job takes 17 calls; with 128,000 it takes 2.

Seventeen calls wouldn't matter if each one were cheap. But the obvious way to keep 17 chunks consistent is to hand the model the whole corpus every time — and now you've sent 138K tokens seventeen times.

import math
CORPUS = 138_108          # tokens of material you need rewritten

for cap in (8_192, 128_000):
    calls = math.ceil(CORPUS / cap)
    naive_input = calls * CORPUS      # full context resent every call
    print(f"cap {cap}: {calls} calls, {naive_input/1e6:.2f}M input tokens")

cap 8192: 17 calls, 2.35M input tokens against cap 128000: 2 calls, 0.28M. At GPT-5.6's official $2.50 per million input and $15 per million output, that's $7.94 versus $2.76 for byte-identical work. The output bill never moved — 138K tokens either way. The cap multiplied the input side, 8.4×.

The fix is boring and it works

Stop resending the whole thing. Send the chunk you're rewriting plus a short shared brief — a glossary, a style note, the previous chunk's last paragraph. Input then totals about one corpus, not seventeen: $2.42 for the same job, cheaper than the 128K version, because the big cap wasn't the point. Cheap context makes resending feel free; it isn't.

Two habits worth building the same week. Read finish_reason on every call and treat "length" as an error, not a shrug — a truncated answer that your code cheerfully parses is worse than a crash. And set max_tokens explicitly instead of inheriting a default: gateways often impose their own (ours truncates at 4,096 when you omit it), and "the answer got cut off" is the most common first bug in LLM code.

What this doesn't solve

Chunking costs you consistency — terminology drifts between chunk 4 and chunk 12, and a shared glossary only narrows the drift. Prompt caching pushes the resend price down a lot when the prefix is stable, which changes the arithmetic above but not its direction; I can't quote our own hit rates, because our usage records don't split cache tokens out yet. And nothing here helps when a single indivisible answer genuinely exceeds the cap — then the cap is a hard wall and you need a route with a bigger one.

For reference, the per-token prices I used are the vendors' official ones. We resell the same models 10–25% below those (GPT-5.6 at $2.13/$12.75 against the official $2.50/$15.00), which shaves the bill by a fixed percentage — it does not change the multiplier. A 15% discount on seventeen redundant calls is still seventeen redundant calls.

Go look up the output cap of whatever model you're calling right now. If you don't know it, you're not budgeting — you're guessing.

Your JSON output costs 2.6 more than the same data as CSV

Andrey Altrouter — Thu, 03 Sep 2026 08:45:16 +0000

Everyone writes response_format={"type": "json_object"} and moves on. It's the obvious choice: your code needs structured data, JSON is structured data, done.

Then the extraction job runs on a hundred thousand rows and the bill arrives, and a surprising share of it went to curly braces, colons and the word subscription_status retyped ten thousand times.

Two words first, since the argument lives in them. A token is roughly ¾ of a word — models bill per million of them. And they bill at two different prices: input tokens (what you send) and output tokens (what the model writes back), with output typically 5–6× the price of input. Formatting you ask the model to produce lands on the expensive side.

The same 200 records, five formats

I generated 200 synthetic customer records — id, name, email, city, amount, signup date, status — and counted them with OpenAI's o200k_base tokenizer, the one behind the current GPT models. Same data every time, only the wrapper changed:

Format	Tokens per row	vs pretty JSON
JSON, `indent=2`	77.8	—
JSON, compact	52.8	−32%
JSONL	53.8	−31%
CSV	29.9	−62%
TSV	29.8	−62%

Pretty-printed JSON costs 2.6× what CSV costs for byte-identical information. Nothing about the data changed. The model did the same work. You just asked it to type more.

Where the tokens actually go

Two places, and both are avoidable.

Indentation is 25 tokens a row. That's the gap between pretty and compact JSON — 32% of the bill, spent on whitespace that no parser needs and no human will read, because this output goes straight into json.loads.

Key names are another 29 tokens a row. "customer_id":, "email_address":, "subscription_status": — the schema, re-typed for every single record. Across 200 rows that's 5,800 tokens spent restating a header you already knew (net 23 a row once CSV's own commas are counted back in).

The tempting fix is renaming fields to a, b, c. I measured that too: it takes compact JSON from 52.8 to 44.8 tokens a row. Eight tokens, in exchange for a payload nobody can debug. Dropping the keys entirely — one header line, then rows — saves 23 net. CSV is short-keys done properly.

What it costs in dollars

Take a bulk extraction job on gpt-5.6 at its list price of $15.00 per 1M output tokens, producing 1,000 rows per call, 100 calls a day.

Pretty JSON: 77,765 output tokens per call → $1.17 a call, $117 a day
CSV: 29,885 output tokens per call → $0.45 a call, $45 a day

That's $2,155 a month for the indentation and the repeated field names. The prompt is one sentence different.

Count yours before you argue about it

Don't take my row numbers — yours have different field names and different value lengths, and both move the ratio. Fifteen lines:

import json, csv, io, tiktoken

enc  = tiktoken.get_encoding("o200k_base")   # current GPT models
rows = json.load(open("sample_output.json")) # 100+ real rows

buf = io.StringIO()
w = csv.DictWriter(buf, fieldnames=list(rows[0]))
w.writeheader(); w.writerows(rows)

for name, text in [("json pretty", json.dumps(rows, indent=2)),
                   ("json compact", json.dumps(rows, separators=(",", ":"))),
                   ("csv", buf.getvalue())]:
    n = len(enc.encode(text))
    print(f"{name:14s} {n:7d} tokens  {n / len(rows):6.1f}/row")

If the gap is under 20%, your values are long enough that the wrapper doesn't matter — keep JSON and go optimize something else. If it's the 60% I measured, your records are short and mostly schema.

What this doesn't fix

Three honest limits. This is a bulk trick: for a single object with five fields, the difference is pennies and JSON's error-tolerance is worth more. Tool calling and JSON mode force JSON on you — the model's structured-output path emits it and you can't ask for CSV there, so the win is limited to plain-text completions you parse yourself. And CSV needs real escaping: commas and newlines inside values will corrupt a naive split(","), so use csv.reader, or TSV, which measured identically and collides with far less.

Also, I counted with OpenAI's tokenizer. Anthropic and Google tokenize differently, so the ratio holds directionally but the per-row numbers won't transfer exactly — recount on the model you actually bill.

Format work shrinks the token count. It does nothing to the other multiplier, the price per token, and your bill is the product of the two. That's the gateway's half: altrouter.ai resells the same models 10–25% below the vendors' list prices — gpt-5.6 at $12.75 per 1M output against OpenAI's $15.00 — over the OpenAI-compatible API, so it's a base_url change. Its honest gap here: we don't expose a token-counting endpoint, so the measurement above still has to run locally against the vendor's own tokenizer.

The one number to take away

77.8 versus 29.9 tokens per row. Before you tune anything else about an extraction pipeline, look at what format you asked for — and whether anybody was ever going to read the indentation you're paying output price to generate.

The same six-second AI video costs $0.27 or $3.20

Andrey Altrouter — Wed, 02 Sep 2026 04:34:44 +0000

If you call a text model, you know roughly what a request costs. Everyone prices per million tokens, so you can put two vendors in one column and subtract.

Images and video have no such habit. Four vendors, four units, and nothing on the pricing pages lines up. So most teams pick a video model the way they pick a font, then find the number at the end of the month.

The units don't line up

Here is what you actually get quoted, all of it real:

Per second of output. Seedance, Hailuo, Wan, PixVerse, Kling. You pay for duration.
Per clip, flat. Veo 3.1 charges one price whether you asked for 4, 6 or 8 seconds. Duration is free.
Per image. Most image models: one price per generation, resolution included.
Per image, with a resolution tier. GPT Image 2 and Nano Banana Pro charge separately for a 4K render.

Four units means four different mistakes. A per-second model looks cheap in the docs and isn't, because your clips are 10 seconds. A flat-rate model looks expensive and isn't, because you were going to generate the maximum length anyway. And a "per image" price tells you nothing until you know whether your resolution is included in it.

None of this is hidden. It's just spread across four vendor sites in four shapes, and nobody does the arithmetic before shipping.

One six-second clip, five prices

So do the arithmetic. Same job — one 6-second, text-to-video clip, no retries — at vendor list prices as recorded on 2026-08-27. Every model here supports a 6-second output, so nothing is being compared across durations:

Model	Metered as	List price	One 6s clip
Veo 3.1	per clip	$3.20 / clip	$3.20
Seedance 2.5	per second	$0.473 / s	$2.84
Wan 2.7 Video	per second	$0.10 / s	$0.60
Hailuo 2.3	per second	$0.0467 / s	$0.28
PixVerse V6	per second	$0.045 / s	$0.27

Twelve times, top to bottom, for the same six seconds. For comparison, the gap people actually argue about in text — Claude Opus 4.5 at $25 per 1M output tokens against Claude Sonnet 5 at $10 — is 2.5x. Video is where the money is, and it's the surface nobody benchmarks in dollars.

"They're not the same quality" is the fair objection, and it's true — Veo lands shots that Hailuo doesn't. The point isn't that they're interchangeable. It's that a 12x spread deserves a deliberate decision, and right now it usually gets a default.

The two traps inside the units

Flat-rate clips punish short videos. Veo 3.1 costs $3.20 whether you generate 4 seconds or 8. That's $0.80 per second at 4 seconds and $0.40 at 8. If your product shows 4-second loops, you are paying double per second of what you ship. Generate at the maximum length and trim in ffmpeg — same bill, more material.

The 4K surcharge exists on some models and not others. Taking two models off one price list, so the comparison is apples to apples: GPT Image 2 charges 2.2x its standard-resolution price for a 4K render, while Seedream V4 charges exactly the same for both. So "just render everything at 4K" is free on one model and more than doubles the bill on the next, with the same line of code.

Normalize before you compare

One function, and every price page becomes one column. Feed it the numbers off the vendor docs:

def clip_cost(price, unit, seconds=6, n=1):
    """Cost of n outputs, normalized. unit: per_second | per_clip | per_image"""
    if unit == "per_second":
        return price * seconds * n
    return price * n            # per_clip and per_image ignore duration

catalog = [
    ("veo-3.1",       3.20,   "per_clip"),
    ("seedance-2.5",  0.473,  "per_second"),
    ("wan-2.7-video", 0.10,   "per_second"),
    ("hailuo-2.3",    0.0467, "per_second"),
    ("pixverse-v6",   0.045,  "per_second"),
]

for name, price, unit in sorted(catalog, key=lambda r: clip_cost(r[1], r[2])):
    print(f"{name:15} ${clip_cost(price, unit, n=1000):>8,.0f} per 1000 clips")

Run it with your own clip length and your own monthly volume before you compare anything. At 1000 clips a month the table above spans $270 to $3,200 — that's a hiring decision, decided by a string in a config file.

What this doesn't fix

Normalizing tells you the cheapest way to buy a given model. It doesn't change the price of the model itself. That's the second lever: altrouter.ai resells the same media models below the vendors' own list prices — Hailuo 2.3 at $0.0375 per second against MiniMax's $0.0467, Veo 3.1 at $2.55 a clip against Google's $3.20 — through one API with the same parameters.

The honest gap, on this exact topic: for most video models our catalog flags its own per-second price as approximate, because the upstream meters in credits rather than seconds. Trust the charge that lands in your usage log over any table, including ours and including this one. Video generation is also asynchronous everywhere — you POST a job and poll for it — so there's no partial billing to inspect mid-render if a clip comes back unusable.

The one number to take away

Pick a unit, convert everything into it, then choose. One clip length, one resolution, one column of dollars. It's twenty minutes of arithmetic against a 12x spread, and it's the only comparison the vendor pages will never do for you.

Prompt caching has a break-even point, and it's 22%

Andrey Altrouter — Sun, 16 Aug 2026 08:46:58 +0000

Every guide to cutting LLM costs eventually says the same thing: turn on prompt caching. Almost none of them mention that caching can make your bill larger.

It has a break-even point, it is computable, and most teams never check which side of it they're on.

Two words first, because the whole argument lives in them. A token is roughly ¾ of a word; models bill per million of them. Prompt caching stores the front part of your prompt — the system prompt, the tool definitions, the documents you resend every time — so the model doesn't reprocess it on the next request. It matches on an exact prefix: change one byte anywhere near the start and everything after it is a miss.

Cache writes cost more than plain tokens

Here is the part that gets skipped. Caching does not have one price, it has two, and one of them is a penalty.

On Anthropic's API, relative to the normal input price per token:

Operation	Multiplier
Normal input token	1.0×
Cache write, 5-minute TTL	1.25×
Cache write, 1-hour TTL	2.0×
Cache read	0.1×

TTL is time-to-live: how long the entry survives after it was last touched. Five minutes by default.

So a cache hit is a 90% discount, and a cache miss is a 25% surcharge. Every request that doesn't find a warm entry writes a new one and pays extra for the privilege. Caching is a bet: you're wagering that the entry you just paid 1.25× to create will be read at least a couple of times before it expires.

The break-even hit rate

Your hit rate is the share of cacheable tokens served from cache rather than written to it. Call it h, the write multiplier W, and the read multiplier R. Caching pays off when the blended cost drops below plain input:

(1 − h)·W + h·R  <  1
h  >  (W − 1) / (W − R)

Plug the numbers in:

5-minute cache: (1.25 − 1) / (1.25 − 0.1) = 21.7%
1-hour cache: (2 − 1) / (2 − 0.1) = 52.6%

Below roughly a 22% hit rate, the five-minute cache is costing you money. The one-hour TTL doubles the write cost, so it needs more than half your requests to hit before it earns its keep — it exists for bursty traffic with gaps longer than five minutes, not as a default upgrade.

Make it concrete. A support bot on claude-sonnet-5 ($2.00 per 1M input tokens at list price) with a 20K-token prefix and 200 requests a day. Uncached input: $8.00/day. At a 90% hit rate: $1.72/day. At a 15% hit rate — requests spread far enough apart that most entries expire unread — $8.62/day. Same code, same feature flag, a bill that moved 8% in the wrong direction.

"But my traffic is steady" is a reasonable objection, and if it's true you're fine. The teams that get burned are the ones with low-volume production traffic, a per-user prefix that's never shared, or a nightly batch job spread thin across an hour.

Measure your actual hit rate

Don't estimate it. Every response reports it. Run your real traffic, in its real order, through this:

import anthropic
client = anthropic.Anthropic()

created = read = fresh = 0
for prompt in prompts:          # your real requests, in real order
    r = client.messages.create(
        model="claude-sonnet-5", max_tokens=512,
        system=[{"type": "text", "text": SYSTEM,
                 "cache_control": {"type": "ephemeral"}}],
        messages=[{"role": "user", "content": prompt}],
    )
    u = r.usage
    created += u.cache_creation_input_tokens
    read += u.cache_read_input_tokens
    fresh += u.input_tokens

hit = read / (read + created)
billed = created * 1.25 + read * 0.10 + fresh
print(f"hit rate {hit:.0%} | billed {billed / (created + read + fresh):.2f}x uncached")

If hit comes back at 0% across repeated requests with an identical prefix, something is silently invalidating it — a timestamp in the system prompt, a UUID, an unsorted json.dumps, a tool list assembled in a different order. The cache key is the exact bytes.

If cache_creation_input_tokens is 0 too, your prefix is below the minimum cacheable length. That threshold is per-model and it is not monotonic across generations: 512 tokens on Claude Opus 5, 1024 on Opus 4.8 and Sonnet 5, 4096 on Opus 4.6 and Haiku 4.5. A 3K-token prompt caches on one and silently doesn't on the other, with no error either way.

What this doesn't fix

A good hit rate cuts the token count. It does nothing to the other multiplier — the price per token — and your bill is the product of both. That's where a gateway comes in: altrouter.ai resells the same models at 10–25% below the vendors' own list prices (claude-sonnet-5 at $1.69 per 1M input against Anthropic's $2.00), through the OpenAI-compatible API, so switching is a base_url change. Its honest gap on this exact topic: our usage records don't break out cache-read tokens yet, so the measurement above has to come from the provider's response body rather than our dashboard.

The one number to take away

Caching is not free money, it's a bet at fixed odds: 22% hit rate on the five-minute cache, 53% on the one-hour one. Print your hit rate before you argue about breakpoint placement. If it's under the line, the cheapest change you can make today is turning caching off.

The cheap model is only cheap for half your tasks

Andrey Altrouter — Sat, 15 Aug 2026 04:55:32 +0000

Most of us pick a model the same way: read a leaderboard, pick the best one we can afford, ship it. Then the bill arrives and the "cheap" model turns out not to be the cheap one.

The reason is that there is no such thing as a cheap model. There is only a model that is cheap for the shape of your traffic — and the ranking reorders when the shape changes.

Quick vocabulary, because the whole argument lives in two words. A token is roughly ¾ of a word; models bill per million of them. Input tokens are what you send (prompt, files, chat history); output tokens are what the model writes back. They have different prices, and the gap between them is not the same for every model.

The number you don't have yet

Every model's price is two numbers, and every vendor publishes the ratio between them without commenting on it. Here it is, list prices per 1M tokens, snapshot taken 12 Aug 2026:

Model	Input $/1M	Output $/1M	Output is
gemini-3.1-flash-lite	0.25	1.50	6× input
grok-4.3	1.25	2.50	2× input
claude-haiku-4-5	1.00	5.00	5× input
gemini-3.5-flash	1.50	9.00	6× input
claude-sonnet-5	2.00	10.00	5× input

Look at row two. Grok 4.3 charges only twice as much for writing as for reading, where everyone else charges five or six times. That single number decides whether it is expensive or a bargain — and which one it is depends entirely on you.

So measure your own ratio before you compare anything. Every OpenAI-compatible response already carries it:

r = client.chat.completions.create(model=MODEL, messages=msgs)
u = r.usage
print(u.prompt_tokens, u.completion_tokens)

# add it up over a few hundred real requests, then:
def cost(price_in, price_out, tok_in, tok_out):
    return tok_in / 1e6 * price_in + tok_out / 1e6 * price_out

print(cost(1.00, 5.00, total_in, total_out))   # claude-haiku-4-5
print(cost(1.25, 2.50, total_in, total_out))   # grok-4.3

Same two models, opposite verdict

Take Haiku 4.5 and Grok 4.3 and run them through two ordinary jobs, per 1,000 requests, at the list prices above.

Classification — you send 4,000 tokens of document and get back one word (50 tokens). Almost all of the bill is reading.

claude-haiku-4-5: 4M × $1.00 + 0.05M × $5.00 = $4.25
grok-4.3: 4M × $1.25 + 0.05M × $2.50 = $5.12

Haiku wins by 17%. Now the same two models on code generation — 1,500 tokens in, 2,500 tokens out. Now almost all of the bill is writing.

claude-haiku-4-5: 1.5M × $1.00 + 2.5M × $5.00 = $14.00
grok-4.3: 1.5M × $1.25 + 2.5M × $2.50 = $8.12

Grok wins by 42%. Nothing changed except the shape of the traffic. Any blog post that tells you which model is cheapest, without asking what your job looks like, is guessing.

When the upgrade is nearly free

The same arithmetic works in the other direction, and this is the part that surprised me. Compare gemini-3.5-flash ($1.50 / $9.00) with claude-sonnet-5 ($2.00 / $10.00). On that RAG workload — 8,000 in, 700 out, per 1,000 requests — Flash costs $18.30 and Sonnet costs $23.00. You are one quarter away from a model in a completely different class, while the word "flash" in the name suggests you are saving a fortune.

Then add the multiplier nobody puts in the spreadsheet: retries. If the cheap model fails one call in five and you re-run those on the expensive one, you pay for the cheap attempt and the expensive one. At a 20% failure rate that Flash job is really $18.30 + 20% × $23.00 = $22.90 — the discount is gone, and you also shipped worse latency.

Prices below list change the same arithmetic without changing its shape. I work on altrouter.ai, which bills the same vendor models under list — Claude Sonnet 5 at $8.50 per 1M output against the official $10.00, Grok 4.3 at $2.12 against $2.50 — so the crossover points move, but you still have to know your own ratio to find them. It also does not host embedding models, and no discount will rescue a model that keeps failing your task.

The actual decision order

Measure prompt_tokens / completion_tokens on a few hundred real requests. This takes ten minutes and everything else depends on it.
If input dominates (10:1 or more), rank candidates by input price only. Output price is noise; ignore the headline number everyone quotes.
If output dominates (below 2:1), rank by output price only — and specifically look for models with a low output-to-input multiple.
Price the next tier up on your own mix before assuming you can't afford it. If it lands under ~1.5× and it removes a retry, it is cheaper, not dearer.
Re-run this per endpoint, not per app. Your classifier and your code generator are different workloads and deserve different models.

What this doesn't solve

This is arithmetic about price, not about quality. It will not tell you whether Grok 4.3 writes code you'd merge — only you can judge that, on your tasks. It ignores latency, rate limits, and prompt caching, which can each move the answer more than the price gap does. And the numbers are a snapshot from 12 Aug 2026; every one of them will be stale within a quarter, though the method won't be.

Pick the model for the task, not for the app. And before you argue about which one is cheaper, go print your two usage numbers — the argument is usually already settled by them.

Your LLM gateway takes a cut. Seventeen lines of Python tell you how big.

Andrey Altrouter — Fri, 14 Aug 2026 05:18:57 +0000

You wanted to try three models from three vendors, so you did the sensible thing: pointed one client at a gateway, put one key in the environment, and stopped thinking about it. A gateway here is a service that speaks the OpenAI API and forwards your requests to Anthropic, Google, OpenAI, xAI and the rest, so switching models is a string change instead of a new SDK.

What almost nobody checks after that: what the gateway charges for the exact same tokens the vendor would have sold you directly.

What you actually buy from a gateway

Three things, and they are worth very different amounts.

One key and one invoice. Real, and boring. It saves you three signups and three billing pages.

Routing and fallback. If a provider returns 529 or a region goes dark, the gateway retries elsewhere. Worth something on production traffic, nothing on a weekend project.

A price per token. This is the one that varies by an order of magnitude between services, and the one nobody puts in a comparison table — because there is no single "gateway price". There is a price per model, and a service can sit below vendor list on one model and far above it on another.

Vocabulary, in case it is new: models are billed per million tokens (a token is roughly ¾ of a word), input and output priced separately, with output typically 3–10× more expensive. The list price is what the vendor publishes on its own pricing page. That list price is your baseline — a gateway is either below it, at it, or taking a markup on top.

Seventeen lines that answer it

Many gateways publish per-model pricing in their models endpoint. If yours does, this reads it and prices out your token mix, not a benchmark's:

import json, urllib.request

GATEWAY = "https://api.altrouter.ai/v1/models/public"   # your gateway's models endpoint
LIST = {"claude-sonnet-5": (2.00, 10.00), "gpt-5.6": (2.50, 15.00)}  # vendor list $/1M in, out
MIX = (200, 40)  # your monthly millions of tokens: input, output

req = urllib.request.Request(GATEWAY, headers={"User-Agent": "price-check"})
models = json.load(urllib.request.urlopen(req))["data"]

for m in models:
    inp, out = m["pricing"].get("prompt_per_1m_usd"), m["pricing"].get("completion_per_1m_usd")
    if m["id"] not in LIST or inp is None:
        continue
    li, lo = LIST[m["id"]]
    mine = MIX[0] * inp + MIX[1] * out
    vendor = MIX[0] * li + MIX[1] * lo
    print(f"{m['id']:<18} gateway ${mine:>8.2f}  list ${vendor:>8.2f}  {mine / vendor - 1:+.1%}")

Run on 2026-08-14 against the endpoint in the snippet, it prints:

gpt-5.6            gateway $  935.09  list $ 1100.00  -15.0%
claude-sonnet-5    gateway $  677.91  list $  800.00  -15.3%

If your gateway does not publish prices in that endpoint — plenty don't — use the invoice method instead: take last month's charge, divide by the input and output tokens the gateway logged, and you have your real effective $/1M. That number is the only one that matters, and it is the one you compare to the vendor's page.

The obvious objection: the mix is made up. It is — replace MIX with your own two numbers before you believe any of the output. The ratio between input and output tokens is workload-specific, and since output costs several times more, a chat workload and a document-summarizing workload rank gateways differently.

The four things I'd check before switching

Price per model, on your mix. The script above. A gateway that is cheap on a small model can be expensive on the large one you actually run.
Whether base_url is genuinely the only change. Test streaming, tool calls, and the usage object in the response. Those are where compatibility usually breaks.
What happens on a provider outage. Ask for the retry and fallback behaviour explicitly, then unplug a model and watch.
Your exit. If leaving means rewriting anything beyond a URL and a key, that is not a gateway, that is a platform.

Disclosure, since the URL in the snippet is mine: I work on altrouter.ai, which is why I could hardcode an endpoint that publishes its prices. It resells vendor models below list — Claude Sonnet 5 at $1.69 / $8.50 per 1M against the official $2.00 / $10.00, Opus 4.5 at $3.74 / $18.75 against $5.00 / $25.00, discounts running 10–25% by model, prices as of August 2026. Point the script at whatever you use now; the arithmetic does not care.

What this does not tell you

Price, and only price. The script says nothing about latency, uptime, rate limits, or whether prompt caching is passed through at the vendor's discount — check that separately, it can outweigh a 15% price gap. And the honest gaps on my side: no embedding models, no data residency options, no SLA of our own beyond the upstream vendor's. If you are on one vendor and never plan to leave, buy direct — you save exactly the middleman's margin.

Run the seventeen lines on your own gateway this week. Whatever the number is, it is better to know it than to assume the convenience was free.

Your LLM bill has two levers. Your team only pulls one.

Andrey Altrouter — Thu, 13 Aug 2026 07:19:53 +0000

Every guide to cutting LLM costs is about the same thing: send fewer tokens. Trim the system prompt, cache the prefix, retrieve less, route easy requests to a small model. All of it works. And all of it is one half of the equation.

Your bill is:

bill = tokens × price_per_token

Two factors. Multiplication. Every cost-cutting article I have read attacks the left one and treats the right one as a law of physics. It is not — it is a number on a pricing page, and pricing pages are not the only place those models are sold.

The two levers, in the same units

A quick vocabulary check, because the whole argument lives in these words. A token is roughly ¾ of a word — models are billed per million of them. Input tokens are what you send (prompt, files, chat history), output tokens are what the model writes back. Output costs 3–10× more per token, which is why it usually dominates the bill. Prompt caching means the provider stores your unchanged prefix and re-reads it at a large discount, often around 10% of the input price — you pay a small premium the first time to write the cache, then read it cheaply.

Now, one workload. A team runs a coding agent, and last month it burned 200M input and 40M output tokens on Claude Sonnet 5. At list price ($2.00 in / $10.00 out per 1M):

200 × $2.00 + 40 × $10.00 = $400 + $400 = $800

Lever one — fewer tokens. Say 70% of that input is a stable prefix (the same repo files, the same instructions) and you make it cacheable. That 140M drops to roughly $0.20 per 1M instead of $2.00. Input becomes 140 × $0.20 + 60 × $2.00 = $148. Output does not cache — it is generated fresh every time. New total: $548, a 31% cut. That is a good month of engineering work.

Lever two — cheaper tokens. Same tokens, same model, price reduced 15%. $800 becomes $680. Zero engineering work.

Neither is the interesting part. The interesting part is that they multiply:

Configuration	Monthly bill
List price, no caching	$800.00
Caching only	$548.00
Cheaper price only	$680.00
Both	$465.80

Pulling both levers is a 42% cut. Pulling the one everybody writes about is 31%.

Why the second lever gets ignored

Because it does not feel like engineering. Caching is a diff you can review, benchmark, and put in a changelog. "We pay less per token" is a procurement decision, so it falls between the team that owns the code and the team that owns the invoice, and nobody optimizes it.

It is also the lever with no quality tradeoff. Every token-side optimization trades something: caching adds cache-invalidation bugs, retrieval trims context the model might have needed, routing to a smaller model risks a worse answer that costs more in retries. Changing the price per token changes exactly one thing — the price per token. The weights, the context window, and the output are identical.

Where the price is actually negotiable

Disclosure before the numbers: I work on altrouter.ai, a gateway that resells the same vendor models below their list price — Claude Sonnet 5 at $1.69 / $8.50 per 1M against the official $2.00 / $10.00, Opus 4.5 at $3.74 / $18.75 against $5.00 / $25.00, GPT-5.6 at $2.13 / $12.75 against $2.50 / $15.00. Discounts run 10–25% depending on the model, the API is OpenAI-compatible, so switching is a base_url change. Prices as of August 2026.

The honest gaps, so you hear them from me and not from the comments: no embedding models, no data residency options, no SLA of our own beyond what the upstream vendor delivers, and no invoicing for legal entities yet. If any of those is a requirement, this lever is closed for you and lever one is all you have.

What to do this week

Pull last month's spend and split it into input, output, and cached input. If you cannot produce that split, you cannot price anything — in/out prices differ by 3–10×, and cached input by another 10×.
Multiply your actual token counts by two or three candidate prices. Not benchmark tokens — yours. The ratio is workload-specific.
Do the caching work anyway. The levers multiply; skipping one to take the other is leaving the smaller half on the table.

The point is not that one lever beats the other. It is that you are running an optimization with two variables and holding one constant for no reason other than habit.

Your model benchmark measures everything except the bill

Andrey Altrouter — Wed, 12 Aug 2026 11:30:17 +0000

Every model benchmark I read ranks the same two things: how good the output was, and sometimes how many tokens it took. Almost none of them print the number you actually get charged for.

That gap matters, because the two rankings are not the same list. A model can win on quality, lose on tokens, and still be the cheapest way to finish your task — or the opposite. You cannot tell which without doing the multiplication yourself.

Here is the multiplication.

The formula

cost = (input_tokens / 1M) × price_in + (output_tokens / 1M) × price_out

That is the whole thing. Two numbers from the benchmark, two numbers from the vendor's pricing page. The reason it is worth writing down is that every term in it moves independently, and benchmarks only ever report the first two.

A worked example

Take one coding task. Say it consumes 40K input tokens and 12K output tokens — the shape of a real agent run with a few files in context and a couple of iterations. These numbers are an assumption for the example, not a measurement; substitute your own from your logs.

Vendor list prices per 1M tokens, as of August 2026:

Model	in $	out $	Cost of one task	100 tasks
Claude Opus 4.5	5.00	25.00	$0.500	$50.00
GPT-5.6	2.50	15.00	$0.280	$28.00
Claude Sonnet 5	2.00	10.00	$0.200	$20.00
Grok 4.5	2.00	6.00	$0.152	$15.20
Claude Haiku 4.5	1.00	5.00	$0.100	$10.00
Gemini 3 Flash	0.50	3.00	$0.056	$5.60

Nine times between the top and the bottom row. No leaderboard shows you that spread, because no leaderboard multiplies.

The trap in the cheap column

Now the part that makes the table dangerous if you read it naively.

Token counts are not a property of the task. They are a property of the model on that task. A weaker model that misunderstands the requirement, retries twice, re-reads the whole file and asks a clarifying question burns three or four times the tokens of a model that gets it in one pass.

Run the arithmetic again. Haiku 4.5 is five times cheaper per token than Sonnet 5. If it needs four times the tokens to converge, one task costs $0.40 instead of $0.10 — and Sonnet, at $0.20, is now twice as cheap as the "cheap" model.

This is why "just use a smaller model" is advice, not a strategy. The only way to know is to log input and output tokens per task, per model, and multiply. Both numbers, separately: output usually dominates the bill, and it is the one that explodes when a model starts thinking out loud.

Same reason time-to-finish deserves a column. A model that produces correct code after forty minutes of tool calls is not free just because each call is small.

What to actually do

Log tokens per task, split into input and output. If your framework does not expose them, most gateways do. Without the split you cannot price anything, because in/out prices differ by 3–10×.
Exclude cache reads or price them separately. Cached input is usually an order of magnitude cheaper; mixing it in makes your numbers look better than reality.
Price your top three candidates on your own tasks, not on a public benchmark's tasks. The token ratio is workload-specific — long-context refactoring and short algorithmic prompts produce completely different bills.
Recompute quarterly. Model prices move, and new models land every few weeks at a different point on the curve.

Do this once and the model choice stops being a debate about vibes. It becomes a number with a currency symbol on it.

Where the price itself is negotiable

One more term in that formula is worth attention, and it is the one everybody treats as fixed: price_in and price_out.

Vendor list price is not the only price those models sell at. I work on altrouter.ai, a gateway that resells the same models below the vendor's list — Claude Opus 4.5 at $3.74 / $18.75 per 1M against the official $5.00 / $25.00, Sonnet 5 at $1.69 / $8.50 against $2.00 / $10.00, and so on across 44 models on an OpenAI-compatible API. The discount is a standing price, shown next to the official one on every model page, not an intro offer.

Plug it into the same example: the 100-task run on Opus 4.5 costs $37.44 instead of $50.00. Same weights, same tokens, same code — only the third and fourth term of the formula changed.

I am obviously not a neutral party on that last paragraph, so treat it the way you should treat the rest of this post: as arithmetic you can check. The formula is public, the list prices are public, and your token counts are in your own logs.