Your model benchmark measures everything except the bill

Andrey Altrouter — Wed, 12 Aug 2026 11:30:17 +0000

Every model benchmark I read ranks the same two things: how good the output was, and sometimes how many tokens it took. Almost none of them print the number you actually get charged for.

That gap matters, because the two rankings are not the same list. A model can win on quality, lose on tokens, and still be the cheapest way to finish your task — or the opposite. You cannot tell which without doing the multiplication yourself.

Here is the multiplication.

The formula

cost = (input_tokens / 1M) × price_in + (output_tokens / 1M) × price_out

That is the whole thing. Two numbers from the benchmark, two numbers from the vendor's pricing page. The reason it is worth writing down is that every term in it moves independently, and benchmarks only ever report the first two.

A worked example

Take one coding task. Say it consumes 40K input tokens and 12K output tokens — the shape of a real agent run with a few files in context and a couple of iterations. These numbers are an assumption for the example, not a measurement; substitute your own from your logs.

Vendor list prices per 1M tokens, as of August 2026:

Model	in $	out $	Cost of one task	100 tasks
Claude Opus 4.5	5.00	25.00	$0.500	$50.00
GPT-5.6	2.50	15.00	$0.280	$28.00
Claude Sonnet 5	2.00	10.00	$0.200	$20.00
Grok 4.5	2.00	6.00	$0.152	$15.20
Claude Haiku 4.5	1.00	5.00	$0.100	$10.00
Gemini 3 Flash	0.50	3.00	$0.056	$5.60

Nine times between the top and the bottom row. No leaderboard shows you that spread, because no leaderboard multiplies.

The trap in the cheap column

Now the part that makes the table dangerous if you read it naively.

Token counts are not a property of the task. They are a property of the model on that task. A weaker model that misunderstands the requirement, retries twice, re-reads the whole file and asks a clarifying question burns three or four times the tokens of a model that gets it in one pass.

Run the arithmetic again. Haiku 4.5 is five times cheaper per token than Sonnet 5. If it needs four times the tokens to converge, one task costs $0.40 instead of $0.10 — and Sonnet, at $0.20, is now twice as cheap as the "cheap" model.

This is why "just use a smaller model" is advice, not a strategy. The only way to know is to log input and output tokens per task, per model, and multiply. Both numbers, separately: output usually dominates the bill, and it is the one that explodes when a model starts thinking out loud.

Same reason time-to-finish deserves a column. A model that produces correct code after forty minutes of tool calls is not free just because each call is small.

What to actually do

Log tokens per task, split into input and output. If your framework does not expose them, most gateways do. Without the split you cannot price anything, because in/out prices differ by 3–10×.
Exclude cache reads or price them separately. Cached input is usually an order of magnitude cheaper; mixing it in makes your numbers look better than reality.
Price your top three candidates on your own tasks, not on a public benchmark's tasks. The token ratio is workload-specific — long-context refactoring and short algorithmic prompts produce completely different bills.
Recompute quarterly. Model prices move, and new models land every few weeks at a different point on the curve.

Do this once and the model choice stops being a debate about vibes. It becomes a number with a currency symbol on it.

Where the price itself is negotiable

One more term in that formula is worth attention, and it is the one everybody treats as fixed: price_in and price_out.

Vendor list price is not the only price those models sell at. I work on altrouter.ai, a gateway that resells the same models below the vendor's list — Claude Opus 4.5 at $3.74 / $18.75 per 1M against the official $5.00 / $25.00, Sonnet 5 at $1.69 / $8.50 against $2.00 / $10.00, and so on across 44 models on an OpenAI-compatible API. The discount is a standing price, shown next to the official one on every model page, not an intro offer.

Plug it into the same example: the 100-task run on Opus 4.5 costs $37.44 instead of $50.00. Same weights, same tokens, same code — only the third and fourth term of the formula changed.