DEV Community

modelmenu
modelmenu

Posted on

Why I stopped comparing LLMs by price per token

Claude Sonnet 5.5 and GPT-6.1 Sol have the same price sheet: $2 per million input tokens, $10 per million output, $0.10 for cached input. On paper they cost the same.

They don't. Artificial Analysis runs a suite of ten evals (its Intelligence Index) on each model and records the real bill, reasoning tokens included. Divide that by the number of tasks and you get cost per task. At high effort, Sonnet 5.5 comes out at $0.88 a task. GPT-6.1 Sol comes out at $0.32, and it scores higher on Humanity's Last Exam too, 51.4 vs 45.8.

Same price per token, almost 3x the cost per task. The token price tells you what a token costs. It doesn't tell you how many tokens the model burns before it answers, and with reasoning models that's most of the bill.

Here are a few other things I keep running into when I compare models this way.

Effort levels

The big labs' models now take an effort setting. The token price doesn't change with it. The token count does.

Claude Opus 5.5 per task: $0.55 at low, $1.34 at medium (the default), $1.82 at high, $3.46 at xhigh, $5.98 at max (full breakdown). Going from the default to max costs 4.5x.

Sometimes that buys a lot. Opus's Humanity's Last Exam score goes from 54.7 to 61.4. Sometimes it buys nothing. On Terminal-Bench 4.0, xhigh and max both score 59.6, so a coding agent running at max pays 73% extra for the same score.

Effort also flips the "cheap model vs expensive model" idea. Sonnet 5.5 at max is $5.46 a task. Opus 5.5 at xhigh is $3.46, even though Opus tokens cost twice as much. Which one's better depends on the job: Opus wins on Humanity's Last Exam (57.5 vs 55.0), Sonnet wins on Terminal-Bench (63.6 vs 59.6).

Cheap tokens don't save you either. Gemini 3.8 Flash lists at $0.75/$3.75, well under half of Gemini 4 Argon's $2/$10. At high effort, though, Flash costs $3.49 a task and Argon $1.62, and Argon scores higher on both benchmarks.

Caching

If you're building agents, you resend the same prefix on every call: system prompt, tool definitions, the conversation so far. On Opus 5.5, cached reads cost $0.20 per million instead of $4.

Anthropic charges extra to write the cache: $5 per million for a 5-minute entry, $8 for a 1-hour one. Quick math for an agent with a 40k-token prefix making 30 calls in an hour. Uncached, that's 1.2M input tokens, or $4.80. With a 1-hour cache you pay $0.32 to write it once and $0.23 for the 29 reads. About $0.55 in total, 88% less.

Cache prices can also drop between versions while the headline price stays put. Sonnet 5 and Sonnet 5.5 both list $2 input, but cached reads went from $0.20 to $0.10.

Promo prices

Gemini 4 Argon's $2/$10 is a promo that ends November 30. After that it's $4/$20, and its $1.62 per task roughly doubles to a little over $3. Gemini 3.6 Flash and 3.8 Flash double on December 31. Mistral Large 4 already went through this: its launch promo ended on October 23 and it jumped from $0.68/$2.09 to $1.36/$4.18.

Prices move down too. GPT-5.6 Luna got 80% cheaper on July 30. I've been logging these changes, and there have been 26 since June: 17 cuts and 9 increases.

Different hosts

Open-weight models add another spread. GPT OSS 120B is listed by 24 providers, from $0.03/$0.15 to $0.35/$0.75 per million. Hosts also differ in speed and limits, so the cheapest one isn't automatically the right one.

What I check now

Before I pick a model, I look at:

  • the benchmark that matches the job (Terminal-Bench for coding agents, Humanity's Last Exam for knowledge-heavy chat), not one blended score
  • cost per task at the effort I'll actually run
  • how much of my input will hit the cache
  • the price after any promo ends

Numbers in this post are as of October 27, 2026. Disclosure: I run Model Menu, a free site where I track these prices and per-task costs.

Top comments (1)

Collapse
 
iteracompute profile image
IteraCompute •

Two more items for the "same price sheet, different bill" list:

  1. Tokenizers. The same text doesn't produce the same token count across vendors. In The New Stack's DeepSeek V4.1 Flash vs Claude Haiku 5.5 test (10-23), one triage prompt counted 153,082 tokens on DeepSeek and 216,811 on Haiku, about 1.42x. So $/1M tokens isn't even $/character across vendors.
  2. Prompt-length tiers. That same prompt pushed Haiku past its 100K line, where Anthropic bills the whole request at 5x ($0.50 / $2.50 instead of $0.10 / $0.50 per 1M, checked 2026-10-30). The run cost $0.28 on Haiku vs $0.06 on DeepSeek. At Haiku's ≤100K rates the same tokens would have been about $0.056. Most of that "4x cheaper" gap came from the tier, not the model.

For a cost-per-task table, recording the max prompt size per task next to effort level would make tier effects visible.