I paid 9x more for the “smarter” LLM. It got exactly the same answers right.

AICostbudget.com — Thu, 29 Oct 2026 15:53:31 +0000

Most of us pick an LLM by looking at a pricing table: dollars per million input tokens, dollars per million output tokens. Then we pick the model that sounds smart enough and hope the bill is fine.

That number answers the wrong question. You don't pay for tokens. You pay for a task that came out right, plus everything it took to get there:

the retries when the first answer failed your check,
the reasoning tokens a "thinking" model writes and you never see,
the answers that come back empty or malformed, which you pay for anyway.

So I built a small open benchmark that measures the number you actually pay:

cost per successful task = everything you paid for all attempts ÷ results that passed the check

How it works

16 tasks that look like production work: invoice field extraction, receipt images, strict JSON for a schema, text-to-SQL over three SQLite databases, intent classification, small bug fixes, tool calling, Q&A over a ~30K-token handbook sent as a cacheable prefix, contact-record normalisation, and regex from examples. Ten are "standard" and six are a harder tier.
Every check is code, not an LLM judge. Extracted fields are compared after normalisation, SQL is run and its result set compared, bug fixes run against hidden unit tests, regexes are tested on hidden strings.
Up to 3 attempts per item. When a check fails, the validator's reason goes back to the model as the next message, the way a real app would retry.
Provider defaults. No temperature, no reasoning-effort setting, no fallbacks, because that's what most people run.
Every input is generated from a fixed seed by code in the repo (invented names, companies, databases), so none of it can be in a training set, and generate.py --check proves the data matches byte for byte.
Prices come from an open pricing dataset, pinned to a commit, so every cost in the report can be traced to an official price page.

First run: DeepSeek flash vs DeepSeek V4 Pro

I ran both models on every text task, both tiers: 720 graded answers per model. The image task was skipped because neither model takes images.

Per 1M successful tasks (list price)	deepseek-flash	deepseek-v4-pro	V4 Pro ÷ flash
Standard tier (9 tasks)	$347	$1,698	4.9x
Hard tier (6 tasks)	$596	$5,478	9.2x
Success within 3 attempts	100%	100%

Both models eventually got every item right. The flagship cost 4.9x as much per success on everyday tasks and 9.2x on the hard ones.

Per task, the gap ranged from 2.3x (intent classification, where answers are a single label) to 14x (hard regex):

Hard-tier task	flash $/success	V4 Pro $/success	ratio
Invoice extraction (10 fields, VAT, deposits)	$0.00079	$0.00667	8.4x
Strict JSON (nested schema, unit conversions)	$0.00057	$0.00281	4.9x
Text-to-SQL (window functions, HAVING)	$0.00053	$0.00505	9.5x
Tool calling (relative dates, decoys)	$0.00024	$0.00117	4.9x
Long-document multi-hop QA	$0.00027	$0.00150	5.6x
Regex (leap years, backreferences, lookaheads)	$0.00129	$0.01800	14.0x

Where the money goes

Hidden reasoning. V4 Pro's output price is 3.3x flash's. If that were the whole story, it would cost about 3–4x per task. It cost up to 9x, because it also writes far more output: 1.3x to 4.3x as many tokens per attempt on the hard tier, almost all of it reasoning. On the hardest regex tasks V4 Pro averaged 4,294 output tokens per attempt against flash's 1,003. The harder the task, the more it "thinks", and the wider the gap.

Paying for nothing. Three times, V4 Pro reasoned until it hit the 16,000-token output limit and returned an empty answer. Every one of those tokens was billed. The retry passed, and was billed again.

Cheap retries. Flash made three genuine mistakes in 720 answers: an empty reply on a bug fix, a phone-number field, and a regex that missed one case. Each was fixed by a single retry with the validator's feedback. A retry on flash costs less than a first attempt on V4 Pro.

Caching does the heavy lifting on long prompts. The long-document task sends a ~30K-token handbook with every question. About 98% of that input was served from DeepSeek's cache, so it cost about the same per answer as the short-prompt tasks.

The benchmark had bugs too

Running real models against a new benchmark finds problems in the benchmark. I fixed them before publishing:

The first run surfaced ground-truth errors: an invoice layout where the "No." label could be read as part of the invoice number, and three product descriptions that fit two categories. I regenerated the data and re-ran.
The tool-calling hard tier failed on both models at first. The cause was the runner, not the models: it decided how to grade a reply by checking whether the task id was exactly tool_calling, so the hard variant was graded as plain text. It now reads a REPLY attribute on the task, and the self-test fails if a task that sends tools doesn't set it.
Three questions turned out ambiguous: the SQL databases contain two different people with the same name, and the questions didn't say whether to list that name once or twice; and one tool-calling request asked about "this coming Thursday" on a Thursday. Those were reworded. The expected answers didn't change.

Of the nine first-attempt failures across both models, three were genuine model errors, three were the output-limit cut-offs, and three were these ambiguous questions.

What this does not show

That small models are always enough. This is one provider, synthetic tasks, and one run per model. On your workload the flagship may well earn its price.
That these numbers transfer to other providers. That's the next thing worth measuring, which is why the runner supports OpenAI, Anthropic and Gemini too.
What you'll actually be billed. Costs here are at DeepSeek's peak list price so the two models compare like for like. I ran off-peak, where DeepSeek bills half, and the console charged about 46% of the logged list cost. The whole experiment, smoke tests included, cost about $1.40.

The point isn't "always use the cheap model". It's that price per token tells you almost nothing about cost per result, and measuring the real number is cheap.

Run it on your own models

Python 3.11+, bring your own keys. Nothing is sent anywhere except to the providers you choose.

git clone https://github.com/aicostbudget/llm-cost-per-task-benchmark
cd llm-cost-per-task-benchmark
pip install -r requirements.txt -r requirements-runner.txt

python runner.py --estimate                      # what a run would cost, no API calls
export ANTHROPIC_API_KEY=...                     # only the providers you use
python runner.py --run --models claude-haiku-5-5 --limit 1 --budget 1   # smoke test
python runner.py --run --tier hard --models claude-haiku-5-5 --budget 3 --run-id haiku-hard

--budget is a hard cap: the runner stops before any attempt that could push spend past it, and the same --run-id resumes where it stopped. No key at all? The Gemini free tier works with --pace to stay under the rate limit.

Each run writes a report with the cost per success, the retry rate and the cached-input share. You can paste those into a model switch cost calculator to see what a switch would save per month (I built that too).

Repo: https://github.com/aicostbudget/llm-cost-per-task-benchmark
Full results and caveats: results/2026-10-29-deepseek.md

If you run it on GPT, Claude or Gemini models, I'd love to add your results. Post them in issue #1. And if you have a task where you're sure the flagship clearly wins, tell me in issue #2 and I'll build it.