Most of us pick an LLM by looking at a pricing table: dollars per million input tokens, dollars per million output tokens. Then we pick the model that sounds smart enough and hope the bill is fine.
That number answers the wrong question. You don't pay for tokens. You pay for a task that came out right, plus everything it took to get there:
- the retries when the first answer failed your check,
- the reasoning tokens a "thinking" model writes and you never see,
- the answers that come back empty or malformed, which you pay for anyway.
So I built a small open benchmark that measures the number you actually pay:
cost per successful task = everything you paid for all attempts ÷ results that passed the check
How it works
- 16 tasks that look like production work: invoice field extraction, receipt images, strict JSON for a schema, text-to-SQL over three SQLite databases, intent classification, small bug fixes, tool calling, Q&A over a ~30K-token handbook sent as a cacheable prefix, contact-record normalisation, and regex from examples. Ten are "standard" and six are a harder tier.
- Every check is code, not an LLM judge. Extracted fields are compared after normalisation, SQL is run and its result set compared, bug fixes run against hidden unit tests, regexes are tested on hidden strings.
- Up to 3 attempts per item. When a check fails, the validator's reason goes back to the model as the next message, the way a real app would retry.
- Provider defaults. No temperature, no reasoning-effort setting, no fallbacks, because that's what most people run.
-
Every input is generated from a fixed seed by code in the repo (invented names, companies, databases), so none of it can be in a training set, and
generate.py --checkproves the data matches byte for byte. - Prices come from an open pricing dataset, pinned to a commit, so every cost in the report can be traced to an official price page.
First run: DeepSeek flash vs DeepSeek V4 Pro
I ran both models on every text task, both tiers: 720 graded answers per model. The image task was skipped because neither model takes images.
| Per 1M successful tasks (list price) | deepseek-flash | deepseek-v4-pro | V4 Pro ÷ flash |
|---|---|---|---|
| Standard tier (9 tasks) | $347 | $1,698 | 4.9x |
| Hard tier (6 tasks) | $596 | $5,478 | 9.2x |
| Success within 3 attempts | 100% | 100% |
Both models eventually got every item right. The flagship cost 4.9x as much per success on everyday tasks and 9.2x on the hard ones.
Per task, the gap ranged from 2.3x (intent classification, where answers are a single label) to 14x (hard regex):
| Hard-tier task | flash $/success | V4 Pro $/success | ratio |
|---|---|---|---|
| Invoice extraction (10 fields, VAT, deposits) | $0.00079 | $0.00667 | 8.4x |
| Strict JSON (nested schema, unit conversions) | $0.00057 | $0.00281 | 4.9x |
| Text-to-SQL (window functions, HAVING) | $0.00053 | $0.00505 | 9.5x |
| Tool calling (relative dates, decoys) | $0.00024 | $0.00117 | 4.9x |
| Long-document multi-hop QA | $0.00027 | $0.00150 | 5.6x |
| Regex (leap years, backreferences, lookaheads) | $0.00129 | $0.01800 | 14.0x |
Where the money goes
Hidden reasoning. V4 Pro's output price is 3.3x flash's. If that were the whole story, it would cost about 3–4x per task. It cost up to 9x, because it also writes far more output: 1.3x to 4.3x as many tokens per attempt on the hard tier, almost all of it reasoning. On the hardest regex tasks V4 Pro averaged 4,294 output tokens per attempt against flash's 1,003. The harder the task, the more it "thinks", and the wider the gap.
Paying for nothing. Three times, V4 Pro reasoned until it hit the 16,000-token output limit and returned an empty answer. Every one of those tokens was billed. The retry passed, and was billed again.
Cheap retries. Flash made three genuine mistakes in 720 answers: an empty reply on a bug fix, a phone-number field, and a regex that missed one case. Each was fixed by a single retry with the validator's feedback. A retry on flash costs less than a first attempt on V4 Pro.
Caching does the heavy lifting on long prompts. The long-document task sends a ~30K-token handbook with every question. About 98% of that input was served from DeepSeek's cache, so it cost about the same per answer as the short-prompt tasks.
The benchmark had bugs too
Running real models against a new benchmark finds problems in the benchmark. I fixed them before publishing:
- The first run surfaced ground-truth errors: an invoice layout where the "No." label could be read as part of the invoice number, and three product descriptions that fit two categories. I regenerated the data and re-ran.
- The tool-calling hard tier failed on both models at first. The cause was the runner, not the models: it decided how to grade a reply by checking whether the task id was exactly
tool_calling, so the hard variant was graded as plain text. It now reads aREPLYattribute on the task, and the self-test fails if a task that sends tools doesn't set it. - Three questions turned out ambiguous: the SQL databases contain two different people with the same name, and the questions didn't say whether to list that name once or twice; and one tool-calling request asked about "this coming Thursday" on a Thursday. Those were reworded. The expected answers didn't change.
Of the nine first-attempt failures across both models, three were genuine model errors, three were the output-limit cut-offs, and three were these ambiguous questions.
What this does not show
- That small models are always enough. This is one provider, synthetic tasks, and one run per model. On your workload the flagship may well earn its price.
- That these numbers transfer to other providers. That's the next thing worth measuring, which is why the runner supports OpenAI, Anthropic and Gemini too.
- What you'll actually be billed. Costs here are at DeepSeek's peak list price so the two models compare like for like. I ran off-peak, where DeepSeek bills half, and the console charged about 46% of the logged list cost. The whole experiment, smoke tests included, cost about $1.40.
The point isn't "always use the cheap model". It's that price per token tells you almost nothing about cost per result, and measuring the real number is cheap.
Run it on your own models
Python 3.11+, bring your own keys. Nothing is sent anywhere except to the providers you choose.
git clone https://github.com/aicostbudget/llm-cost-per-task-benchmark
cd llm-cost-per-task-benchmark
pip install -r requirements.txt -r requirements-runner.txt
python runner.py --estimate # what a run would cost, no API calls
export ANTHROPIC_API_KEY=... # only the providers you use
python runner.py --run --models claude-haiku-5-5 --limit 1 --budget 1 # smoke test
python runner.py --run --tier hard --models claude-haiku-5-5 --budget 3 --run-id haiku-hard
--budget is a hard cap: the runner stops before any attempt that could push spend past it, and the same --run-id resumes where it stopped. No key at all? The Gemini free tier works with --pace to stay under the rate limit.
Each run writes a report with the cost per success, the retry rate and the cached-input share. You can paste those into a model switch cost calculator to see what a switch would save per month (I built that too).
- Repo: https://github.com/aicostbudget/llm-cost-per-task-benchmark
- Full results and caveats: results/2026-10-29-deepseek.md
If you run it on GPT, Claude or Gemini models, I'd love to add your results. Post them in issue #1. And if you have a task where you're sure the flagship clearly wins, tell me in issue #2 and I'll build it.
Top comments (6)
I'm building Tokenade, and the table you describe ranks each model by its listed rate. The bill ranks each model by what a passing task costs. A model with a higher listed rate can still cost less if it finishes in fewer turns, and the cheap one can cost more if it fails and retries.
Yeah, agreed. In this run it didn't play out that way because both models passed almost everything on the first try, so turn count barely differed. On agent loops I'd expect the opposite to happen sometimes. A multi-turn task with several tool calls is the next thing I want to add. If Tokenade has real numbers on turns per task I'd be curious what you're seeing.🤝🏻
Cost per successful task is the right unit, and your breakdown of where the money goes is what makes it convincing. The hidden reasoning tokens and the three empty answers from hitting the output limit are exactly the costs a price-per-token table hides. The part I would keep in mind before generalizing is your own caveat: one provider, synthetic tasks, one run per model. Still, as a first filter for whether a flagship is worth it on routine work, this is a much better starting point than comparing list prices.
Thanks. And yes, that caveat is real. One provider and synthetic tasks is a narrow base. Next step is running GPT, Claude and Gemini pairs. If anyone has keys and wants to try, I'm collecting results here: github.com/aicostbudget/llm-cost-p...
Both models hit 100% within 3 attempts, but the 9.2x hard tier gap still hides the real cost, since V4 Pro hit the 16,000 token output limit 3 times and returned empty answers you still pay for.
Small clarification: those empty attempts are already in the 9.2x.
Every attempt gets billed into the total, including the ones that came back empty, and the total is divided by successes. So the 9.2x is the real cost. Without the wasted ones the gap would be a bit smaller.really!😄