Efficiency effects cost
Cost per token gets talked about a lot when evaluating AI coding models — but that doesn't really tell us the full story.
A cheaper model that uses significantly more tokens to complete a task can still work out more expensive overall.
When we factor in that model performance varies significantly at different thinking levels, the situation gets even more complicated to understand.
To understand cost properly we have to consider the models token efficiency.
This is where benchmarks like DeepSWE v1.1 really shine.
What is DeepSWE
DeepSWE is a well-regarded third-party benchmark that measures AI coding agents on real, long-horizon software engineering tasks.
Crucially it reports performance alongside average cost per task.
The chart plots models on two axes — 'pass rate' vs 'average cost'.
The ideal zone is the top-right corner — the closer a model gets, the more performant and cost-efficient it is.
Note — Some models have been omitted in the above screenshot for chart readability.
Surprising results!
At the High thinking level (which many developers use day-to-day), Opus 4.8 wins against Sonnet 5 on both axes simultaneously.
| Model | Pass Rate | Avg Cost/Task |
|---|---|---|
| claude-opus-4.8 [high] | 52% | $4.28 |
| claude-sonnet-5 [high] | 48% | $7.43 |
This makes Opus 4.8 more performant AND cheaper when With efficiency considered. That's not what most people would expect.
Additionally, Sonnet 5 at the Max thinking level works out far more expensive than Opus 4.8 at any thinking level. Also unexpected.
A caveat worth noting
Not all benchmarks are independent, and results won't perfectly mirror every real-world use case. But the underlying point stands: models vary wildly in token efficiency, and that has very real effects on cost — even at small scales.
The takeaway
These charts give you a sharper lens to make more informed decisions.
For full and up-to-date results see the DeepSWE website.
👉 deepswe.datacurve.ai/blog/deepswe-v1-1
Top comments (0)