DEV Community

Cover image for The cheapest model on my plan loses every benchmark. It still beats models charging 14x more.
Michael Amachree
Michael Amachree Subscriber

Posted on

The cheapest model on my plan loses every benchmark. It still beats models charging 14x more.

The cheapest model I can run on my plan costs $0.14 per million tokens. The one I actually reach for costs $0.44. I put them side by side expecting the cheap one to be close.

It lost all four benchmarks. Coding, reasoning, math, real-world bug-fixing — four for four.

I'm still recommending it. Not because it competes with the good models. Because of what it does to the models in the middle.

The trade, stated plainly

MiMo-V2.5 Deepseek v4 flash
Input / 1M $0.14 $0.44
Output / 1M $0.28 $1.32
Requests per 5h window 30,100 7,600
Coding 56.8 69.1
Reasoning 38.0 51.8
Math 85.0 86.0
Bug-fixing (SWE-Bench) 43.1 49.9

Two numbers matter more than the rest. MiMo gives you roughly four times as many requests before your window empties, at a third of the input price. And it gives up twelve points of coding ability to do it.

That's the whole decision. Not "which model is better" — which resource you're actually short on.

Where my instinct was wrong

I went in believing the cheap models had quietly caught up. The number I expected to find was a two- or three-point gap.

Math cooperated: 85.0 against 86.0. One point. That's noise, and it's a real result — for arithmetic-heavy work the cheap model is simply fine.

Reasoning did not cooperate. 38.0 against 51.8 is a 27% drop, the widest gap of the four. On paper that reads like a rounding difference. In practice it's the difference between a model that follows a four-step refactor and one that loses the thread at step two and confidently ships step three.

So "close enough" was wrong, and it was wrong in the direction that costs you the most: the reasoning gap is exactly the gap you feel when you hand a model something ambiguous.

The comparison I should have run first

The interesting fight was never cheap-versus-best. Look at what sits between them.

Model Coding Input / 1M Requests / 5h
MiMo-V2.5 56.8 $0.14 30,100
Qwen3.7-Plus 55.9 $1.20 4,300
GLM-5.1 55.8 $1.40 880
Qwen3.6 Plus 54.5 $2.00 3,300

The $0.14 model beats all three.

Qwen3.6 Plus charges fourteen times more per input token and codes a point and a half worse. GLM-5.1 charges ten times more, scores a point lower, and burns your quota thirty-four times faster — 880 requests against 30,100.

That's the finding. The floor didn't rise to meet the ceiling. It rose to swallow the middle. Every model priced between one and two dollars is now paying a premium for benchmark scores it doesn't actually have, and the only reason that isn't obvious is that nobody puts price, benchmark, and quota burn in the same row.

Which is the entire reason I built the thing.

Why "cheapest" needs an asterisk

MiMo-V2.5 isn't literally the cheapest model on my plan. There's one below it: Muse spark 1.2 contributor, at $0.10/$0.20, with 45,300 requests per window and a coding score of 72.2 — better than DeepSeek, at a quarter of the price.

That combination doesn't happen by accident. The word contributor is sitting right there in the model name. I don't have documentation explaining the arrangement, so I'm not going to assert one. I'll say this instead: when one model is cheaper, faster-burning, and better than everything around it, the price is buying something other than compute. Treat it as an outlier until you know what.

Set it aside and MiMo-V2.5 is the floor. Hy3 ties it on input price at $0.14 and actually out-codes it slightly at 58.8 — but charges twice as much on output and gives you 4,300 requests instead of 30,100.

What I got wrong building this

The first version of my burn scoring inferred quota cost from price. Cheaper token, slower burn. It seemed obviously true.

It was wrong, and it was wrong in an embarrassing direction: it ranked expensive models as quota-hungry even when the provider gave them generous request allowances, and it flattered cheap models that burn through a window fast. I was measuring the thing I could see instead of the thing I cared about.

The fix was to stop inferring and start reading — the published request counts per window, straight from the docs. That's when the table started disagreeing with my intuitions, which is when it started being useful. The MiMo-versus-middle finding above only exists because burn stopped being a function of price.

There's still a bug I haven't fixed: at the low end, a couple of models render a burn band that disagrees with their own request count. It's on my list. I'd rather say that here than have you find it.

What this doesn't tell you

Benchmarks are a proxy and a coarse one. A twelve-point coding gap is real, but I can't tell you it maps cleanly onto your codebase, your language, or your prompting habits. Some of these scores come from different evaluation suites and get normalized to sit in the same column — that normalization is a judgment call, and it's mine.

Nothing here is hand-curated. Scores, tags, and burn bands are all computed from three upstream sources, so when they update, my conclusions change without my permission. That's the design. It also means I'm occasionally wrong in public.

And I have no historical data. I can tell you what the board looks like right now. I can't tell you what moved.

The part I actually want to argue about

I think most people pick a model once, by reputation, and never revisit it — and then spend the rest of the month rationing requests they didn't need to ration.

The table is free, needs no account, and links every number back to its source: zp.svelte-apps.me · code at github.com/Michael-Obele/zenpick · open an issue if a number looks wrong to you, because sometimes it is.

So: when you're choosing a model, are you short on capability or short on requests? I've been answering that question wrong for months — optimizing for the smartest model I could afford, then quietly avoiding using it. Curious whether anyone else has caught themselves doing the same thing.

Top comments (1)

Collapse
 
unitbuilds profile image
UnitBuilds

Very true. I've been trying out Qoder lately, a switch from my usual, which is using the foundry via API keys, which gets expensive. So when I tried Qoder, logically, I first took advantage of the 98% discount on Qwen 3.8 Max, then the free 1k Ultimate calls per day, then when those 2 stopped, I tried the Lite model and my mind was blown... With a caveat. Lite, consistently performed as well as both of those for 99% of the tasks, except it's completely free. That being said, there's a catch, that was with the Generate Wikis tool running, which kept the context clean and the agent on track, but it cost a few pennies to run, that eventually ate up my credit pool. Since then, I've been running Lite on it's own and with over 6m LOC of highly complex rust code written, tested, verified, etc. It's impressive for free... I did figure out a workaround though, if generate wikis is off the board, just create a scheduled task for a Lite agent to keep the .qoder folder up to date, problem solved.

Note, it's got around (by my calculations) 10m token budget on a free account, once you switch to a paid plan ($20 a month), it's genuinely uncapped and if you have the system for it (mine barely holds on at 10 agents), you could potentially run 20+ agents concurrently. With those numbers, you can beat MiMo in terms of cost per month.

Qoder is the brainchild of Alibaba, so its 'lite' model is more likely than not a highly trained Qwen Coder model, which really does not disappoint, it's actually insulting to call it Lite with the quality of work and speed.