Correction (Sep 2026): The lid question that decided this comparison was later found to be mis-keyed. The customer never stated the neck size, so asking was the correct answer, not confirming. Under the corrected key the cheap model's answer was right and the expensive model's was a guess. The re-scoring is in this post.
I gave the same 29-question order-reading exam to two models.
A cheap one (Haiku 4.5) and one that costs about three times as much (Sonnet 5).
The result
Cheap model 28 of 29 questions clean
Expensive model all 28 executed questions clean
Fatal errors zero for both
(One of the expensive model's questions never ran — rate limit.)
The 3x model really was better. By exactly one question.
That one question
10 boxes of the clear PET 300, and 5 bags of the white lids
The catalog has two kinds of lids: 300-neck and 500-neck.
"White lids" alone doesn't tell you which. But the same sentence starts with "PET 300." Nobody orders 300-size bottles and 500-size lids in the same breath.
The expensive model read that context and confirmed the 300 lid.
The cheap model asked back: "diameter unspecified, needs confirmation."
In the first post of this series, "needs confirmation" was the correct answer for the clear tape — so why is confirming correct here? The difference is the clue. The tape sentence contained nothing to decide the width. This sentence has "PET 300" on the same line. If the message itself contains the information to decide, confirming is correct. If it doesn't, asking is correct.
And look at how the cheap model was wrong. It didn't ship the wrong lid. It asked. No accident happens. The user just gets mildly annoyed if every lid order comes back as a question. That's the size of the gap.
So which did I pick?
The cheap one.
The criterion is severity, not score (earlier post). Both models at FATAL 0 is a tie on the grade that matters. On a tie, take the cheap one.
If it had gone the other way, there'd be nothing to think about — pay up. Say the scoreboard had looked like this:
Cheap model 28/29 FATAL 1 ← higher score, unusable
Expensive model 27/29 FATAL 0
Pick by "how many did it get right" and you pick the wrong model.
Bonus trap — same exam, different scores
One thing kept nagging me.
I gave the same question to the same model again, and the result changed. Wrong once, right on the retry, then right three more times. One miss in five runs.
The cause is temperature — the knob that controls randomness in model output. Pin it to 0 and the same input gives the same answer. The problem: the CLI tool I was using had no such option.
Two lessons:
- If you can't pin temperature, a single run's score is not reproducible. Final scores should come from multiple runs, reported as a pass rate
- Production calls run at temperature 0. Especially for work where the same order sheet must produce the same answer every day
Takeaways
- Compare by severity, not score
- FATAL 0 vs FATAL 0 is a tie. Take the cheap one
- Pin the temperature or don't trust the number
Before you pay for the expensive model, count what the cheap one actually gets wrong. In our exam the entire gap was one question — and even that one was guilty of nothing but asking.
P.S. Next up: the exam was passed — and that's exactly why it can't ship yet.
All 29 questions and the grader are public → github.com/ramses203/llm-test-harness
Top comments (7)
The lid question turned out to be mis-keyed: the customer never gave the neck size, so asking was the correct answer. Re-scored, the cheap model was right and the expensive model guessed. Full account: dev.to/ramses203/i-rewrote-one-exa...
FATAL 0 against FATAL 0 is a tie in counts, but it is 29 items run once each, and your own retry section puts one of those items at one miss in five runs. Zero fatal out of 29 single trials is consistent with a true fatal rate up to about 10% at 95% confidence, and the expensive model only executed 28, so the two zeros are not even measured over the same number of trials. Running the whole exam five times per model and comparing fatal rates per item would tell you whether the tie is real or just the resolution of one pass, and it is the same order of cost as the run you already did.
You suggested five runs per model instead of one. I did: 29 questions × 5 runs × 2 models — 290 answers, same setup as the original.
Fatal errors: zero. All 290, both models. So the conclusion — tie on fatal, take the cheap one — holds, now on 290 trials instead of 29.
My title didn't hold. The whole margin in that post was one question, the lids, where the expensive model confirmed instead of asking. Across five fresh runs it never did that again — "which lid? please confirm," five out of five. The cheap model confirmed it once. That margin was run-to-run luck, and I published it as a stable fact.
Full question-by-question table in the next post, all 290 raw sheets going into the repo. Thank you for pushing me to check.
Zero in 145 trials per model puts each fatal rate under about 2% at 95% confidence, down from about 10% on the single pass, so the tie is now measured at that resolution rather than shown to be equality, and that is worth a line in the next post. The 290 also maps where the variance actually is: 28 items that never moved across five runs carry almost no further information, so another full exam is the most expensive way to buy resolution. The lids item is the only one that produced disagreement, and 1-of-5 against 0-of-5 is not a difference either, so I would spend the next batch on paraphrase variants of that one question. The fatal mode needs ambiguity and only that item supplies it.
That line went in — the write-up is up: dev.to/ramses203/the-question-that.... The tie is described exactly as you framed it: measured at 2% resolution, not proven equality.
And your budget point holds: the 28 items that never moved are bought and paid for; the lids item is the only one still carrying information. So the next batch goes there — paraphrase variants of that one question. Conveniently, the experiment I published yesterday showed the AI author is very good at stamping variants of a type it's given. "Ambiguity variants of the lid order" is exactly that kind of order.
Spot on analysis. That PET 300 example is such a great illustration of why grading by error severity matters way more than raw percentage points—asking a clarifying question isn't a fatal production bug.
It really highlights a practical framework for how people pick LLMs:
Use-Case First (Top-Down): Define the specific task and test starting from the cheapest model tier. For lightweight DevOps scripts or narrow configs, Haiku easily hits "zero fatal errors." Sonnet might be slightly sharper on edge-case context, but Haiku delivers 95% of what you need at a fraction of the cost and speed. Opus isn't even in the picture.
Model First (Bottom-Up): Use a high-reasoning frontier model to explore complex tasks and see what's actually possible—then optimize down to cheaper tiers by tightening the prompt or adding deterministic guardrails.
When you measure success by Cost-per-Successful-Run rather than raw capability, it's clear that right-sizing the model tier to the actual task scope is where the real value is.
Cost-per-Successful-Run is a better name than anything I used in the post. One addition: count how runs fail, not just whether. "Please confirm which item" costs one human question. Confirming the wrong item costs an undo, plus the risk that nobody notices. A plain success rate treats those two the same — a severity-weighted one doesn't, and that can change which model looks cheaper. And use-case-first is exactly the route this project took: cheapest model first, tested against the must-never-happen accident list, upgrade only if a fatal error shows up. It hasn't yet.