The Model That Costs 3x More Won by Exactly One Question

I gave the same 29-question order-reading exam to two models.

A cheap one (Haiku 4.5) and one that costs about three times as much (Sonnet 5).

The result

Cheap model       28 of 29 questions clean
Expensive model   all 28 executed questions clean
Fatal errors      zero for both

(One of the expensive model's questions never ran — rate limit.)

The 3x model really was better. By exactly one question.

That one question

10 boxes of the clear PET 300, and 5 bags of the white lids

The catalog has two kinds of lids: 300-neck and 500-neck.

"White lids" alone doesn't tell you which. But the same sentence starts with "PET 300." Nobody orders 300-size bottles and 500-size lids in the same breath.

The expensive model read that context and confirmed the 300 lid.

The cheap model asked back: "diameter unspecified, needs confirmation."

In the first post of this series, "needs confirmation" was the correct answer for the clear tape — so why is confirming correct here? The difference is the clue. The tape sentence contained nothing to decide the width. This sentence has "PET 300" on the same line. If the message itself contains the information to decide, confirming is correct. If it doesn't, asking is correct.

And look at how the cheap model was wrong. It didn't ship the wrong lid. It asked. No accident happens. The user just gets mildly annoyed if every lid order comes back as a question. That's the size of the gap.

So which did I pick?

The cheap one.

The criterion is severity, not score (earlier post). Both models at FATAL 0 is a tie on the grade that matters. On a tie, take the cheap one.

If it had gone the other way, there'd be nothing to think about — pay up. Say the scoreboard had looked like this:

Cheap model       28/29  FATAL 1   ← higher score, unusable
Expensive model   27/29  FATAL 0

Pick by "how many did it get right" and you pick the wrong model.

Bonus trap — same exam, different scores

One thing kept nagging me.

I gave the same question to the same model again, and the result changed. Wrong once, right on the retry, then right three more times. One miss in five runs.

The cause is temperature — the knob that controls randomness in model output. Pin it to 0 and the same input gives the same answer. The problem: the CLI tool I was using had no such option.

Two lessons:

• If you can't pin temperature, a single run's score is not reproducible. Final scores should come from multiple runs, reported as a pass rate
• Production calls run at temperature 0. Especially for work where the same order sheet must produce the same answer every day

Takeaways

1. Compare by severity, not score
2. FATAL 0 vs FATAL 0 is a tie. Take the cheap one
3. Pin the temperature or don't trust the number

Before you pay for the expensive model, count what the cheap one actually gets wrong. In our exam the entire gap was one question — and even that one was guilty of nothing but asking.

---

P.S. Next up: the exam was passed — and that's exactly why it can't ship yet.

All 29 questions and the grader are public → github.com/ramses203/llm-test-harness