The cheaper model cost 18x more
Gemini 3.6 Flash is $0.75 per million input tokens and $3.75 out. Claude Haiku
4.5 is $1.00 and $5.00. On the rate card Gemini wins on both columns, and it
isn't close.
I asked them both the same question. Name the single biggest risk of SELECT *
in production code, one sentence.
Haiku: 31 output tokens, $0.000183.
Gemini: 892 output tokens, $0.003361.
Eighteen times the cost, from the cheaper model.
Where the other 860 tokens went
Gemini reasons before it answers and those tokens bill at the output rate,
whether or not you ever see them. The visible answer was one sentence. Same as
Haiku's. The rest was thinking.
I knew this happened. I did not know it was 18x, and I only found out because I
was looking at the billed token count instead of the words on my screen.
If you price a comparison by reading the rate card and eyeballing the response
length, you will be wrong by an order of magnitude for any model that reasons.
Which is most of them now.
Temperature 0 does not mean what I assumed
Before any of the statistics, I ran the plain two-model comparison twice. Same
command, same prompt, temperature 0, which is the setting you use when you want
the same answer twice.
Gemini gave me a different answer both times. Not reworded around the edges. A
different sentence, leading on memory exhaustion the first time and schema drift
the second.
The judge scored them 8-8 the first time. Second time, 8-7 to Haiku.
So if I'd run it once and stopped, which is what I would normally do, I'd have
written down a winner.
Was the expensive answer better, though
Eighteen times is fine if you're getting something for it. So I ran it properly:
three temperatures, two runs each, with Claude Opus 5 scoring every response out
of ten.
Metric: score | Test: welch_t_test | Correction: bonferroni | Threshold: p < 0.05
claude-haiku-4-5 (avg 7.333) vs gemini-3.6-flash (avg 7.000): p=0.6667, d=0.408 (small)
Nothing. p=0.6667 isn't in the same postcode as significance, and the effect
size is small and pointing at Haiku anyway.
Both answers were genuinely fine, for what it's worth. Haiku said SELECT *
breaks when someone adds a column and the application gets data it wasn't
written to handle. Gemini said the same thing with more clauses about memory and
network bandwidth. Read them side by side and you'd struggle to defend a
preference. The numbers agree.
The latency, same story
gemini-3.6-flash 1.0 27016 ± 23762 ms CV 0.880
claude-haiku-4-5 1.0 1138 ± 31 ms CV 0.027
That standard deviation is almost as big as the mean. One call came back in
seven seconds, another took nearly fifty, identical input both times.
Any single measurement of that number is close to worthless. A comparison that
ran once would have reported whichever draw it happened to get, and I'd have
believed it.
Haiku's coefficient of variation is 0.027. It does the same thing every time.
Things the tool told me that I didn't ask for
Claude Opus 5 rejects a temperature setting, so my judge ran at the provider
default rather than 0, which means its scores aren't reproducible run to run.
That got printed as a warning. Fair enough, and worth knowing before I quote a
score at anyone.
Gemini scored 7 in all three cells. Two consequences. scipy threw a precision
loss warning, because a variance test on constant data is unreliable. And
Gemini has no score confidence interval at all in the output. Haiku's reads
score [95% CI: 6.000, 8.000]; Gemini's line just doesn't have one, because a
bootstrap over 7, 7, 7 has nothing to resample.
A missing interval beats a made-up one.
What I'd actually do differently
Count the tokens the provider billed, not the ones on your screen. That gap is
where the money is.
And run it more than once. If a difference survives repetition it's probably
real. If it doesn't, you've just saved yourself from an opinion you'd have
defended in meetings for six months.
Here, the difference wasn't real. Which makes the decision easy: take the one
that costs 18 times less.
Caveat I should state plainly, because it cuts against my own headline. This is
one prompt. One judge. Significance on one question is significance on one
question and nothing more. A different prompt might well flip it, and if the
expensive model is better at something I didn't ask about, this tells you
nothing.
I ran all of this with cli-modelarium,
a CLI I wrote because I got tired of doing this in a spreadsheet. Runs locally,
stores nothing, twelve providers, your own keys.
pip install cli-modelarium
The whole experiment above cost me about nineteen cents, most of it the judge.
Two earlier attempts died on a free-tier rate limit before I got a clean run,
which is its own small lesson about measuring things.
Top comments (0)