Correction (Sep 2026): The lid question was later found to be mis-keyed. Every "harmless" mark in the tables below was the model asking which neck, which the corrected key counts as correct. Re-scored: cheap model 143 clean / 0 harmless / 2 risky, expensive model 144 / 0 / 1, fatal still 0 for both. Details in this post.
Eleven hours after the model-comparison post went up, a reader named Vinh Nguyen left a comment that unraveled its headline:
FATAL 0 against FATAL 0 is a tie in counts, but it is 29 items run once each. [...] Zero fatal out of 29 single trials is consistent with a true fatal rate up to about 10% at 95% confidence. [...] Running the whole exam five times per model and comparing fatal rates per item would tell you whether the tie is real or just the resolution of one pass, and it is the same order of cost as the run you already did.
He's pointing at a self-contradiction I published without noticing. My own post says, in the temperature section: "final scores should come from multiple runs, reported as a pass rate." And then the comparison itself — the whole "won by exactly one question" headline — sat on one pass per model.
No defense available. So I ran it.
The rerun
29 questions × 5 fresh runs × 2 models = 290 trials. Same harness, same unpinned temperature as the original — that variance is the thing being measured. Every raw answer sheet saved. All 290 completed this time (the original expensive-model run had dropped one question to a rate limit).
Haiku (cheap) Sonnet (3x price)
trials 145 145
FATAL 0 0
RISKY 1 1
MISSED 0 0
HARMLESS 4 5
clean 140/145 139/145
per run 29·28·27·28·28 28·28·28·28·27
Two things in that table killed my original framing.
The question that picked the winner was a coin flip
The original comparison came down to exactly one question — the lids. The customer writes: "10 boxes of the clear PET 300, and 5 bags of the white lids." The catalog has four lids to pick from:
CAP-300-W white lid, 300-neck (fits the PET-300 bottle)
CAP-300-BK black lid, 300-neck
CAP-500-W white lid, 500-neck (fits the PET-500 bottle)
CAP-500-BK black lid, 500-neck
"White" narrows it to two — but which neck? The clue sits in the same sentence: the customer is ordering PET 300 bottles, and nobody pairs 300 bottles with 500 lids. The expensive model read that context and confirmed the 300 white lid — and I called that the win. The cheap model asked "which neck?" instead.
Across five fresh runs, the expensive model never did that again. Zero confirms in five passes — it asked "which lid?" every single time. The only model that produced the winning confirm even once was the cheap one, in one run of five.
So the one-question gap my title stood on disappears the moment the exam is rerun. It was never a real ability gap between the two models. It was a coin flip that happened to land the expensive model's way on the one day I ran the exam once — and I published that single flip as the result.
The two RISKY trials landed exactly where the flag said
Each model produced one RISKY trial in 145 — the grade for "confirmed something ambiguous; right this time, fatal next time."
- Haiku, run 3, on "the usual, 3 boxes": two items in the order history both fit, and it presented only one of them as a candidate instead of both
- Sonnet, run 5, on the ultra-abbreviated "250 5": five products start with 250, and it confirmed one anyway
Both questions carry rule_decidable: false — the flag a reader talked me into adding: "can the reference data alone pin the answer down to exactly one?" On decidable questions, the run-to-run wobble drifted safe — extra clarifying questions, nothing worse. The dangerous direction — confirming what cannot be decided — showed up only on the flagged questions, at about 0.7% of trials per model.
The flag didn't just audit my answer key. It predicted where run-to-run variance turns dangerous.
So does the original conclusion survive?
The conclusion of that post — it's a tie on fatal errors, so use the cheap model — survives. What gets replaced is the evidence it stands on.
The old evidence was one pass: I ran the exam once, fatal errors came out 0 to 0, and the whole visible difference was that one lid question. The lid question is out — it was luck. What takes its place: fatal errors stayed 0 to 0 through 145 trials per model, and RISKY came out tied too, 1 to 1. By the rule of three, 145 trials tighten the fatal-rate bound on this exam from about 10% (at 0/29) to about 2%. On questions the model hasn't seen, I still have only 29 kinds — so that ceiling stays near 10%. The tie is no longer one lucky afternoon — it is a measured result. And on a tie, take the cheap one: the conclusion stands.
What doesn't stand is the margin I published. Clean counts wobbled between 27 and 29 per run for the cheap model and 27–28 for the expensive one. On any single pass, either model can rank first by a question or two. A one-pass margin of one question is below this exam's resolution — which is what the comment said before I spent a cent measuring it.
Takeaways
- A single pass ranks models with ±1–2 questions of noise. Don't publish a one-pass margin — mine didn't survive its own rerun
- "Zero fatal" means little until you say over how many trials — and of what. 0/29 caps this exam's fatal rate near 10%; five reruns (0/145) cap it near 2%, on this exam. New questions are a separate count. The reruns cost about as much as the first run
- Variance has grades too. On decidable questions it drifted safe (over-asking). The dangerous drift — confirming the undecidable — appeared only on questions pre-flagged as rule-undecidable
- The exam caught its author again. Not in the answer key this time, and not in the grader — in how confidently I read a single run
P.S. All 290 raw answer sheets, the 5× driver, and the aggregator are in the repo → github.com/ramses203/llm-test-harness
P.P.S. If you'd rather get new experiments by email: ramses203.substack.com/subscribe
Top comments (6)
The five reruns are the part I would keep. A single clean pass is useful for catching silly failures, but it cannot tell you whether the question is stable. Saving every raw answer sheet was the right move, because the argument moved from a model preference to a repeatable harness.
The detail I keep coming back to: the winning behavior didn't just fail to hold up - it reproduced zero times in five runs. The result you published wasn't the model's typical behavior, it was its rarest. That's a stronger warning than "margins are noisy": a single pass doesn't just blur the ranking, it can crown the outlier.
Your post arrived at an uncomfortable moment for us - we're mid-run on a paired benchmark comparison (same 248 questions, two ingest formats, one pass each), and your resolution argument just rewrote our reporting rules before the results landed: the headline gets only the overall paired difference (large enough to survive ±1–2 noise, with an exact McNemar), and the small per-category cells - one has 13 questions - get demoted to "direction, not ranking." Cheaper to adopt your lesson than to re-learn it publicly.
The finding I'd frame is the rule_decidable one. Variance didn't spread evenly - the dangerous kind concentrated exactly on the questions your flag had marked as undecidable from the data. That suggests run-to-run wobble isn't noise on top of ability; it's a probe that finds the places where the task itself is ill-posed. Which raises a checkable question: does the 0.7% RISKY rate survive at temperature 0? If it disappears, the danger is sampling dice and pinning temperature fixes it. If it persists, it's the model's decision boundary sitting on top of an undecidable question - and no amount of rerunning fixes that, only the flag does. That single experiment would tell you whether your flag predicts where dice hurt or something deeper: where certainty is unearned.
"The exam caught its author again - in how confidently I read a single run" belongs on the wall next to Vinh's original comment. Series keeps getting better.
The
0/145to roughly 2% bound assumes 145 independent Bernoulli trials, but these are five reruns of the same 29 questions. If question difficulty drives failures, those repeats are clustered, so they reduce run-to-run uncertainty without giving 145 independent draws from the task distribution. I would report fixed-exam instability over the reruns separately from item-level generalization over 29 question clusters, perhaps with a cluster bootstrap, or the second claim can look five times tighter than the corpus supports.You're right — the post claims more than the experiment showed.
What I actually did was run the same 29 questions five times each. That proves one thing: on these 29 questions, the model stayed accident-free through all five passes. This exam is stable when rerun.
But it proves nothing new about questions the model has never seen. To judge that, what matters is how many different kinds of question I tested — and that number is still 29. Running the same 29 five times doesn't add a single new kind. So on new questions, my confidence is exactly what it was after one pass.
I'll fix the post: the 2% figure gets labeled "on this exam," and I won't claim more about new questions than 29 questions can support. The only way to learn about new questions is to write new questions — which is what the paraphrase batch you suggested is for. Thank you for catching it.
The 0/145 bound is doing less work than the 0/29 one. Five reruns of the same 29 items are correlated draws, not fresh trials, so what tightens is the per-item variance, not the exam's fatal rate. An item that fails on a real rule gap fails every pass; the rule of three assumes each trial can independently surprise you. You say as much when you keep the ceiling near 10% for
unseen questions, then put the 2% in the takeaways anyway.
Fair pressure on the 2%. Rather than re-litigate old runs, the clean test is new questions — a 50-paraphrase set of the lid question is next in the queue, and it measures exactly what reruns can't.