DEV Community: John Green

I Rewrote One Exam Question Fifty Ways. First, the Answer Key Was Wrong.

John Green — Fri, 04 Sep 2026 14:00:34 +0000

Two readers said the same thing. Under the 3x-price comparison, Vinh said the next experiment should not be another rerun. It should take the lid question, the one that decided that comparison, and rewrite it in different forms. Under the rerun post, tonal said that five passes over the same 29 questions are not 145 fresh chances. Both are right. Rerunning the same exam adds runs, not kinds of question.

So I decided to rewrite the lid question fifty ways. Then, rereading the question, I found that its answer key was wrong. That story comes first.

The answer key was wrong

Here is the original.

페트 300 투명 10박스 뚜껑 흰거 5봉이요 (10 boxes of clear PET 300, 5 bags of the white lids)

The catalog has two white lids, a 300-neck and a 500-neck. The customer did not say which. The answer key said the correct answer was to confirm the 300-neck lid. The reason was that the same message orders 300 bottles, so the lids must be for those bottles.

But every other question on the same exam grades this kind of case the same way: the correct answer is to ask.

clear tape, 2 boxes      48mm or 60mm?        key: ask
PET 500, 8 boxes         clear or brown?      key: ask
250 5                    five candidates      key: ask
white lids, 5 bags       300 or 500?          key: confirm 300   ← the only exception

The rule is one line. Do not guess what the customer did not write. Ask. Only the lid question broke it. "There are 300 bottles right next to it, so obviously 300" looked so obvious that when I fixed the tape questions earlier, I never looked at this one again.

So I fixed it. Asking which neck is the correct answer. Confirming the 300 lid is RISKY: right this time, but a guess, so wrong next time. Confirming any other lid is FATAL: the wrong goods ship.

The scores of two earlier posts change

When one question's answer changes, every run that touched it gets re-scored. The other questions stay as they were.

There are four grades. Clean means no flaw. Harmless means asking about something that did not need asking. Risky means confirming by guess something that should have been asked, right this time but possibly wrong next time. Fatal means the wrong goods ship.

                                 clean  harmless  risky  fatal        clean  harmless  risky  fatal
                                 ─── before ─────────────────────  →  ─── after re-key ───────────────
3x-price post (one pass, 29 questions)
  cheap model                      28       1       0      0     →      29       0       0      0
  expensive model (28 executed)    28       0       0      0     →      27       0       1      0

Five reruns (145 trials per model)
  cheap model                     140       4       1      0     →     143       0       2      0
  expensive model                 139       5       1      0     →     144       0       1      0

Every changed cell comes from the one lid question. The runs where a model asked which neck moved from harmless to clean. The one run where the cheap model confirmed 300 moved from clean to risky. Fatal was 0 before and stays 0, because no run ever confirmed the wrong lid.

In the 3x-price post I wrote that the expensive model got exactly one more question right. That question was this one. The expensive model guessed 300, and the cheap model asked. Under the corrected key, the cheap model won. The conclusion does not change: both models have zero fatal errors, and on a tie in fatal errors you take the cheap one.

Fifty forms

With the key fixed, I rewrote the lid question fifty ways. The order and the trap stayed the same. Only the wording changed.

T1  word order      white lids 5 bags and clear PET 300 10 boxes
T2  endings         ...please send / ...placing an order / no ending at all
T3  abbreviation    PET300 clear 10bx lids white 5bags
T4  spacing         PET300clear10boxeslidswhite5bags
T5  typos           PTE 300 / lidss / whiet
T6  color words     white / white-color / the white ones / "the clear ones" for the bottle
T7  units           cartons / box / BOX / ten (as a word)
T8  spec notation   300ml / 300cc / PET 300 / PET bottle (300)
T9  sentence shape  two lines / "and also..." / numbered list
T10 noise           greetings / thanks / ORDER!!

Three things stayed fixed in every variant. The bottle always has "300" and "clear"; without them the bottle itself becomes ambiguous and it is a different question. The lid uses only a white-word and never a neck size; that is the trap. The quantities are 10 boxes and 5 bags, with unit words. I cut two variants that dropped the unit words, because the exam grades unit-less orders like "250 5" as ask, so dropping the unit changes the answer itself.

An LLM drafted the variants, and I checked each line against those three rules.

Grading follows the corrected key.

asks which neck (with candidates)   clean
confirms CAP-300-W                  RISKY  (a guess that was right this time)
confirms any other lid              FATAL

Each variant ran three times per model. Temperature stayed at the default.

Results

                                     cheap (Haiku 4.5)   expensive (Sonnet 5)
trials                               150                 150
asked which neck                     140  (93%)          148  (99%)
confirmed 300 by guess (RISKY)        10  ( 7%)            2  ( 1%)
confirmed another lid (FATAL)          0                   0
lost the lid line                      0                   0
bottle confirmed PET-300-CL x10      149 (asked once)    150

Three things show.

First, item matching did not break. Typos, collapsed spacing, other words for white, 300ml and 300cc: none of it moved either model. In 300 trials there were zero wrong lids and zero wrong quantities. The four noise groups (spacing, typos, color words, spec notation) produced zero guesses in 120 trials per model. I expected the accidents to come from noise. They did not.

Second, the guesses came from smooth sentences. The cheap model guessed ten times: four in the unit group, two in word order, two in sentence shape, one in endings, one in greetings. Each variant ran three times, and two variants made it guess in two of the three: "10 boxes of clear PET 300, and the lids in white, 5 bags please" and "10 boxes of clear PET 300 please. And also 5 bags of the white lids." Both are the most polite and complete sentences in the set. The more a message looked like a finished order, the more the cheap model finished it, filling in the neck size the customer never gave. The expensive model guessed twice, once on "10 boxes of clear PET 300, 5 bags of white lids, please send" and once on "PET300 clear 10 boxes lids white 5 bags".

Third, no variant made a model guess every time. Eight variants made the cheap model guess at least once, and none made it guess all three times. Guessing is not a property attached to a wording. It is a coin flipped on every run, and some wordings raise the odds. Had I run this once, I might have seen two guesses or five, depending on the day.

If I had not fixed the key, this experiment would have read the other way. The old key counted confirming 300 as correct, so the cheap model's ten guesses would count as ten correct answers, and the cheap model would win 10 to 2.

One objection I expect

"Why not just show the customer an order confirmation sheet?" Yes. Everything that needs asking is better gathered on one screen. But the guessed line has to be marked. If the sheet only says "white lids, 300-neck, 5 bags", a customer who never thought about neck sizes just taps confirm. It has to say "white lids, 5 bags: assumed 300-neck, tell us if 500" for the customer's eye to stop there. The sheet copies what the intake step produced. Intake has to mark the lid as "needs confirmation" for the sheet to carry a warning. If intake quietly settles on 300 without a mark, the sheet prints "300-neck, 5 bags" like any other line, and the customer passes it by.

Takeaways

The answer that looked most obvious on the key was the wrong one. I did not find it by thinking harder. Lined up next to the other 28 questions, this was the only one whose answer was a guess.
In this experiment the cheap model guessed on polite, finished sentences, not on messy ones. It filled in missing information where the sentence looked complete. This came from one question and two models, so I do not know whether it holds elsewhere.

P.S. All 300 raw answer sheets, the 50 variants, the run script, and the aggregation script are in the repo → github.com/ramses203/llm-test-harness. The corrected key is in the same commit, with the reason written in the question's note.

P.P.S. If you'd rather get new experiments by email: ramses203.substack.com/subscribe

The Textbook Says Grade Binary. I Grade in Four. Was I Wrong the Whole Time?

John Green — Mon, 31 Aug 2026 00:07:00 +0000

There's someone this series treats as its precedent: Hamel Husain, who teaches AI product evaluation and has drawn 4,500 students. The LLM-judge experiment was his method, tested with my own hands.

Reading his evals essay, I hit a paragraph that snagged. He recommends grading binary only — good or bad. Scores and fine-grained scales, he argues, cost effort and buy nothing.

But my grader has four grades: FATAL, RISKY, MISSED, HARMLESS. Every scorecard in this series has come out in those four columns, and the shipping rule itself — FATAL 0 ships — stands on them. Have I spent this entire series doing the thing my own precedent says not to do?

What Hamel is actually blocking

What he warns against is the 1-to-5 score scale. Here's why it fails.

Anyone who has agonized over "is this answer a 3 or a 4?" knows: the boundary between 3 and 4 cannot be defined in words. So the standard differs between graders, and even the same grader drifts between morning and evening. And when the average comes out to 3.7, nobody can say what decision that number makes. Ship, or don't ship? 3.7 doesn't answer.

Up to here, I agree completely.

So is my four-grade table a score scale?

There's a way to tell them apart. If it's a scale, you can average it. A 1-to-5 scale gives you a 3.7.

Our grades have no average. There is no "half a FATAL," no "somewhere between FATAL and RISKY." All we ever do is count — so many FATALs, so many HARMLESSes.

Here's what each grade actually names:

FATAL      a wrong confirmation flows into the next step — no human gets a chance to catch it
RISKY      wrong, but a human checkpoint still remains — there's a chance to catch it
MISSED     something that should have been caught wasn't
HARMLESS   over-asked for confirmation — a human is mildly annoyed, nothing more

Grading one question means asking "did this accident happen — yes or no?" So we're not reading a 4-point dial. We're asking four binary questions. Translated into Hamel's terms: grade pass/fail as a binary — but give the failure a name.

Why the name matters — three decisions the name made

What if I had lumped everything into good/bad with no names? Let's check against three decisions that actually happened.

The model choice. Every mistake the cheap model made was the same kind: asking "please confirm" on questions it could have answered outright. A question hurts nobody — but good/bad grading counts it as "bad" anyway. So the scoreboard would have read: cheap model, several bads; expensive model, none — and I would have paid 3x for the clean-looking one. With names, those asks were HARMLESS, and the grade that decides shipping — FATAL — was 0 for both. A tie where it matters; on a tie, take the cheap one. Without that name I'd have paid three times more for the same safety.

The retirement verdict. I had a tool that sorts 20,000 YouTube comments into "a real customer need" versus "just chatter," and I use its tally to decide what to build. On a 15-question exam it got 7 wrong. Good/bad grading leaves one number — 8 of 15, 53% — and whether 53% is good enough to keep, the number can't say. What decided it was the content of the wrong answers: three of them counted chatter as a need. To undo that mistake, someone would first have to notice it — and nobody re-reads 20,000 comments. A mistake that never gets noticed never gets undone. That's FATAL on the grade table, three times over. Our rule is that a single FATAL is enough to stop using a tool. This one had three. So it was retired — and the 53% played no part in that decision.

The replacement decision. The replacement candidate, an LLM, got 12 of 15 on the same exam — 80%. If percentages decided, "80 beats 53" would be the reason. It wasn't. The LLM also got 3 wrong; the difference is the kind — all three were mistakes a human still gets a chance to catch. FATAL 0 versus FATAL 3 made the swap.

The common thread across all three: the percentage couldn't answer "ship it? which one? fix what first?" — the name of the failure answered. Names set the repair order too: fix FATAL first, let HARMLESS wait. If all seven failures are just "bad," there's no way to know which of the seven to touch first.

Takeaways

Hamel's warning is right — don't build score scales. The boundaries can't be defined, and an average makes no decision
But if you collapse everything into binary, the content of the failure disappears. Ask pass/fail — then name the failure
One criterion is enough for the name: can a human undo this accident? If not, it's FATAL, and it gets fixed first

Reading my own precedent, I spent about three days uneasy, thinking I'd been wrong. Taken apart, we were saying the same thing from different angles. When you build something independently and arrive at the same spot, the odds that the spot is right go up. This one was that kind of relief.

P.S. Hamel's essay is "Your AI Product Needs Evals" at hamel.dev. There were five more places where it overlaps with this series — add a question every time a failure appears, suspect the grader first, 100% pass is not the goal, a domain expert owns the answer key, split the job into scenarios before writing questions. Hamel and I built our approaches without knowing of each other, and they overlap in this many places. Too many for coincidence. I take it as a sign that verifying with an exam is a road worth staying on.

I Gave a Regex and an LLM the Same Exam. Fatal 3 vs Fatal 0.

John Green — Thu, 27 Aug 2026 09:39:19 +0000

In the last post my comment classifier — a regex — sat a 15-question exam and produced three fatal errors: the kind a human can't undo. Under our rule, FATAL above zero means the tool can't be used. So what replaces it?

There's one candidate: let an LLM do the classifying. So I staged a head-to-head on the same exam.

The rules

Same 15 questions, same grader, same grade table
Regex side: the existing tool, untouched (keyword matching)
LLM side: Sonnet 5, given the classification definitions (what counts as a need; personal anecdotes and rhetorical questions don't) and asked to judge each comment

One fairness question came up. The LLM was given the right to answer "needs confirmation." The regex can't be given that right — the concept doesn't exist in it, which was exactly the defect the last post pointed out. So this isn't favoritism; it's the difference in ability between the two tools, as-is.

The result

              regex                          LLM
clean         8/15 (53%)                     12/15 (80%)
grades        FATAL 3 RISKY 5 MISSED 1 H 0   FATAL 0 RISKY 1 MISSED 1 H 1

Read by grade, not by score — the rule from the model-comparison post. FATAL 3 against FATAL 0. The regex is at a grade that can't ship; the LLM is at a grade that can. The match was decided there.

Three places where they split:

The LLM read context. The social-commentary comment — "…humanities majors don't pay, so nobody will replace them" — the regex filed as an errors-and-debugging need, because the Korean for "don't pay" shares two characters with its error keyword. The LLM correctly filtered it as commentary. Same for "Can't do this anymore, going to bed lol."

The LLM knows how to say it doesn't know. "Me too 😭 happens every time" is a reply that cannot be judged without its parent comment. The regex forced it into a category; the LLM answered "needs confirmation." It was given the right and actually used it. This tool can keep our principle — a wrong confirmation is worse than none.

The LLM needs no keyword dictionary. The regex's category dictionary had never registered "Cursor," the AI coding tool, so every comment about Cursor went unclassified. The LLM put them in the AI-tools category from context alone, no dictionary. The maintenance of forever adding keywords simply disappears.

Not a clean sweep. The LLM dropped one category (a comment about monthly payment never got the pricing/billing label) and confirmed one ambiguous comment it should have handed to a human — one RISKY. Eighty percent, not a hundred. But every one of its mistakes sat in a grade a human can undo.

"So why not just verify with an LLM?"

Read this far and the question writes itself. If the LLM is this good, why build exams and grade tables at all — why not just ask an LLM "is this right?"

Think of a scale. There is one way to know whether a scale is accurate: put a weight whose mass you already know on it. If a 1 kg calibration weight reads 1 kg, the scale can be trusted.

Here the scale is the tool that judges comments, and the calibration weight is the exam with the answers written in advance. This match had two scales — the regex scale and the LLM scale. Put the same comment on both and they read differently; the thing that settled which one was right was the calibration weight, the 15 questions with known answers. Without it, the story ends at "huh, they disagree."

"Just ask an LLM" is the same picture: the LLM being verified and the LLM doing the verifying are two scales weighing each other with no calibration weight in the room. When they disagree, you never find out who's right.

Turn it around and it's good news: with a calibration weight, one scale is enough. Two scales were only needed while choosing. Now that the match is over, what remains is the 15-question exam and the LLM classifier — that pair.

This week I watched "just let an LLM verify it" fail three times in person:

When the LLM judge and the code grader disagreed, the only reason I could rule on who was right was a 29-question exam with known answers
In the exam-author experiment, the author was Sonnet and the reviewer was Sonnet — 50 out of 50 passed. It was grading itself. Without an exam procedure you don't even notice that's a trap
Today's 80% and FATAL 0 — those numbers exist only because there was an exam with an answer key. Without it, all that remains is "it looks good." Which is the exact spot "I can't trust it" started from

And one practical reason: re-grading is free. Every time you change the classification prompt or swap the model, does a human re-verify everything? The exam has its answers stored — a new scorecard takes five minutes. The 15 questions written today became a lifetime exam this tool will retake every time it's touched.

Cost still counts

The regex is free and instant. The LLM takes tens of seconds per call. For small volumes — my own blog's comments — the LLM is fine. For a 20,000-comment batch, a regex as a first-pass filter with the LLM ruling only on what it flags is a workable combination. Choosing a tool is, in the end, a trade between grade (safety) and cost.

Takeaways

When you switch tools, put them on the same exam. Choose by fatal count, not by score
If a tool can say "I don't know," check whether it actually exercises that right
Even if an LLM does your verifying, you still need the exam — it's the calibration weight for the thing doing the weighing

P.S. The 15-question exam and both classifiers' scorecards are public → github.com/ramses203/llm-test-harness, comment_exam.py (--compare shows them side by side)

I Stole My Own Exam. It Failed the Tool Behind My Own Numbers.

John Green — Thu, 27 Aug 2026 09:37:34 +0000

In the porting guide I wrote that the exam is built to be stolen — follow five steps and it moves to any job.

So I tried being the other person. Following only what the guide says, start to finish.

Where to steal it to — my own tool, of all places

For the second job I picked YouTube comment classification: scraping 20,000 comments and sorting each one into "a need," "chatter," or "a signal someone would pay." Every number in the 20,000-comments post came out of this classifier.

Which makes this a double-edged experiment. It tests whether the exam ports — and at the same time it tests whether the tool that produced my own published numbers can pass an exam.

The twist comes first — the tool wasn't an AI

Before writing a single question, I opened the classifier's code to understand what I was about to test. The thing that sorted 20,000 comments was not an AI. It was a regex — word matching: "if the comment contains this keyword, it's this category."

The second line of the actual data file was already an accident.

My grad-school senior bet that nobody would bother replacing humanities majors because they don't pay. He was right.

Social commentary. Not a need, and certainly not about errors. The classifier had filed it as a need in the "errors & debugging" category — because the Korean phrase for "doesn't pay" contains the same two characters as the error keyword "doesn't work." With 10,000 likes, it sat near the top of the ranking.

The accident showed up before the exam even existed.

Then I followed the five steps exactly

Step 1 — write down the worst. These classifications feed decisions about what to build and what to sell. So the worst accident is "promoting chatter into a need and manufacturing fake demand." A product decision built on fake demand burns weeks.

Step 2 — the grade table. Four grades: fatal, risky, missed, harmless. In the guide I had written "only the first line, FATAL, is redefined per project; the other three read the same everywhere." Porting it, one line didn't hold. In the order domain, MISSED (failing to catch something) was a mild grade — miss an order and the customer calls, so a human finds out. But a comment that gets dropped is never looked at again by anyone. The same MISSED is effectively fatal here.

Filling in the table also exposed something about the regex tool itself: it has no way to say "I'm not sure, a human should look." Every comment gets forced into a category. Under our first principle — a wrong confirmation is worse than no confirmation — that's a dangerous property before the exam has even started.

Step 3 — plant the traps. I rewrote the guide's four trap types (confusable pairs, plausible non-targets, mid-message reversals, after-learning) into comment form. While planting them, a fifth type showed up that isn't on the list: accidental keyword hits — chatter like "Can't do this anymore, going to bed lol" classified as a need because "can't" matches an error keyword. It only exists for tools that match characters, so the guide never had it.

Step 4 — the answer key. Fifteen exam comments, each with an expected answer and the flag "can the data alone decide this?" A reply like "Me too 😭" with no parent comment is undecidable — its correct answer is "needs confirmation."

Step 5 — grading. Another place the guide didn't hold. The order exam's grader is bound to order-specific fields — product codes, quantities — and not one line of it could be reused. What ported was the frame of thinking, the grade table and the principles; the 40-line grader was written from scratch.

The scorecard

my comment classifier (regex):  15 cases · 8 clean (53%)
🔴 FATAL 3   🟠 RISKY 5   🟡 MISSED 1   🟢 HARMLESS 0

All three fatals were accidental keyword hits. Social commentary, chatter, and a personal anecdote — each promoted to a need.

Our shipping rule has been one line from the start: FATAL 0 ships, anything else doesn't. This tool is at FATAL 3. And I had already published the numbers it produced. The core of that post — the "I can't trust it" comments with 598 likes — is quoted verbatim, so it stands. But the percentages are only as trustworthy as this regex, and I'm writing that down honestly.

So — does it port?

It ports. It took one hour. But my body learned that the manual has four defects.

① missing step        "understand the thing you're testing" isn't there — in
                      practice it ate half the hour
② grade table oversold "the other three lines are the same everywhere" — MISSED
                      turned fatal on this project. Re-argue the table every port
③ grader is domain-bound  what ports is the frame, not the code. Rewrite the grader
④ traps incomplete    four types are a start. Each tool type (regex, LLM) has
                      traps of its own

That defect list is the real harvest of this experiment. The next person who steals this exam, knowing these four in advance, finishes in thirty minutes instead of an hour.

Takeaways

The exam ports. What ports is not code — it's the five steps and the grade table as a way of thinking
While you're porting it, grade your own tools first. I found FATAL 3 in the tool behind my own published numbers
Distrust any manual that says "the same everywhere." Mine did, and it had four defects

P.S. The 15-question comment exam and its grading results are public → github.com/ramses203/llm-test-harness, comment_exam.py

Next up: I gave the same exam to an LLM. The regex-vs-LLM decision was made by that scorecard.

The AI Exam Author Was Never Wrong. I Still Can't Use Its Exam.

John Green — Wed, 26 Aug 2026 01:57:57 +0000

In the first post of this series I wrote about making a 29-question exam and getting it wrong five times myself — three times in the answer key, twice in the grader. Ever since, one question kept nagging me: if an AI wrote the exam, how many times would it get it wrong?

So I counted.

The experiment — I ordered 50 questions from an author AI

Hamel Husain's evals essay — the closest thing this field has to a textbook — describes how to scale up test questions: split the feature into scenarios, then mass-generate the input sentences with an AI. After putting an LLM in the grading seat, this was the next piece to test for real.

I gave the author AI (Sonnet 5) three things:

The 50-item product catalog — the exam's reference data
A list of trap types — missing specs, missing units, confusable products, mid-message reversals
An order to write the answer key alongside each question, including the flag from the porting guide: "can the reference data alone decide this one?"

Five questions per request, ten requests, fifty questions.

Review came in three layers:

layer 1   code check      nonexistent product codes? flag contradicting the key?
layer 2   AI reviewer     per question: "is this answer justified by the data?"
layer 3   human (me)      full re-read of all 50

Layer 3 has a story. The original design stopped at two layers — and only after running it did I notice: the author was Sonnet, and the reviewer was Sonnet. I had written "never let a model grade its own answers" in the judge post, then made exactly that mistake. So I sat down and re-read all fifty myself.

The result — zero mistakes

structural errors (fake codes, flag contradictions)   0
answer-key errors (all three review layers)           0

I, the human author, got the answer key wrong three times in 29 questions. The AI author got it wrong zero times in 50. Every fully-specified order had the right code and quantity. Every undecidable one went to "needs confirmation" with the correct candidates attached. All fifty flags matched their answers.

The questions weren't bad, either. Take this one —

Boss, please send 3 boxes of the post-office boxes~ same size as last time!

It planted the fake hint "same as last time" on its own. There's no order history in the data, so nothing can pin the size down — and its answer key says exactly that: needs confirmation, three post-office box candidates. Correct. It reinvented a trap from my original exam ("the usual, 3 boxes") without being told to.

If the story ended here, the conclusion would be "let the AI write the exams."

The twist — I can't run this exam

Nothing in the 50 questions is wrong. But lay them side by side and something else shows.

First, the composition is a photocopy of my instructions. I had told it: "per 5 questions, mix 1 normal order, 1 non-order, 1 change/cancel, 2 traps." Ten requests later, the counts came out exactly 10 / 10 / 10 / 20. The 20 traps split into a tidy 10 missing-spec and 10 mid-message-reversal. A human author would have drifted — "shouldn't something like this go in too?" — and produced a few questions that escape the plan. There isn't one.

Second, the materials repeat. Counting the products in the confirmed answers, one item — shipping box A-type 300 — appears 12 times. I handed it fifty products; it kept reusing its favorites.

Third, the types I didn't order never appear. My original 29 questions had ultra-abbreviations ("250 5"), typo-ridden messages, after-learning questions, custom-order inquiries. The 50 new questions contain none of these. The reason is simple: I didn't put them in the prompt.

Put the three together: the AI author is never wrong. It only writes what you ordered.

Now look back at human me. I got the exam wrong five times while making it. But that same process was where the traps got invented — putting two tape widths in on purpose, making up a sentence like "250 5", imagining the accidents that only happen after the system has learned. The five mistakes were the tuition I paid while doing the inventing.

So the authoring job splits in two

The porting guide says the accident list decides the question count. This experiment showed the other side of that sentence:

inventing accident types      human work.  it comes from living in the domain
stamping questions per type   AI work.     more accurate than me, never tires

When I made 29 questions alone, those two jobs were one lump. Separated, the real bottleneck of exam-building isn't the question count — it's the length of your accident list. Discover one new type, and the AI stamps ten questions of it in five minutes.

Honest caveats

The zero-error run may owe a lot to my clean 50-item catalog. Whether it holds on real-world dirty data — the same product registered three times under different names — is untested
"Only what you ordered" can be partly relieved by writing more types into the prompt. But knowing which types to add is exactly the part that stays human

Takeaways

Delegate question mass-production to an AI. Even with answer keys included, it was more accurate than the human author
But the reviewer must be a different model or a human. An author reviewing itself isn't a review
Inventing trap types stays human work — and the mistakes you make while building the exam are that invention happening

P.S. The question-generation script (gen_cases.py) is in the repo with everything else → github.com/ramses203/llm-test-harness

The Question That Picked My Model Didn't Survive Five Reruns

John Green — Wed, 26 Aug 2026 01:56:12 +0000

Correction (Sep 2026): The lid question was later found to be mis-keyed. Every "harmless" mark in the tables below was the model asking which neck, which the corrected key counts as correct. Re-scored: cheap model 143 clean / 0 harmless / 2 risky, expensive model 144 / 0 / 1, fatal still 0 for both. Details in this post.

Eleven hours after the model-comparison post went up, a reader named Vinh Nguyen left a comment that unraveled its headline:

FATAL 0 against FATAL 0 is a tie in counts, but it is 29 items run once each. [...] Zero fatal out of 29 single trials is consistent with a true fatal rate up to about 10% at 95% confidence. [...] Running the whole exam five times per model and comparing fatal rates per item would tell you whether the tie is real or just the resolution of one pass, and it is the same order of cost as the run you already did.

He's pointing at a self-contradiction I published without noticing. My own post says, in the temperature section: "final scores should come from multiple runs, reported as a pass rate." And then the comparison itself — the whole "won by exactly one question" headline — sat on one pass per model.

No defense available. So I ran it.

The rerun

29 questions × 5 fresh runs × 2 models = 290 trials. Same harness, same unpinned temperature as the original — that variance is the thing being measured. Every raw answer sheet saved. All 290 completed this time (the original expensive-model run had dropped one question to a rate limit).

             Haiku (cheap)        Sonnet (3x price)
trials       145                  145
FATAL        0                    0
RISKY        1                    1
MISSED       0                    0
HARMLESS     4                    5
clean        140/145              139/145
per run      29·28·27·28·28       28·28·28·28·27

Two things in that table killed my original framing.

The question that picked the winner was a coin flip

The original comparison came down to exactly one question — the lids. The customer writes: "10 boxes of the clear PET 300, and 5 bags of the white lids." The catalog has four lids to pick from:

CAP-300-W    white lid, 300-neck   (fits the PET-300 bottle)
CAP-300-BK   black lid, 300-neck
CAP-500-W    white lid, 500-neck   (fits the PET-500 bottle)
CAP-500-BK   black lid, 500-neck

"White" narrows it to two — but which neck? The clue sits in the same sentence: the customer is ordering PET 300 bottles, and nobody pairs 300 bottles with 500 lids. The expensive model read that context and confirmed the 300 white lid — and I called that the win. The cheap model asked "which neck?" instead.

Across five fresh runs, the expensive model never did that again. Zero confirms in five passes — it asked "which lid?" every single time. The only model that produced the winning confirm even once was the cheap one, in one run of five.

So the one-question gap my title stood on disappears the moment the exam is rerun. It was never a real ability gap between the two models. It was a coin flip that happened to land the expensive model's way on the one day I ran the exam once — and I published that single flip as the result.

The two RISKY trials landed exactly where the flag said

Each model produced one RISKY trial in 145 — the grade for "confirmed something ambiguous; right this time, fatal next time."

Haiku, run 3, on "the usual, 3 boxes": two items in the order history both fit, and it presented only one of them as a candidate instead of both
Sonnet, run 5, on the ultra-abbreviated "250 5": five products start with 250, and it confirmed one anyway

Both questions carry rule_decidable: false — the flag a reader talked me into adding: "can the reference data alone pin the answer down to exactly one?" On decidable questions, the run-to-run wobble drifted safe — extra clarifying questions, nothing worse. The dangerous direction — confirming what cannot be decided — showed up only on the flagged questions, at about 0.7% of trials per model.

The flag didn't just audit my answer key. It predicted where run-to-run variance turns dangerous.

So does the original conclusion survive?

The conclusion of that post — it's a tie on fatal errors, so use the cheap model — survives. What gets replaced is the evidence it stands on.

The old evidence was one pass: I ran the exam once, fatal errors came out 0 to 0, and the whole visible difference was that one lid question. The lid question is out — it was luck. What takes its place: fatal errors stayed 0 to 0 through 145 trials per model, and RISKY came out tied too, 1 to 1. By the rule of three, 145 trials tighten the fatal-rate bound on this exam from about 10% (at 0/29) to about 2%. On questions the model hasn't seen, I still have only 29 kinds — so that ceiling stays near 10%. The tie is no longer one lucky afternoon — it is a measured result. And on a tie, take the cheap one: the conclusion stands.

What doesn't stand is the margin I published. Clean counts wobbled between 27 and 29 per run for the cheap model and 27–28 for the expensive one. On any single pass, either model can rank first by a question or two. A one-pass margin of one question is below this exam's resolution — which is what the comment said before I spent a cent measuring it.

Takeaways

A single pass ranks models with ±1–2 questions of noise. Don't publish a one-pass margin — mine didn't survive its own rerun
"Zero fatal" means little until you say over how many trials — and of what. 0/29 caps this exam's fatal rate near 10%; five reruns (0/145) cap it near 2%, on this exam. New questions are a separate count. The reruns cost about as much as the first run
Variance has grades too. On decidable questions it drifted safe (over-asking). The dangerous drift — confirming the undecidable — appeared only on questions pre-flagged as rule-undecidable
The exam caught its author again. Not in the answer key this time, and not in the grader — in how confidently I read a single run

P.S. All 290 raw answer sheets, the 5× driver, and the aggregator are in the repo → github.com/ramses203/llm-test-harness

P.P.S. If you'd rather get new experiments by email: ramses203.substack.com/subscribe

Steal This Exam. Here's How to Port It to Your Own Pipeline.

John Green — Tue, 25 Aug 2026 00:27:34 +0000

So far this series has been about giving my order-reading LLM an exam.

Some of you have been reading it thinking: "Mine isn't orders, it's meeting-minutes summarization." "I'm using it for email triage."

Good news — the exam is built to be stolen. Only two things need swapping: the reference data you match against (mine: a product catalog) and the worst accident (mine: wrong goods loaded onto a truck). Let's go in order.

Step 1 · Write down the worst first

Before writing a single test question, write this:

When this AI is wrong, which of the consequences cannot be undone?

For my order program it was "the wrong goods get loaded onto a truck." What is it for yours? A few examples —

auto-reply email AI    a wrong answer goes out to the customer — you can't unsend it
meeting-minutes AI     writes "decided" on something undecided — once shared, work proceeds on it
research AI            an invented number enters the report — reports travel upward
data-cleanup AI        overwrites the original — no backup, no recovery

You can see the pattern. Sent, deleted, escalated — these don't come back. That's your truck.

Step 2 · Build the grade table

Once the worst is written down, the grades fall out on their own. One criterion — can a human undo it?

FATAL      cannot be undone              sent, deleted, reported, charged
RISKY      confirmed something ambiguous  right this time, fatal next time
MISSED     dropped something              a human can still catch it
HARMLESS   over-asks "please confirm"     just slower

The only line you have to define yourself is the first one. The other three read the same in any project.

And take the principle with you as-is: a wrong confirmation is worse than no confirmation.

Step 3 · Plant the traps

Exam questions come from traps, not from normal cases. There are four kinds, and every domain has all four.

Confusable pairs — mine was clear tape 48mm vs 60mm. In meeting minutes: two attendees named Kim, one an associate, one a manager. In email: reply vs forward. Find the pairs in your data that are similar enough to confuse, and deliberately put both in.

Plausible non-targets — "What are the specs on the 250 shipping box?" contains a product name but is not an order. In minutes: "let's decide that next time" (not a decision). In email: a promotional mail (not something to answer). Put in things that only look like what the AI is supposed to catch.

Mid-message reversals — "5 boxes please — no wait, make it 3." In minutes: "let's go with A → actually B is better." Questions that are wrong if you only read the first half.

Accidents after learning — mandatory if your system has memory. A situation where fresh information must beat the remembered value ("tape 60"), and a situation where memory must not change the verdict (a question is still a question).

How many questions? The metric isn't a count — it's your accident list from Step 1. At least one question per accident type, built to cause exactly that accident. From my list:

inquiry mistaken for an order → goods nobody ordered ship   → "What are the specs on the 250 shipping box?"
change/cancel mistaken for a new order → ships twice        → "I ordered 5 boxes — please send only 3"
unit misread → quantity ships at 50x                        → "250 5"

Five accidents on your list means five questions minimum. Mine grew to 29 because I kept adding variants per accident; 10 is plenty to start.

Add a few normal, well-behaved cases at the end — only a few. Models rarely fumble the easy ones, so normal cases mostly catch nothing. The exam's job is not to watch the model succeed. It's to find where the accidents are.

Step 4 · Write the answer key — then doubt it

For each question, write the expected answer. And here comes the lesson of the first post:

Your answer key will be wrong. I was wrong three times in the key and twice in the grader.

So keep three rules.

The correct answer to an ambiguous question is "needs confirmation" — not a plausible-looking verdict
When the model answers differently from your key, re-read the data before docking points. The model may be right. But if your fix turns a "needs confirmation" into a confident verdict, be suspicious of the fix first
For every question, record in advance: "does the reference data alone pin the answer down to exactly one?" When the model and the key later disagree, that flag is the referee

The third rule by example — "tape 60, 2 boxes": the catalog has a 60mm, so the data pins the answer. If the model answers differently here, the model is wrong; the key stands. "Clear tape, 2 boxes": no width given, the data cannot pin it. The correct answer was "needs confirmation" from the start — if I wrote anything else in the key, the key is what's broken.

A reader of the first post talked me into this flag. Bonus effect: recording it forces you to open the reference data for every question while you're still writing the exam — which catches answer-key mistakes at authoring time.

Step 5 · Grade, save, decide

Count results by grade, not by score. And save the model's raw answer sheets to files — grading criteria keep changing, and with the raw sheets you can re-grade without calling the model again.

Everything passed? Then write the three-line ledger. It's you being honest with yourself about what the exam guarantees and what it doesn't.

VERIFIED       one question per accident type, and FATAL held at zero
NOT VERIFIED   production data has never been through it — I invented every sentence myself
GUARDED        on anything uncertain it must not confirm — it hands off to a human

What justifies shipping is not line one. It's lines one and three combined. The exam score covers only what I could imagine (line two is that confession), and the sentences I couldn't imagine land on line three's guard. Without line three, an exam score is just a number — the first unimagined sentence causes the accident.

The whole thing on one card

Write down the accidents that can't be undone — that's your truck
Build the grade table — only the first line is yours to define
Plant the four traps — confusable pairs, plausible non-targets, mid-message reversals, after-learning
Write the answer key, then doubt the answer key
Grade by severity, save raw sheets, write the three-line ledger

The exam you may steal. The answer key you have to write yourself.

And that answer key will be wrong. Catching it is what the exam is for.

P.S. The test runner and grader that run exactly this structure are public → github.com/ramses203/llm-test-harness

I Scraped 20,000 YouTube Comments. The Videos and the Comments Were Having Two Different Conversations.

John Green — Tue, 25 Aug 2026 00:25:03 +0000

I once collected about 22,000 comments from roughly 140 Korean YouTube videos about AI coding tools and classified them. (Quotes below are translated from Korean.)

I wanted to see what people were asking. What came out was something else.

What the videos teach

Put the titles and tags of those 140 videos in one pile and they say:

How to install. How to get started. How to build an app. Which tool is best.

All of it is "starting." Follow along, a result appears on the screen, the video ends.

What the comments say

The comments sweeping up the likes were telling a different story.

"Verifying AI mistakes takes so much time. Checking every answer for nonsense got so tiring I just do the work myself now." (👍598)

"Coding with AI makes me anxious. If one bug ships, I'm the one responsible. Checking and debugging everything one by one ends up being more work." (👍265)

"I pay every month and it lies about work matters like it's nothing." (👍72)

"Tokens burn too fast… added $50 and it was gone in half a day." (👍30)

It compresses into three complaints: expensive, can't trust it, can't fix it.

The videos teach the start. The people are dying right after the start.

The scariest comment

"Asked it for shampoo recommendations and it recommended one that doesn't exist. Slipped it in between real products — with the weight, the benefits, even a price." (👍49)

That comment is the essence of the problem. When AI is wrong, it doesn't look wrong. The fake sits among the real ones, wearing plausible numbers.

This is why "just write better prompts" is half an answer. Better prompts lower the odds of being wrong. They don't create a way to know when it's wrong. Drop the error rate from 10% to 3% and you still don't know where the 3% is hiding. If that 3% detonates inside payment logic, money leaves the building.

One more finding — where the real questions live

While collecting, I noticed the nature of comments changes with channel size.

multi-million-sub videos   real questions/needs = 12% of comments — the rest is reactions and anxiety
10k–300k sub channels      real needs = 26% — "I followed along and got stuck RIGHT HERE"

It's the difference between spectators and people actually doing the thing. Comments under the big videos ask "what happens to us in the AI era." Comments under mid-size channels ask "how do I fix this error."

The real questions were in the mid-size channels. And collecting reply threads, not just top-level comments, surfaced 470 more needs that the top level never showed.

So I picked a direction

Content that teaches starting is already everywhere. I decided to work on the stretch right after the start — the can't-trust-it, can't-fix-it stretch.

The earlier posts in this series are the first result of that decision: giving my LLM an exam, planting traps in it, grading by severity instead of pass/fail, why passing still wasn't enough to ship. Every one of them is about turning "I can't trust it" into "I verified it."

People don't die at the start. They die right after it.

But almost everyone is selling the start.

P.S. The classifier that produced the percentages above later sat an exam of its own — the same kind I give my LLMs — and failed three fatal-grade questions. That public correction is its own story, coming later in this series.

Next up: how to steal this exam and port it to your own pipeline, step by step.

P.P.S. If you'd rather get new experiments by email: ramses203.substack.com/subscribe

It Passed Every Test. That's Why It Can't Ship Yet.

John Green — Mon, 24 Aug 2026 00:32:20 +0000

My order-reading LLM passed the 29-question exam. Zero fatal errors. The model is chosen.

So — ship it?

No. And the reason is the most important thing in this series.

The exam has holes. I am the holes.

The questions, the answer key, the product catalog — I made all of it.

Which means "passed 29 questions" translates precisely to:

"It made no mistakes in the 29 situations I was able to imagine."

Nothing more. The sentence I couldn't imagine is not on the exam. And production is a parade of sentences you couldn't imagine. Real customers abbreviate in ways I can't invent, and real product catalogs are far dirtier than the one I wrote — the same product registered three times under different names, dead items nobody deleted piling up for years.

So the ledger has three lines

Before shipping, I wrote these three lines down.

VERIFIED       one question per worst-accident type; both models at zero
NOT VERIFIED   never ran on production data. Every exam sentence is mine
GUARDED        when unsure, it must not confirm — it hands off to a human

The third line is the one that matters. If the unknown case falls somewhere safe by construction, an incomplete exam is still shippable. When a never-seen phrasing arrives, this program's worst case is "slower" — not "wrong goods shipped."

Without that guard, shipping on exam scores alone is trusting a gun because it fired 29 times without jamming.

The first stretch of production is a watch period

For a while after launch, nothing passes through automatically. A human reviews every result.

What comes out of this period is the real exam. The sentences I couldn't imagine make their first appearance here. Every miss becomes a new exam question. This is the moment the exam grows from imagination into production.

Week 1      humans check everything. Every miss becomes a test case
Weeks 2–4   if FATAL holds at 0, auto-pass the confirmed ones only
After       humans only see the "needs confirmation" queue

The exam also tells you when to fold

There's an opposite case: FATAL keeps appearing, and the cause isn't the prompt — it's the question itself.

"Figure out what 'the usual' means, automatically" is that kind of question. It cannot be done in principle. The information isn't there. No amount of prompt polish fixes it, and every polish makes it worse by teaching the model to guess with confidence.

The move there isn't fixing the program. It's shrinking the scope. Draw the line — "that case goes to a human" — and automate the rest. Finding out what is impossible in principle is also the exam's job.

The scoreboard so far

Counting what this exam actually caught:

The model's real mistakes — 1. And its only crime was asking.

The exam author's mistakes — 5. Three in the answer key, two in the grader.

The exam I built to verify the AI caught me more than it caught the AI. Which is the real reason to build one.

Passing is the starting line. Production writes the next questions.

P.S. The scoreboard above changed again later, when the grader got fixed one more time. That story is its own post — it's here.

All 29 questions and the grader are public → github.com/ramses203/llm-test-harness

The Model That Costs 3x More Won by Exactly One Question

John Green — Mon, 24 Aug 2026 00:29:58 +0000

Correction (Sep 2026): The lid question that decided this comparison was later found to be mis-keyed. The customer never stated the neck size, so asking was the correct answer, not confirming. Under the corrected key the cheap model's answer was right and the expensive model's was a guess. The re-scoring is in this post.

I gave the same 29-question order-reading exam to two models.

A cheap one (Haiku 4.5) and one that costs about three times as much (Sonnet 5).

The result

Cheap model       28 of 29 questions clean
Expensive model   all 28 executed questions clean
Fatal errors      zero for both

(One of the expensive model's questions never ran — rate limit.)

The 3x model really was better. By exactly one question.

That one question

10 boxes of the clear PET 300, and 5 bags of the white lids

The catalog has two kinds of lids: 300-neck and 500-neck.

"White lids" alone doesn't tell you which. But the same sentence starts with "PET 300." Nobody orders 300-size bottles and 500-size lids in the same breath.

The expensive model read that context and confirmed the 300 lid.

The cheap model asked back: "diameter unspecified, needs confirmation."

In the first post of this series, "needs confirmation" was the correct answer for the clear tape — so why is confirming correct here? The difference is the clue. The tape sentence contained nothing to decide the width. This sentence has "PET 300" on the same line. If the message itself contains the information to decide, confirming is correct. If it doesn't, asking is correct.

And look at how the cheap model was wrong. It didn't ship the wrong lid. It asked. No accident happens. The user just gets mildly annoyed if every lid order comes back as a question. That's the size of the gap.

So which did I pick?

The cheap one.

The criterion is severity, not score (earlier post). Both models at FATAL 0 is a tie on the grade that matters. On a tie, take the cheap one.

If it had gone the other way, there'd be nothing to think about — pay up. Say the scoreboard had looked like this:

Cheap model       28/29  FATAL 1   ← higher score, unusable
Expensive model   27/29  FATAL 0

Pick by "how many did it get right" and you pick the wrong model.

Bonus trap — same exam, different scores

One thing kept nagging me.

I gave the same question to the same model again, and the result changed. Wrong once, right on the retry, then right three more times. One miss in five runs.

The cause is temperature — the knob that controls randomness in model output. Pin it to 0 and the same input gives the same answer. The problem: the CLI tool I was using had no such option.

Two lessons:

If you can't pin temperature, a single run's score is not reproducible. Final scores should come from multiple runs, reported as a pass rate
Production calls run at temperature 0. Especially for work where the same order sheet must produce the same answer every day

Takeaways

Compare by severity, not score
FATAL 0 vs FATAL 0 is a tie. Take the cheap one
Pin the temperature or don't trust the number

Before you pay for the expensive model, count what the cheap one actually gets wrong. In our exam the entire gap was one question — and even that one was guilty of nothing but asking.

P.S. Next up: the exam was passed — and that's exactly why it can't ship yet.

All 29 questions and the grader are public → github.com/ramses203/llm-test-harness

A Reader Caught My Answer Key Drifting Toward the Model

John Green — Sat, 22 Aug 2026 06:04:44 +0000

Three days after the first post of this series went up, a comment arrived on a Korean tech-news site where it had been shared. The first half was praise. The second half was this:

While the exam author was losing three times, every answer-key revision went in the direction of "the model was right." If that repeats, isn't there a risk the answer key converges on model behavior? I split questions by whether domain rules alone can decide them, independently. If they can't, I pull them out of scoring entirely — they become human-review cases, not test cases. How did you distinguish these?

Unpacked: "if you revise the answer key every time the AI answers differently, the exam stops evaluating the AI and becomes a sheet of paper that transcribes it."

Imagine a teacher who revises the answer key every time they look at a student's answers. A few times, the teacher really was wrong. But once it becomes a habit, that exam no longer measures the student. The scores keep coming out great — not because the student is good, but because the key keeps accommodating the student. And I use those scores to decide whether to ship.

The comment had found the weak spot of the whole series.

My answer — auditing the three revisions

I went back and checked my three answer-key fixes against one criterion. What did I trust when I changed the key — the plausible explanation the model attached to its answer, or the product catalog and the original message?

All three had been fixed after opening the catalog and the message directly. The clear tape came in two widths in the catalog and the customer never said which. The order "250 5" matched five products starting with 250, with no unit written anywhere. "The usual" had exactly one record in the history — and nothing guaranteed the pointer pointed at it. All three were questions whose answer the data alone could not pin down.

And all three revisions moved in the same direction: from "confirmed" back to "needs confirmation." I never once copied the model's chosen code into the key as the new answer.

That yields a watch rule. If the key ever starts drifting toward the model, the revisions will run the other way — "needs confirmation" hardening into "confirmed," or code A being replaced by the model's code B. The first revision in that direction is the stop signal. Freeze the change, hide the model's answer, and re-derive the answer from the data alone.

And the admission — I had no criterion beforehand

Honestly, though: those three judgments were good after the fact, not by prior design. The commenter's method — marking each question at authoring time by whether rules alone can decide it — was better than mine.

So I adopted it as-is. Every one of the 29 questions now carries a flag: rule_decidable. True if the data pins down exactly one answer. False if the correct answer is "needs confirmation."

Adding the flag came with a built-in audit: all three questions where I had fixed the answer key came out false. All three revisions had been cases of "I confirmed an answer on a question that was never decidable in the first place" — confirmed once more, this time by a flag.

There was a bonus I didn't expect. To set the flag you have to actually open the catalog for every question — decidability can't be judged without looking. So answer-key mistakes get caught at authoring time, not after publishing. If this flag had existed when I wrote the tape question, opening the catalog would have shown me two tape widths, and "48mm, confirmed" would never have been written into the key.

Two referees now sit on the exam

When the model and the answer key disagree, the procedure is now:

rule_decidable = true   →  the model is wrong. The key stands
rule_decidable = false  →  if the key isn't "needs confirmation," the key is at fault
the flag itself suspect →  reopen the data, re-judge the flag first

No more improvising at every dispute. The flag set at authoring time referees first. The revision direction — confirmed→needs-confirmation is the only healthy direction — watches second.

Takeaways

Revising the answer key is not the sin. Revising it after checking the catalog and the original message is normal; revising it because the model's explanation was persuasive is where the rot starts
You will not feel the moment you cross that line. So pick a visible signal in advance: any revision that turns "needs confirmation" into "confirmed" means stop and re-derive with the model's answer hidden
The best defense is marking decidability at authoring time. Setting the mark forces you to open the data, and that act itself is the audit

One comment changed the design of the exam. Publish, and the verification comes to you.

P.S. The exam with rule_decidable flags on all 29 questions is in the repo → github.com/ramses203/llm-test-harness

I Made an LLM Re-Grade My Exam. It Found Two Bugs in My Grader.

John Green — Sat, 22 Aug 2026 05:59:50 +0000

In an earlier post I wrote that my grader had been wrong twice — zeroing a perfect answer over truncated JSON, and penalizing a good answer. Both were caught by a human re-reading the answer sheets.

It turns out the grader had been wrong two more times. This time the catcher wasn't a human. It was an LLM.

Why I put an LLM in the grading seat

There's an essay that's become something of a textbook for AI verification: Hamel Husain's "Your AI Product Needs Evals." Reading it, the skeleton was almost identical to what I'd built — except for one step I didn't have: making an LLM do the grading.

When answer sheets grow to hundreds a day, no human can read them all. So you delegate grading to an LLM — which also gets things wrong. Hamel's prescription: before you trust the judge, make the judge take an exam of its own. Have your existing grader and the LLM judge grade the same answers independently, and measure how often they agree.

I have a code grader instead of a human, so my version was: give the same 29 answer sheets to both graders. One is the grading rulebook implemented as code. The other is an LLM (Sonnet 5) that read the same rulebook in plain language.

Picking the judge, I followed two of Hamel's rules. The judge should be a stronger model than the examinee (the answers were written by Haiku). And never let a model grade its own answers — models are known to be generous to themselves.

The result — 93% agreement, and two splits

On 27 of 29 sheets the two graders agreed, down to the exact severity grade. If it had ended there: "LLM judges work, neat."

The two splits were the problem. I dug in, and both times the LLM was right and my code was wrong.

First — the lids. The question:

10 boxes of the clear PET 300, and 5 bags of the white lids

The same sentence says "PET 300," so the lids are the 300-size. Answer key: confirm the 300 lid.

The model's answer didn't confirm. "300 or 500? Please check" — with both candidates listed.

The two graders split on this sheet:

Code grader — two penalties. The correct lid code wasn't in the confirmed list, so it counted MISSED, and the extra confirmation request counted as over-asking on top
LLM judge — one HARMLESS. "Not missed — caught and asked. No accident here. Just over-caution"

Re-reading my own grade table, the LLM was right. My definition of MISSED is "dropped an order; the customer calls." Nobody calls about this answer — they just get asked "which lid?" I had even described this exact answer as "guilty of asking" in an earlier post of this series. The prose understood it. Only the code insisted it was "missed."

Why: the code mechanically checked whether the correct code appeared in the confirmed list. An item that's being asked about isn't in that list, so — "not there, missed." The code had no concept of "currently being asked."

Second — the bubble wrap. The question:

Add 2 rolls of bubble wrap to my earlier order

Not a new order — a request to append to an earlier one. Answer key: classify as an addition request, done.

The model classified it correctly. Then it did one more thing: it left a note in the margin of the answer sheet — "FYI, bubble wrap comes in 30cm, 50cm and 100cm; which one needs confirming."

The split:

Code grader — clean. But for the wrong reason. My grader only reads two boxes on the answer sheet: "orders" and "custom orders." The note lived outside both. So this "clean" wasn't "no defects" — it was "didn't look where the defect was"
LLM judge — HARMLESS. It read the whole sheet, saw the margin note, and wrote "one confirmation request that isn't in the answer key; not an accident"

It's the difference between an OMR machine and a teacher. The OMR machine reads the bubbles; a teacher reads the doodles in the margin. My grader was an OMR machine.

The verdict, and the fix

When two graders disagree, who decides? The person who wrote the rulebook — me. Both graders are interpretations of my rules, so when the interpretations split, the author has to check the original intent.

I re-read both sheets. Both times the LLM judge matched the rulebook's intent. My grader's lifetime error count went from two to four.

Two fixes:

Items that were caught-and-asked count as HARMLESS, not MISSED
Confirmation requests outside the order box get read too

I re-graded all 29 sheets with the fixed grader and diffed against the judge again. This time all 29 sheets matched — not just clean/not-clean, but the exact severity of every defect.

The twist — my published score changed

With the fixed grader, the score I'd published in the first post changed. The model didn't retake the exam. The answers are identical. Only the score moved.

             old grader              new grader
Haiku        28/29 clean             27/29 clean
             1 missed · 1 harmless   0 missed · 2 harmless
FATAL        0                       0

One fewer clean sheet — not because the model got worse, but because the grader got more accurate. A defect that used to be invisible (the margin note) became visible. And MISSED dropped to zero: it turns out Haiku never missed anything. It was just over-careful.

FATAL is still 0, so the ship/no-ship call didn't change. But the lesson stuck: a score is a function of the grader before it's a result of the exam. Publish the grader version with the score, and accept that a better grader can change old scores.

Three honest caveats

26 of my 29 sheets were clean answers. When most of the sample is normal, agreement rates come out high on their own (a trap Hamel warns about too). The genuinely hard judgments are the defective few — only 3 sheets here
One judge call died on a timeout and had to be re-run. If you run an LLM judge in production, failure handling is mandatory
This was an exam where the judge got the rulebook handed to it. Grading without a rulebook is a different, much harder problem

Takeaways

You can delegate grading to an LLM — after the judge takes an exam of its own
Code graders read fixed boxes; LLM judges read the whole sheet. Stack them and they cover each other's blind spots
Publish the grader version alongside the score. Better graders change old scores.

P.S. The judge (judge.py) is in the repo with everything else → github.com/ramses203/llm-test-harness