I gave my LLM a 29-question order-reading exam. Last time was how to build the exam. Today: grading.
Grading gets its own post for a reason. Build the grading wrong, and the score lies to you.
5 wrong out of 29 — can I ship?
No idea. Because "which 5" is missing.
If it missed 5 typo-riddled questions, ship it. But if one of those 5 was reading "please cancel my order" as a NEW order? Then even with everything else perfect, you can't ship. That program sends goods to a customer who just cancelled.
So don't grade by count. Grade by severity.
Severity = "can a human undo this?"
My grader has 4 grades. One criterion — is it reversible? In this program, the irreversible moment is when the wrong goods get loaded onto a truck.
FATAL Wrong goods on the truck. Cannot be undone
RISKY Confirmed something ambiguous without asking. Right this time — fatal next time
MISSED Dropped an order. The customer calls. Fixable
HARMLESS Over-asked "please confirm." Just slower
One principle falls out of this:
A wrong confirmation is worse than no confirmation.
Sounds obvious. In production you'll be tempted to flip it. Someone complains "it asks for confirmation too often," so you lower the confidence bar. The screen gets cleaner. And the accidents start happening off-screen.
The same 28/29 splits two ways
FATAL 0 · MISSED 1 → Ship it. Humans catch what it drops
FATAL 1 · everything else perfect → Don't ship. You don't know when that 1 comes back
Same score. Opposite fates.
Two accidents my grader caused
The grader is code I wrote. Like all code I write, it had bugs.
Accident one — zero points over formatting. A model answer was perfect in content, but the JSON wrapper arrived with the tail cut off. The grader ruled "broken format = fatal." A 100-point answer, zeroed over one missing brace.
The fix is simple: count the open brackets and close what's missing (ignoring brackets inside strings). The actual code is in parse_json in the repo.
Accident two — penalizing a good answer. For "250 boxes, 5 units" the model answered:
Verdict: needs confirmation
Likely candidate: shipping box 250
Reason: if "units" means boxes it's 5 boxes; if sheets, 0.1 box — cannot confirm
A considerate answer — asks for confirmation AND offers a hint. But my grader saw the candidate field filled in and ruled "aha, you confirmed!" Wrong answer. I fixed it to read the verdict field first.
Lesson: when the grader is wrong, you end up "fixing" a healthy model. And every fix makes it worse.
Two operational tips
Save every model answer to a file. Never throw them away. In this project, what kept changing wasn't the model's answers — it was the grading side. I fixed the answer key three times and the grader twice. Each fix means re-grading all 29 questions. With saved answers, re-grading takes seconds. Without them, one re-grade means calling the model 29 times again. It's the difference between re-marking stored answer sheets and calling every student back to retake the exam. I built this as a --rescore flag.
Write results to disk after every single case. In another experiment I collected 5,578 items with a save-at-the-end design. The last request failed and took all 5,000 with it. Paid API — every lost item was money. Learned that one the hard way.
The takeaway
When the results come in, the question is not "how many did it get right?"
It's "among the failures, is anything irreversible?"
That's why I only really read one line of the grader's output. Fatal = 0: ship. Fatal = 1: don't — even if everything else is perfect.
P.S. Next up: the model that costs 3x more won by exactly one question.
All code and the 29 questions are public → github.com/ramses203/llm-test-harness
Top comments (2)
This third part really completes the picture for me. 👏
The shift from “how many did it get right?” to “is any failure irreversible?” is exactly the kind of distinction I want to see in production AI systems.
A 98% score means very little if the remaining 2% contains one action that crosses an irreversible boundary. Severity-based evaluation makes much more sense than treating every failure as equivalent.
I also really like that the grader itself becomes part of the system under test. Across the three parts, the failure surface keeps moving: first the model, then the exam design and learned state, and finally the grading logic that decides whether the model is safe to ship.
That creates a much more interesting verification chain than a simple benchmark: model → expected behaviour → grading logic → deployment decision.
And the operational details about persisting every answer and writing results incrementally are exactly the kind of things that turn an experiment into something reproducible and trustworthy. 🔐
Really enjoyed the whole series, John. Excellent work. 👏
Thanks for reading the whole series this carefully, Marco. Your four-stage chain — model → expected behaviour → grading logic → deployment decision — is honestly a better map of this series than my own table of contents. The last stage just got its own post today: same exam, two models, FATAL 0 on both, and the tie-breaker that decides what actually ships.