Grade Your LLM Pass/Fail and You Will Ship a Disaster

I gave my LLM a 29-question order-reading exam. Last time was how to build the exam. Today: grading.

Grading gets its own post for a reason. Build the grading wrong, and the score lies to you.

5 wrong out of 29 — can I ship?

No idea. Because "which 5" is missing.

If it missed 5 typo-riddled questions, ship it. But if one of those 5 was reading "please cancel my order" as a NEW order? Then even with everything else perfect, you can't ship. That program sends goods to a customer who just cancelled.

So don't grade by count. Grade by severity.

Severity = "can a human undo this?"

My grader has 4 grades. One criterion — is it reversible? In this program, the irreversible moment is when the wrong goods get loaded onto a truck.

FATAL     Wrong goods on the truck.  Cannot be undone
RISKY     Confirmed something ambiguous without asking.  Right this time — fatal next time
MISSED    Dropped an order.  The customer calls.  Fixable
HARMLESS  Over-asked "please confirm."  Just slower

One principle falls out of this:

A wrong confirmation is worse than no confirmation.

Sounds obvious. In production you'll be tempted to flip it. Someone complains "it asks for confirmation too often," so you lower the confidence bar. The screen gets cleaner. And the accidents start happening off-screen.

The same 28/29 splits two ways

FATAL 0 · MISSED 1        →  Ship it. Humans catch what it drops
FATAL 1 · everything else perfect  →  Don't ship. You don't know when that 1 comes back

Same score. Opposite fates.

Two accidents my grader caused

The grader is code I wrote. Like all code I write, it had bugs.

Accident one — zero points over formatting. A model answer was perfect in content, but the JSON wrapper arrived with the tail cut off. The grader ruled "broken format = fatal." A 100-point answer, zeroed over one missing brace.

The fix is simple: count the open brackets and close what's missing (ignoring brackets inside strings). The actual code is in parse_json in the repo.

Accident two — penalizing a good answer. For "250 boxes, 5 units" the model answered:

Verdict: needs confirmation
Likely candidate: shipping box 250
Reason: if "units" means boxes it's 5 boxes; if sheets, 0.1 box — cannot confirm

A considerate answer — asks for confirmation AND offers a hint. But my grader saw the candidate field filled in and ruled "aha, you confirmed!" Wrong answer. I fixed it to read the verdict field first.

Lesson: when the grader is wrong, you end up "fixing" a healthy model. And every fix makes it worse.

Two operational tips

Save every model answer to a file. Never throw them away. In this project, what kept changing wasn't the model's answers — it was the grading side. I fixed the answer key three times and the grader twice. Each fix means re-grading all 29 questions. With saved answers, re-grading takes seconds. Without them, one re-grade means calling the model 29 times again. It's the difference between re-marking stored answer sheets and calling every student back to retake the exam. I built this as a --rescore flag.

Write results to disk after every single case. In another experiment I collected 5,578 items with a save-at-the-end design. The last request failed and took all 5,000 with it. Paid API — every lost item was money. Learned that one the hard way.

The takeaway

When the results come in, the question is not "how many did it get right?"

It's "among the failures, is anything irreversible?"

That's why I only really read one line of the grader's output. Fatal = 0: ship. Fatal = 1: don't — even if everything else is perfect.

---

P.S. Next up: the model that costs 3x more won by exactly one question.

All code and the 29 questions are public → github.com/ramses203/llm-test-harness