I Made an LLM Re-Grade My Exam. It Found Two Bugs in My Grader.

In an earlier post I wrote that my grader had been wrong twice — zeroing a perfect answer over truncated JSON, and penalizing a good answer. Both were caught by a human re-reading the answer sheets.

It turns out the grader had been wrong two more times. This time the catcher wasn't a human. It was an LLM.

Why I put an LLM in the grading seat

There's an essay that's become something of a textbook for AI verification: Hamel Husain's "Your AI Product Needs Evals." Reading it, the skeleton was almost identical to what I'd built — except for one step I didn't have: making an LLM do the grading.

When answer sheets grow to hundreds a day, no human can read them all. So you delegate grading to an LLM — which also gets things wrong. Hamel's prescription: before you trust the judge, make the judge take an exam of its own. Have your existing grader and the LLM judge grade the same answers independently, and measure how often they agree.

I have a code grader instead of a human, so my version was: give the same 29 answer sheets to both graders. One is the grading rulebook implemented as code. The other is an LLM (Sonnet 5) that read the same rulebook in plain language.

Picking the judge, I followed two of Hamel's rules. The judge should be a stronger model than the examinee (the answers were written by Haiku). And never let a model grade its own answers — models are known to be generous to themselves.

The result — 93% agreement, and two splits

On 27 of 29 sheets the two graders agreed, down to the exact severity grade. If it had ended there: "LLM judges work, neat."

The two splits were the problem. I dug in, and both times the LLM was right and my code was wrong.

First — the lids. The question:

10 boxes of the clear PET 300, and 5 bags of the white lids

The same sentence says "PET 300," so the lids are the 300-size. Answer key: confirm the 300 lid.

The model's answer didn't confirm. "300 or 500? Please check" — with both candidates listed.

The two graders split on this sheet:

• Code grader — two penalties. The correct lid code wasn't in the confirmed list, so it counted MISSED, and the extra confirmation request counted as over-asking on top
• LLM judge — one HARMLESS. "Not missed — caught and asked. No accident here. Just over-caution"

Re-reading my own grade table, the LLM was right. My definition of MISSED is "dropped an order; the customer calls." Nobody calls about this answer — they just get asked "which lid?" I had even described this exact answer as "guilty of asking" in an earlier post of this series. The prose understood it. Only the code insisted it was "missed."

Why: the code mechanically checked whether the correct code appeared in the confirmed list. An item that's being asked about isn't in that list, so — "not there, missed." The code had no concept of "currently being asked."

Second — the bubble wrap. The question:

Add 2 rolls of bubble wrap to my earlier order

Not a new order — a request to append to an earlier one. Answer key: classify as an addition request, done.

The model classified it correctly. Then it did one more thing: it left a note in the margin of the answer sheet — "FYI, bubble wrap comes in 30cm, 50cm and 100cm; which one needs confirming."

The split:

• Code grader — clean. But for the wrong reason. My grader only reads two boxes on the answer sheet: "orders" and "custom orders." The note lived outside both. So this "clean" wasn't "no defects" — it was "didn't look where the defect was"
• LLM judge — HARMLESS. It read the whole sheet, saw the margin note, and wrote "one confirmation request that isn't in the answer key; not an accident"

It's the difference between an OMR machine and a teacher. The OMR machine reads the bubbles; a teacher reads the doodles in the margin. My grader was an OMR machine.

The verdict, and the fix

When two graders disagree, who decides? The person who wrote the rulebook — me. Both graders are interpretations of my rules, so when the interpretations split, the author has to check the original intent.

I re-read both sheets. Both times the LLM judge matched the rulebook's intent. My grader's lifetime error count went from two to four.

Two fixes:

• Items that were caught-and-asked count as HARMLESS, not MISSED
• Confirmation requests outside the order box get read too

I re-graded all 29 sheets with the fixed grader and diffed against the judge again. This time all 29 sheets matched — not just clean/not-clean, but the exact severity of every defect.

The twist — my published score changed

With the fixed grader, the score I'd published in the first post changed. The model didn't retake the exam. The answers are identical. Only the score moved.

             old grader              new grader
Haiku        28/29 clean             27/29 clean
             1 missed · 1 harmless   0 missed · 2 harmless
FATAL        0                       0

One fewer clean sheet — not because the model got worse, but because the grader got more accurate. A defect that used to be invisible (the margin note) became visible. And MISSED dropped to zero: it turns out Haiku never missed anything. It was just over-careful.

FATAL is still 0, so the ship/no-ship call didn't change. But the lesson stuck: a score is a function of the grader before it's a result of the exam. Publish the grader version with the score, and accept that a better grader can change old scores.

Three honest caveats

• 26 of my 29 sheets were clean answers. When most of the sample is normal, agreement rates come out high on their own (a trap Hamel warns about too). The genuinely hard judgments are the defective few — only 3 sheets here
• One judge call died on a timeout and had to be re-run. If you run an LLM judge in production, failure handling is mandatory
• This was an exam where the judge got the rulebook handed to it. Grading without a rulebook is a different, much harder problem

Takeaways

1. You can delegate grading to an LLM — after the judge takes an exam of its own
2. Code graders read fixed boxes; LLM judges read the whole sheet. Stack them and they cover each other's blind spots
3. Publish the grader version alongside the score. Better graders change old scores.

---

P.S. The judge (judge.py) is in the repo with everything else → github.com/ramses203/llm-test-harness