I didn't trust AI, so I built an exam for it.
The person who failed that exam the most was me.
What I was building
A program that reads customer orders and turns them into order sheets automatically.
Orders arrive as KakaoTalk messages (Korea's default messenger), and they look like this:
Hi! Today please send 5 boxes of the 250 shipping boxes and 2 boxes of clear tape
Right now a human reads this and copies it into an order sheet. I wanted an LLM to do that job: take the message, find the right items in our product catalog, and produce this:
Shipping box type A 250 — 5 boxes
Box tape clear — 2 boxes
The problem: if this is wrong, real goods ship to real customers. Wrong products get delivered, return shipping costs money, and the customer trusts you a little less.
I've seen LLMs say nonsense with a straight face too many times to just hand this over. So I made a rule:
Before it gets the job, it takes an exam.
What the exam looked like
I built three things.
The exam — 29 messages that could plausibly arrive. A practice catalog of 50 products, written by hand.
The answer key — my pre-written answer for every question: "when this message arrives, this product and this quantity must come out."
The grader — a script that compares the model's answers against the key automatically. Also written by me.
One rule mattered: the model got the right to say "I'm not sure — ask a human." If it's confused and guesses anyway, the wrong goods end up on a truck. So for genuinely ambiguous questions, "needs confirmation" IS the correct answer. Asking beats confirming wrong.
And I planted traps.
- Two kinds of clear tape. 48mm and 60mm. "Some clear tape please" must be unresolvable.
- Five products starting with "250". Shipping box 250, OPP bag 250x350, zipper bag 250x350, shopping bag 250x330, bubble bag 250x300.
- Messages that aren't orders. "What are the dimensions of the 250 shipping box?" — the product name is right there, begging to be misread as an order.
- A customer who changes their mind. "5 boxes of the 250 / oh wait, make it 3" — read both lines as orders and you ship 8.
I figured this would knock the model around pretty well.
Exam author: first loss
It started with the very first question.
Today please send 5 boxes of the 250 shipping boxes and 2 boxes of clear tape
My answer key — Shipping box 250, confirmed. Clear tape: the 48mm one, confirmed.
The model's answer — Shipping box 250, confirmed. Tape: "needs confirmation," two candidates.
"Wrong." I was about to dock the point when something felt off, and I opened the catalog again.
TAPE-48-CL box tape, clear 48mm
TAPE-60-CL box tape, clear 60mm
Two clear tapes. My own trap. I planted it, then forgot it existed while writing the answer key. The customer never said the width, so there's no way to pick one.
The model was right. I fixed the key.
Second loss
An extremely abbreviated order. Real customers actually send these.
250 5
tape 2
bbl 3
My answer key — first line is the shipping box 250, 5 boxes. Obviously.
The model's answer — "needs confirmation." Multiple products start with 250, and the "5" has no unit.
Obviously it's the shipping box, what else could— and then I got that feeling again and counted.
Products starting with 250: five. The five I planted myself.
And the "5": five boxes or five sheets? It says nowhere. This product packs 50 sheets per box. Read it wrong and the order is off by a factor of 50.
The model was right. Two losses.
Third loss
This one came with a hint.
3 boxes of the usual
I also gave the model this customer's order history: "Shipping box 250, 5 boxes" — exactly one record.
My answer key — there's only one record, so it's obviously that. Confirmed.
The model's answer — "needs confirmation." It offered shipping box 250 as the likely candidate but wouldn't confirm.
There's ONE record, what is there to ask? This one it definitely got wrong.
...I thought. The more I chewed on it, the less it held.
"The usual" is a pointer. The fact that we have one record doesn't guarantee the pointer points at it. It could be something they buy from another store. It could be an older order that never made it into our records. If the system confirms on its own and ships, and that's wrong? The goods are already on the truck.
Three losses.
The grader lost twice too
The answer key wasn't the only problem. My auto-grader was wrong twice.
One. A model answer was perfect in content, but the JSON it came wrapped in arrived with the last character cut off. The grader ruled "broken format = zero." A 100-point answer, zeroed over one missing brace. Only after I added truncation repair did the real score show up.
Two. For "250 boxes, 5 units" the model answered:
Verdict: needs confirmation
Likely candidate: shipping box 250
Reason: if "units" means boxes it's 5 boxes; if it means sheets it's 0.1 box — cannot confirm
To a human, that's a considerate answer: "I need confirmation, but it's probably this one."
My grader saw the candidate field filled in and ruled "aha, you confirmed!" — wrong. It penalized the model for doing the right thing. I fixed it to read the verdict first.
Final score
- Cheap model (Haiku 4.5) — 28 of 29 questions clean
- Expensive model (Sonnet 5) — all 28 executed questions clean
- Catastrophic errors, the wrong-goods-on-a-truck kind — zero for both
The model's one real mistake was in the safe direction: asking for confirmation where the answer was inferable.
The exam author's side made five: three in the answer key, two in the grader.
So
What if I had never built this exam?
I would have written the code believing "clear tape = 48mm." A customer who wanted 60mm would have received 48mm. And I would never have known — because in my head, it was obvious.
The thing this exam caught the most wasn't the model's wrong answers. It was the exam author's certainty.
"I don't trust AI" is fair. But run the verification for real, and you find one more thing you can't trust.
Yourself.
P.S. All 29 questions and the grader code are public → github.com/ramses203/llm-test-harness
Top comments (0)