DEV Community

John Green
John Green

Posted on

I Gave a Regex and an LLM the Same Exam. Fatal 3 vs Fatal 0.

#ai #llm #programming #testing

In the last post my comment classifier — a regex — sat a 15-question exam and produced three fatal errors: the kind a human can't undo. Under our rule, FATAL above zero means the tool can't be used. So what replaces it?

There's one candidate: let an LLM do the classifying. So I staged a head-to-head on the same exam.

The rules

  • Same 15 questions, same grader, same grade table
  • Regex side: the existing tool, untouched (keyword matching)
  • LLM side: Sonnet 5, given the classification definitions (what counts as a need; personal anecdotes and rhetorical questions don't) and asked to judge each comment

One fairness question came up. The LLM was given the right to answer "needs confirmation." The regex can't be given that right — the concept doesn't exist in it, which was exactly the defect the last post pointed out. So this isn't favoritism; it's the difference in ability between the two tools, as-is.

The result 

              regex                          LLM
clean         8/15 (53%)                     12/15 (80%)
grades        FATAL 3 RISKY 5 MISSED 1 H 0   FATAL 0 RISKY 1 MISSED 1 H 1
Enter fullscreen mode Exit fullscreen mode

Read by grade, not by score — the rule from the model-comparison post. FATAL 3 against FATAL 0. The regex is at a grade that can't ship; the LLM is at a grade that can. The match was decided there.

Three places where they split:

The LLM read context. The social-commentary comment — "…humanities majors don't pay, so nobody will replace them" — the regex filed as an errors-and-debugging need, because the Korean for "don't pay" shares two characters with its error keyword. The LLM correctly filtered it as commentary. Same for "Can't do this anymore, going to bed lol."

The LLM knows how to say it doesn't know. "Me too 😭 happens every time" is a reply that cannot be judged without its parent comment. The regex forced it into a category; the LLM answered "needs confirmation." It was given the right and actually used it. This tool can keep our principle — a wrong confirmation is worse than none.

The LLM needs no keyword dictionary. The regex's category dictionary had never registered "Cursor," the AI coding tool, so every comment about Cursor went unclassified. The LLM put them in the AI-tools category from context alone, no dictionary. The maintenance of forever adding keywords simply disappears.

Not a clean sweep. The LLM dropped one category (a comment about monthly payment never got the pricing/billing label) and confirmed one ambiguous comment it should have handed to a human — one RISKY. Eighty percent, not a hundred. But every one of its mistakes sat in a grade a human can undo.

"So why not just verify with an LLM?"

Read this far and the question writes itself. If the LLM is this good, why build exams and grade tables at all — why not just ask an LLM "is this right?"

Think of a scale. There is one way to know whether a scale is accurate: put a weight whose mass you already know on it. If a 1 kg calibration weight reads 1 kg, the scale can be trusted.

Here the scale is the tool that judges comments, and the calibration weight is the exam with the answers written in advance. This match had two scales — the regex scale and the LLM scale. Put the same comment on both and they read differently; the thing that settled which one was right was the calibration weight, the 15 questions with known answers. Without it, the story ends at "huh, they disagree."

"Just ask an LLM" is the same picture: the LLM being verified and the LLM doing the verifying are two scales weighing each other with no calibration weight in the room. When they disagree, you never find out who's right.

Turn it around and it's good news: with a calibration weight, one scale is enough. Two scales were only needed while choosing. Now that the match is over, what remains is the 15-question exam and the LLM classifier — that pair.

This week I watched "just let an LLM verify it" fail three times in person:

  • When the LLM judge and the code grader disagreed, the only reason I could rule on who was right was a 29-question exam with known answers
  • In the exam-author experiment, the author was Sonnet and the reviewer was Sonnet — 50 out of 50 passed. It was grading itself. Without an exam procedure you don't even notice that's a trap
  • Today's 80% and FATAL 0 — those numbers exist only because there was an exam with an answer key. Without it, all that remains is "it looks good." Which is the exact spot "I can't trust it" started from

And one practical reason: re-grading is free. Every time you change the classification prompt or swap the model, does a human re-verify everything? The exam has its answers stored — a new scorecard takes five minutes. The 15 questions written today became a lifetime exam this tool will retake every time it's touched.

Cost still counts

The regex is free and instant. The LLM takes tens of seconds per call. For small volumes — my own blog's comments — the LLM is fine. For a 20,000-comment batch, a regex as a first-pass filter with the LLM ruling only on what it flags is a workable combination. Choosing a tool is, in the end, a trade between grade (safety) and cost.

Takeaways

  1. When you switch tools, put them on the same exam. Choose by fatal count, not by score
  2. If a tool can say "I don't know," check whether it actually exercises that right
  3. Even if an LLM does your verifying, you still need the exam — it's the calibration weight for the thing doing the weighing

P.S. The 15-question exam and both classifiers' scorecards are public → github.com/ramses203/llm-test-harness, comment_exam.py (--compare shows them side by side)

Top comments (0)