I build practice exams for Dutch certification exams: workplace safety (VCA), the hospitality licence, two boating licences, marine radio and the taxi theory exam. The questions are written by a language model. Nobody reads every question before it goes live. I don't have the time, and I'm not a qualified skipper or safety instructor anyway.
That only works if the checking is better than "ask the model if it's sure". Here's the pipeline we ended up with, and what roughly 10,000 check results taught us.
The four steps
A. Draft from a source, with a quote. Every question starts from an official text: an article of the inland shipping regulations, a law, or the published exam requirements. The drafting model (mostly Claude Opus) writes the question, the options, an explanation, and a verbatim quote from the source that proves the answer.
B. A blind check by a second model. A fresh session on a different model (Claude Sonnet) gets the stem, the options and the source text, but not the key. It answers first and writes down why. Only then does it see our key and explanation, and it fills in six yes/no criteria: key correct, exactly one defensible answer, explanation correct, natural Dutch, unambiguous, at the level of the real exam. Any "no" is a fail.
C. One redraft. A failed question gets one redraft that addresses the findings, and the redraft goes through the checks again. It doesn't get a third round.
D. Independent verification by a third model. Same procedure again, blind first, by a model that neither wrote the question nor did step B (Claude Haiku or Fable). The instructions tell it to be stricter: if you'd hesitate to put this question in a paid exam, fail it.
A question is imported as verified only if both checks pass and the two checker models differ. That rule is in code, not in a prompt:
// Only counts when an independent model did it: the verifier must differ from the first checker.
const verified = !!v && v.model !== check.model && passesCheck(draft, v.result);
And the quote is checked by a test, not by a model:
expect(text, `${q.draftId}: quote not found verbatim in ${src.sourceId}`)
.toContain(normalize(src.quote));
If the drafter "remembers" a rule that isn't in the cited text, CI fails before any model looks at it.
What the numbers say
Across our exams, the checks have produced 5,148 step B results and 4,608 step D results. Those include re-checks of redrafts, so they count results, not unique questions.
- Step B failed 13.4 %.
- Step D failed 14.3 % of what step B had already passed.
I expected the third model to mostly agree with the second. It didn't. And look at why step D failed questions:
| Criterion that failed at step D | Times |
|---|---|
| Not at the level of the real exam | 216 |
| Explanation not correct | 153 |
| Not unambiguous | 140 |
| More than one defensible answer | 109 |
| Not natural Dutch | 25 |
| Key wrong | 12 |
A wrong key, the thing everyone worries about, was the rarest reason. Most failures were about a question that's technically right but not good: too easy, an explanation that cites the wrong reason, or a second option you could argue for. Those are exactly the problems a candidate would complain about, and the first checker had let them through.
Three things that went wrong, and the rule each one left behind
1. A script broke 62 questions after they were written. An early batch balanced which letter is correct by moving keys around after the explanations were written. The explanations still said "b is correct" while b was now a distractor. Step B caught it. The rule now: write options in display order, never re-letter afterwards, and re-read each draft to check that the option the explanation calls correct says what the explanation claims.
2. A "blind" check that wasn't blind. One step B batch opened a file that showed the keys before writing its own answers. Its results looked great and were worthless. The batch had to rely on step D for its blind pass, and the runbook now says it outright: don't run a script or open a file that shows the keys until every blind answer of your batch is written. A check that saw the key first is not blind.
3. All three models missed a pattern. In many questions the right answer was the one option worded differently from the others, so you could find it without reading the question. None of the models flagged it, because each one checks the content, not the shape. That one needed a mechanical check in code, which I wrote about in an earlier post.
What this doesn't guarantee
Three models agreeing is not the same as an expert agreeing. They share training data and can share a blind spot, as the odd-one-out pattern showed. The quote check proves the source says something, not that the question tests it fairly. So the site says plainly that no person reads every question, every question shows its source, and every question in the answer review has a button to report an error. The reports are the human review we do have.
If you're generating questions, quizzes or anything else that has a single right answer:
- Make the checker answer before it sees your answer. A model that reads your key first will usually agree with it.
- Use a different model for each check, and enforce that in code.
- Check the source mechanically. A verbatim quote plus a string match catches invented rules for free.
- Count why things fail, not just how many. The rare failure (a wrong key) wasn't where most of the work was.
The process is described for candidates on our about page (in Dutch), and the exams themselves are at examenzaal.nl.
I wrote this post with help from an AI assistant. The numbers come from our own check files and code.
Top comments (0)