Six practice exams from a thin question bank

I build practice exams for Dutch certification exams. Each exam has a fixed set of practice exams, "Oefenexamen 1" to "6", and each one follows the official format: one question per slot, where a slot is a topic with a set number of points.

For most exams that's easy. The hospitality licence (SVH) has 40 slots with 6 to 8 verified questions in each, so six forms that never share a question come out on their own.

The Klein Vaarbewijs 2, the Dutch boating licence for open and tidal water, was different. Its 27 slots hold between 0 and 8 verified questions. The GPS slot has 2. The tide calculation slot has 2. One slot has none. Here's how we still got six forms out of it, and why they're a JSON file in the repo instead of a random draw.

What the engine did before

The original form builder is simple. Per slot, shuffle the questions with a fixed seed and deal them out: form 1 gets the first, form 2 the second, and so on. First it checks that every slot has enough:

/**
 * Builds the fixed practice exams ("Oefenexamen 1..n"). Forms never share a question,
 * and the same pool + seed always gives the same forms, so a form number stays stable.
 */
export function assembleForms(exam: ExamDefinition, pool: PoolQuestion[], seed: number): ExamItem[][] {
  const count = exam.forms.fixedCount;
  const short = coverage(exam, pool).filter((c) => c.have < c.needForForms);
  if (short.length > 0) {
    throw new Error(`not enough questions for ${count} forms: ...`);
  }
  // ...seeded shuffle per slot, then deal
}

With the KVB2 bank that throws, and every form shows as "binnenkort" (coming soon). Waiting for 6 verified questions per slot would take weeks, and for some slots it wasn't going to happen at all. The GPS slot failed four drafting rounds in our checks: the new questions had cues that gave the answer away, or relied on inference.

The obvious fix, and why I didn't use it

Let questions repeat. Keep the runtime draw, add a counter, and allow each question in at most N forms. That's a small change to the function above.

The problem is what the runtime knows about a question:

export interface PoolQuestion {
  id: string;
  version: number;
  topicId: string;
  slot?: number;
  code?: string;
  key: AnswerKey;
}

No sources. And two of our rules need them:

• Source clashes. Two questions that rest on the same sentence of the same source don't go in one form. One tends to answer the other.
• Exclusive pairs. Some questions give each other away without sharing a sentence. KVB2 lists 8 such pairs. One example: a few slot 17 questions ask what ALAT or NAP is (two reference levels for water height), and a slot 18 question rests on knowing both. Put them in the same form and the second one is free.

A random draw with a max-uses counter keeps the counting rule and loses both of these.

Assembling the forms offline

So KVB2's forms are assembled from the content files, where every draft still has its sources, and written to content/forms/nl-kvb2.json as question codes. The exam definition reads that file, and the engine serves exactly those forms.

Each draft becomes a candidate:

export interface Candidate {
  id: string;
  slot: number;
  /** Answer letter; "" for a question without one (a sleepvraag, ...). */
  key: string;
  units: string[];
  sourceIds: string[];
}

units are the source sentences the draft rests on, found by matching its verbatim quote against the source text split into sentences and lines. Two candidates that share a unit clash.

The assembler fills one form at a time, thinnest slot first. For each slot it ranks the candidates: least used so far, then the answer letter this form has the fewest of, then a seeded random tie-break. Then it filters:

let fits = ranked.filter((c) => !clashes(c, [...chosen.values()], opts.caps, opts.exclusive));
let usable = fits.filter(mayUse);

mayUse is the use cap, with a per-slot override:

const cap = opts.maxUsesBySlot?.[c.slot] ?? opts.maxUsesPerQuestion;
return cap === undefined || (used.get(c.id) ?? 0) < cap;

After a form is filled, a repair pass swaps questions from the most common answer letter to a rarer one, as long as the swap breaks no rule. The whole thing runs 200 times with different seeds and keeps the best result: fewest unfilled slots first, then the best key balance, then the least reuse. It's deterministic, so the same bank always gives the same forms.

For KVB2 the call looks like this:

const assembly = assembleFixedForms(slots, candidates, {
  forms,
  keys,
  caps: kvb2Retirements.caps,
  exclusive: kvb2Retirements.exclusive.map((e) => e.ids),
  seed: KVB2_FORM_SEED,
  maxUsesPerQuestion: KVB2_MAX_USES, // 2
  maxUsesBySlot: KVB2_USES_BY_SLOT,  // { 18: 3, 23: 3 }
  omitted: (nlKvb2.forms.omittedSlots ?? []).map((o) => ({ slot: o.slot, reason: o.reason })),
});

Where the limits came from

Each limit is a decision we wrote down, with a reason.

At most 2 forms per question means a slot needs 3 verified questions to fill all six forms. That became the floor: below 3, the slot goes back to drafting.

At most 3 forms for slots 18 and 23, which got stuck at 2. Further drafting kept producing questions that failed the checks. Someone who does all six forms sees those questions three times. We decided that's better than leaving the slot empty or forcing in weaker questions.

One slot left out. Slot 21 is a course calculation with wind drift, worth 3 points. No public source states the rule, and three drafting rounds failed. So it's omitted by decision. Each form has 26 questions instead of 27, scoring is on 47 points instead of 50, and the pass mark becomes 33 (70 %, rounded up). The Klein Vaarbewijs 2 page says so before anyone buys. The assembler takes omitted as an option, so the slot isn't reported as a gap.

The result from the current bank:

• 6 forms × 26 questions = 156 places.
• 139 different questions fill them: 126 used once, 9 twice, 4 three times.
• Forms 1 and 2 share no questions. Form 6 shares 8 with earlier forms.
• Every form has 7 or 8 each of a, b and c as the right answer.

Here's the part that surprised me. The bank holds 156 eligible questions, exactly as many as there are places. It still can't make six forms without repeats, because slots with 7 or 8 questions have leftovers while slots 18 and 23 run dry. The total doesn't matter. The thinnest slot does.

For SVH the same assembler runs only as a report. It confirms the bank supports six forms without sharing a question, and it writes a drafting brief: which slots are short, how many new drafts they need, and which answer letters would let every form balance.

Forms must not drift

Fixed forms have a second job: staying put. "Oefenexamen 3" has to be the same exam tomorrow as today, or a retake tells you nothing. Two tests guard that.

The first pins the committed file to the assembly of the current bank:

const { assembly } = assembleKvb2Forms(kvb2Bank().candidates);
expect(nlKvb2.forms.fixed).toEqual(kvb2FixedForms(assembly));

If a question is retired or a new one passes the checks, the assembly can change, and then the test fails until someone regenerates the file. So the forms change only in a commit that says so, never as a side effect.

The second came with the engine change. To serve KVB2, the engine learned fixed assignments and omitted slots. Every other exam had to come out exactly as before. The test copies the old engine verbatim from the commit before the change and compares:

for (const seed of [20261020, 1, 7])
  expect(JSON.stringify(assembleForms(exam, p, seed))).toBe(JSON.stringify(legacyAssembleForms(exam, p, seed)));

Two more details help. An attempt stores its own list of questions when it starts, so an exam in progress keeps its questions even if a form changes. And if a fixed form names a question that isn't in the database, that form isn't offered at all. No gap, no substitute.

What I'd take from this

• If a rule needs data the runtime doesn't have, do the draw where the data is, and ship the result as a file.
• Count the thinnest slot, not the bank. A bank with enough questions in total can still fail.
• Make caps per slot, and write down why each one exists.
• Leaving a slot out is better than padding it. Score on the points that are there and tell people before they pay.
• Pin generated output with a test, so it changes only on purpose.

The tide slots are the thinnest part of this exam. If you're curious what they cover, there's a guide to tides for the Klein Vaarbewijs 2 (in Dutch). The roomier bank is Sociale Hygiëne.

I wrote this post with help from an AI assistant. The code, the numbers and the decisions come from our own repository.