Two questions that ask the same thing: how we catch near-twins before they reach a practice exam

Our question bank is written by AI models, in batches, one exam slot at a time. A slot is a topic with a set number of points in an exam. If you ask a model for ten questions on the same rule, you get ten questions on the same rule. Some of them are the same question in different clothes.

For a practice exam that's a real problem. A candidate who sees two near-twins in one sitting gets the second one for free, and the form teaches less than it looks like. This post is about the three places we catch them (import, form assembly and a leak test for free questions), and the one place we deliberately don't look.

1. At import: trigram similarity per slot

When a draft has passed its checks, the importer writes it to the database. Before the insert, it asks Postgres whether the same slot already holds a question with a very similar stem:

select q.code, extensions.similarity(v.stem_text, $1)::float8 as similarity
  from questions q join question_versions v on v.question_id = q.id
 where q.exam_id = $2 and q.slot is not distinct from $3 and q.status <> 'retired'
   and extensions.similarity(v.stem_text, $1) >= $4
 order by similarity desc limit 1

The threshold is 0.8. similarity is pg_trgm's: lower-case the stem, split it into words of letters and digits, pad each word with two spaces in front and one behind, cut it into three-character pieces, and compare the two sets as shared / all.

We also have the same function in TypeScript, so tests can use it without a database:

export function trigrams(text: string): Set<string> {
  const out = new Set<string>();
  for (const word of text.toLowerCase().match(/[\p{L}\p{N}]+/gu) ?? []) {
    const padded = `  ${word} `;
    for (let i = 0; i + 3 <= padded.length; i++) out.add(padded.slice(i, i + 3));
  }
  return out;
}

Two details matter more than the threshold.

The check is per slot. Many stems across different slots start the same way ("You see this sign. What does it mean?"). Comparing within a slot keeps the check to questions that compete for the same place in an exam.

Order decides who wins. The importer lists questions that passed the second, independent verification before those that only passed the first check. So when a verified question and an unverified one clash, the verified one is the one that goes in. The unverified twin is reported as a near-duplicate and skipped.

The import is also idempotent: a draft that is already in the database is counted as imported, not compared with itself. A test inserts one question, offers a second with almost the same stem, and expects exactly one insert and one reported near-duplicate.

2. Where the stem is a frame, skip the check

Stem similarity breaks down when the stem says nothing. A drag question that starts "Drag the right meaning to each symbol" has the same stem for every symbol. A chart task starts "You sail from … to …". Those would all look like twins of each other, and none of them is.

So the importer skips the similarity query for those two types. It also skips it for any question that a fixed practice exam names by code, which covers frames like a matrix question's "State for each statement whether it is true". The fixed forms are already put together to keep near-twins apart, and a form whose question is missing from the database would show up as unavailable, which is worse than a lookalike.

That leaves the duplicates among those questions to something else.

3. In the form assembler: pairs and shared sentences

Offline, when the fixed practice exams are put together, two rules do the work that stem similarity can't. Both are in the form assembler.

• Shared source sentence. Each draft records the verbatim quote it rests on. The assembler finds that quote in the source text, split into sentences and lines. By default, two drafts that rest on the same unit don't go in the same form (definitions cited only as a premise don't count).
• Exclusive pairs. Some questions give each other away without sharing a sentence. A question that asks what one of two reference levels means, and another that only works if you know both. These are listed explicitly per exam in a retirements file. Three of the Klein Vaarbewijs 2 pairs tie one slot-18 question to three different slot-17 questions.

(I wrote about the assembler in more detail in an earlier post.)

4. Free quizzes must not leak paid questions

The same problem shows up in another direction. Some topic guides carry a few free questions. They must never be copies of the questions in the paid practice exams, or the free page hands out the answers.

A test compares every guide question that hasn't failed a check with every single-choice question drafted for the paid Klein Vaarbewijs 1 practice exams. A pair counts as a leak only when three things hold:

similarity(free.stem, paid.stem) >= 0.6 &&
  sameVisual(free, paid) &&
  similarity(key(free), key(paid)) >= 0.4

The stem threshold is lower than at import (0.6, not 0.8) because the test also looks at the picture and the correct answer. Similar stems alone are common boilerplate, so a stem match without the same picture and a similar answer doesn't count.

What I'd take from this

• One similarity score is not a duplicate detector. Use it where the stem carries the meaning, and say in the code where it doesn't.
• Compare inside the competition set, here the slot, not the whole bank.
• Let the stronger question win, not the first one. Sort so the better-checked item is inserted first.
• Add conditions to cut false alarms (same picture, similar answer) instead of raising a threshold until nothing fires.
• Keep an explicit list for what a score can't see. Our exclusive pairs are eight short JSON entries for one exam, and they catch what no trigram would.

If you want to see where the questions end up, the exam overview lists every exam, and how we check the questions (in Dutch) describes the checking steps.

I wrote this post with help from an AI assistant. The code, the thresholds and the decisions come from our own repository.