Four Ways Your Multiple-Choice Quiz Gives Away the Answer

I build VibeLing, a vocabulary app, so all the examples here come from our own code and our own mistakes.

A multiple-choice question looks like the easiest thing in an app. You have the right answer, you grab three wrong ones, you shuffle, you render four buttons. I wrote that version in an afternoon.

Then I watched a recording of someone using it and realised they were answering correctly without reading the question. Not because they knew the words. Because the interface was telling them.

Here are the four leaks we found, in the order they embarrassed us.

1. The capital letter

Our distractors came out of a table of preset translations. The correct answer came from the word the user was actually studying. Two different sources, two different formatting conventions.

So a question would render like this:

Which word means "house"?

  Casa
  perro
  ventana
  libro

The correct answer was capitalised because it came from a place where entries were stored capitalised. The distractors were not. You do not need to speak Spanish to pass that test — you need to see which line looks different.

The bug, reconstructed on our current screen: four options, and only one of them starts with a capital letter. You can answer this without reading the sentence.

Underscores were the same story. Preset data had entries like hacer_la_cama, and a distractor that renders with underscores in it is not a distractor, it is a decoy nobody falls for.

The fix is to stop trusting the format of any option and impose your own, based on the prompt:

private function formatLikeOriginal(string $value, bool $sourceStartsWithUppercase): string
{
    $processed = str_replace('_', ' ', $value);

    if ($sourceStartsWithUppercase) {
        $processed = mb_strtoupper(mb_substr($processed, 0, 1)) . mb_strtolower(mb_substr($processed, 1));
    } else {
        $processed = mb_strtolower(mb_substr($processed, 0, 1)) . mb_substr($processed, 1);
    }

    return $processed;
}

The rule that came out of this: every option in a question must be normalised by the same function, including the correct one. If the correct answer takes a different code path than the distractors, sooner or later that difference becomes visible.

2. The correct answer, sneaking in as a distractor

The obvious guard is to exclude the right answer from the candidate pool. We had that. It compared strings directly, and it did not work.

The word list contained Casa and the correct answer was casa. Direct comparison says these are different, so Casa got picked as a wrong option, ran through the formatter from leak #1, and rendered as casa. The question now had the same answer twice.

That is worse than an easy question. There is no correct response, so a user who knows the word is punished for knowing it.

The same class of bug shows up between distractors: two entries that differ only by case or trailing whitespace become one visible option after formatting, and a four-option question silently becomes a three-option question.

Both are fixed by comparing on a normalised key rather than the display string:

$correctKey = $this->normalizeForCompare($correctTranslation);

foreach ($rows as $row) {
    $translation = trim((string) $row->to_word);
    $key         = $this->normalizeForCompare($translation);

    if ($key === '' || $key === $correctKey || isset($translations[$key])) {
        continue;
    }

    $translations[$key] = $translation;
}

normalizeForCompare is just a lowercase and a trim. The point is not the function, it is that the key you compare on and the string you render are two different things, and dedupe has to happen on the key.

3. The same wrong answers, over and over

This one only appears once you have a real session in front of you, which is why it survived the longest.

Pick three random distractors per question, ask twenty questions, and on a small pool the same handful of words keeps showing up in the wrong slot. Players do not consciously notice. They just start feeling that certain words are never the answer, and they stop reading those lines.

The obvious fix — never reuse a distractor — breaks immediately. Some of our language pairs have small shared dictionaries, so a hard exclusion runs out of candidates halfway through the session and the quiz simply stops.

What worked was making recency a preference rather than a rule. Candidates that have not been used recently are shuffled to the front; recently used ones go to the back of the same list instead of being dropped:

const ordered = [
  ...shuffle(candidates.filter((word) => !recent.has(word.key))),
  ...shuffle(candidates.filter((word) => recent.has(word.key))),
];

On a big pool you never reach the second group. On a tiny pool you do, and the quiz keeps working with slightly more repetition instead of failing. Degrading is a feature.

The window itself scales with what you have:

const recent = createRecentTracker(Math.floor(pool.length / 2));

A fixed window of, say, 20 is meaningless when the pool is 12 words. Half the pool means the rule adapts to whatever content the user actually has, which in our case varies enormously: the deck is built by the user, so one person arrives with thirty words and another with three hundred. That is the awkward part of any app where the flashcard deck is assembled by the learner rather than shipped with the product — you cannot tune anything against a known content size, so the rules have to be relative.

4. Position

The least interesting and most common one. If you build your options array as [correct, ...distractors] and render it in order, the answer is always first.

const options = shuffle([correct, ...distractors]);

Worth saying out loud anyway, because there is a subtler version: shuffling once at question-build time is right, but if any later step re-sorts the array — alphabetically for display, by length to fit a grid, by "relevance" — you have quietly reintroduced the bias. Sorting alphabetically is particularly nasty, because it correlates with nothing most of the time and then correlates perfectly with the answer on the questions where it happens to.

The one that has no clean fix

There is a fifth leak we have not solved: word length and shape.

If the prompt is a short, common word and two of your four options are long compound phrases, those two are gone as real candidates. A player who knows nothing filters on silhouette.

This one is not reconstructed — it is what the screen looks like today. Three single words and one two-word phrase. Before you know what any of them mean, the phrase already feels less likely. Fixing it properly means selecting distractors whose length and part of speech sit close to the correct answer — and every constraint you add shrinks the candidate pool, which pushes you straight back into leak #3.

Right now we accept it, because on our data the pool is the scarcer resource. But I would rather write that down as a known trade-off than pretend the quiz is clean.

The check that would have caught all of this

None of these were found by reading the code. They were found by watching someone play, and by one blunt test I would now write on day one:

Can you score above chance with the question hidden?

Cover the prompt and try to answer anyway. This is the whole test.

Render the four options, hide the prompt, and answer. If you can beat 25% — by case, by length, by "that word is always wrong", by position — the leak is in the options, and no amount of work on the question text will fix it.

That test is cheap, it needs no instrumentation, and it makes the failure obvious in about ten questions.

---

If you want to see what this ends up feeling like from the other side, the quiz described here is the one in our vocabulary app — and if you have hit a fifth leak I have not listed, I would genuinely like to hear it.