Replaying a practice test should not deal the same paper

Our practice games pull their items from pre-authored question banks. Each bank is a JSON array, ordered easiest first, and the engines rely on that order: question 1 should be gentler than question 20, because the real assessments ramp too, and a test that opens with its hardest item measures nerve rather than ability.

That ordering is why the first version of the loader was four lines. Fetch the bank, cache it, index by question number. Done.

It also meant that replaying a game dealt you the identical paper, in the identical order, forever. For a product whose entire value is practice, that is close to a bug.

The obvious fix breaks the thing that worked

Shuffle the bank. One Fisher-Yates pass and every replay is different.

And every replay now opens with whatever the shuffle put first, which half the time is one of the hardest items in the bank. You have traded a stale test for an unfair one.

So the requirement is actually two requirements that pull against each other:

1. a replay should show a different set of items, in a different order;
2. the items shown should still climb from easy to hard.

Group, shuffle inside the group, then sample

Every question carries a difficulty of 1, 2 or 3. The play order is built like this:

const tiers = new Map<number, T[]>();
for (const q of bank) {
  const d = getDifficulty(q);
  const arr = tiers.get(d);
  if (arr) arr.push(q);
  else tiers.set(d, [q]);
}
for (const arr of tiers.values()) shuffleInPlace(arr);

Shuffling inside a tier cannot disturb the curve, because every item in a tier is by definition interchangeable with respect to it. Concatenating the tiers in ascending order gives you a fresh permutation that still ramps.

That alone fixes ordering. The more useful half is the sampling.

The bank is bigger than the test

Several banks hold far more items than their game shows. asx-matrigma holds 135 items against a maximum of 40 displayed, which is deliberate: Assessio calibrated that adaptive pool from five parallel forms of the paper test, so one pool genuinely backing several sittings is fidelity rather than economy.

When the bank is larger than the shown count, you can do better than reordering. You can deal a different subset.

The naive version of that is to take the first N after shuffling, which reintroduces the original problem: a random 40 out of 135 will not have the difficulty mix the test is supposed to have. Some runs get seven hard items, some get twenty.

So the sample is proportional per tier:

for (const k of tierKeys) {
  const tier = tiers.get(k)!;
  const take = Math.round((tier.length / bank.length) * shownCount);
  picks.set(k, take);
  assigned += take;
}

A bank that is 40% easy, 40% medium, 20% hard produces a shown set that is 40/40/20, every time, drawn from different items every time. The candidate gets a genuinely fresh paper with the same shape.

The boring bug that lives in that line

Math.round on each tier independently does not sum to shownCount. Three tiers rounding up gives you 41 items in a 40-item test; three rounding down gives you 38 and the game runs out of questions before its clock does.

The drift correction is unglamorous and necessary:

while (assigned < shownCount) {
  const k = tierKeys.find((key) => picks.get(key)! < tiers.get(key)!.length);
  if (k === undefined) break;
  picks.set(k, picks.get(k)! + 1);
  assigned++;
}
while (assigned > shownCount) {
  const k = [...tierKeys].reverse().find((key) => picks.get(key)! > 0);
  if (k === undefined) break;
  picks.set(k, picks.get(k)! - 1);
  assigned--;
}

Note the asymmetry. When we are short, we add from the easiest tier that still has spare items, walking up. When we are over, we remove from the hardest tier that has anything to give, walking down. Both break on exhaustion so that a bank smaller than its own shown count cannot spin.

The direction is a product decision hiding in a rounding fix. Rounding error should never make a test harder than it was specified to be.

The unshown items are appended, not discarded

The function returns the whole bank, not the sampled front block:

return [...front, ...rest];

The first shownCount entries are the fresh subset; everything else follows behind it. This matters because engines differ in how many items they actually consume. Some stop at a fixed count, some run until a clock expires, and a couple branch and pull an extra item. Handing back a truncated array would turn "the candidate was fast today" into an index out of bounds.

Where the reshuffle is triggered

Once per game start, not once per load:

export function reshuffleQuestions(gameId: GameQuestionBankId, shownCount?: number): void {
  const bank = cache.get(gameId);
  if (!bank || bank.length === 0) return;
  playOrder.set(gameId, buildPlayOrder(bank, shownCount));
}

The bank itself is cached in memory for the session and the fetch is deduplicated through an in-flight map, so replaying a game costs zero network. Only the order is regenerated. Load once, deal many times.

loadQuestions also seeds a play order as soon as the fetch resolves, so an engine that reads before it reshuffles gets a valid ordering rather than null. Optional lifecycle calls that break things when skipped are not optional, they are a trap.

Try it

Pick a provider hub, for example https://cogniprep.app/games/criteria, and play the same test twice. The difficulty ramp is the same shape both times and the items are not the same items. That is the whole feature: two properties that sound like one, implemented separately.

The generalisable bit: when you randomise something users repeat, work out what the randomisation must preserve before you write the shuffle. Ours was a difficulty curve. Yours might be category coverage, or a tutorial always coming first, or never putting two questions about the same passage back to back. The shuffle is five lines. The invariant is the design.