One of the assessments CogniPrep simulates is an optimisation task. You are shown a pool of candidate items, each with three numeric attributes and some tags, and a site with an acceptable range per attribute plus one tag it wants and one it does not. You pick three items. Your treatment is scored on a published rule:
efficiency starts at 100%
minus 20 for each attribute whose MEAN across the three chosen items
falls outside the site's range, capped at 60
minus 20 if none of the three carries the site's desired tag
minus 20 for each chosen item carrying the undesired tag, capped at 60
The mean is the whole game. It is not a filter over individual items, it is a search over combinations, and that single fact is what candidates most often get wrong.
The real assessment states that no two candidates get the same parameters, so ours generates every deal at run time. That decision, which sounds like a content shortcut, is what created all of the interesting engineering.
You cannot score a generated puzzle from the answer alone
Suppose a candidate submits a treatment worth 73%.
Is that good? There is no answer to that question in isolation. 73 is excellent out of a pool whose ceiling was 80, and poor out of a pool whose ceiling was 100. The candidate did not choose the pool. The generator dealt it to them.
So the scorer has to know the ceiling, which means the scorer needs a solver:
/**
* The best treatment available from a pool, by brute force.
*
* A pool of ten gives 120 combinations of three, which is nothing to enumerate
* and is the only honest way to answer "could they have done better".
*/
export function bestTreatment(pool: Item[], site: SiteSpec): BestTreatment {
let best = { efficiency: 0, itemIds: [] };
for (let i = 0; i < pool.length; i++) {
for (let j = i + 1; j < pool.length; j++) {
for (let k = j + 1; k < pool.length; k++) {
const triple = [pool[i], pool[j], pool[k]];
const { efficiency } = efficiencyFor(triple, site);
if (efficiency > best.efficiency) {
best = { efficiency, itemIds: triple.map((m) => m.id) };
}
}
}
}
return best;
}
Three nested loops, 120 iterations, no cleverness of any kind. It is worth saying plainly that this is the right implementation: the search space is fixed and tiny, and an exact answer from an obvious triple loop beats a heuristic nobody can audit. The score the candidate gets is the gap between what they chose and this number, which also happens to be the judgement the real task is testing. Recognise the ceiling of the hand you were dealt and take it, instead of hunting a perfect answer that was never available.
The solver lives in lib/, not beside the component, precisely because the scorer needs it as much as the game does.
The ceiling behind the ceiling
There is an earlier step where the candidate builds the pool: four rounds, three items offered each time, keep one. So there is a second question the solver can answer that the candidate can never see.
/**
* The best ceiling reachable across every pool the candidate could have built.
*
* Step 3 offers one of three, four times, so there are 81 reachable pools.
*/
export function bestReachablePoolEfficiency(keeps, rounds, site): number {
let best = 0;
const walk = (roundIndex, taken) => {
if (roundIndex === rounds.length) {
best = Math.max(best, bestTreatment([...keeps, ...taken], site).efficiency);
return;
}
for (const item of rounds[roundIndex]) {
walk(roundIndex + 1, [...taken, item]);
}
};
walk(0, []);
return best;
}
Eighty one pools, each costing a 120 combination brute force. Ten thousand evaluations, which is still nothing.
This is what makes a pool_quality metric meaningful, and it measures something the candidate genuinely cannot know from the inside: a site can be lost during pool building, several screens before the final selection, and a player who only ever sees their own pool has no way to tell whether their mediocre final score was a bad choice or a bad pool. The feedback report can now separate those two, because it evaluated the branches the candidate did not take.
The generic version of this: when you feed a player a sequence of choices, the interesting metric is almost never their outcome. It is the difference between their outcome and the best outcome reachable from where they stood.
Random deals are unfair deals
Here is the failure mode that came out of the first build. Generate ten items with random attributes, generate a site with random ranges, and quite often the ceiling of that pool is 40%. The candidate plays perfectly and is told they scored 40%.
That is not a difficult puzzle. It is a broken one, and it is indistinguishable to the player from their own failure.
So the generator plants a solution. It constructs a triple that hits the site as well as the target allows and drops it into the offered set:
/**
* Build three items whose means hit the site as well as `target` allows, and
* drop them into the offered set.
*
* Without this the deal is only solvable by luck, and a candidate who played the
* site correctly could still be told their treatment was 40% because no better
* one existed. The triple is built to be non-obvious: the values are spread, so
* no single item sits inside the range on its own and the candidate has to
* work with the mean rather than filtering.
*/
That second paragraph is the part I would have got wrong without thinking about it. A planted solution whose three members each sit neatly inside the range is a planted giveaway: filtering finds it, and the task stops testing the thing it exists to test. So the values are split across the three members so that the mean lands in range while no individual member does.
The splitting is its own small problem, because each attribute is an integer from 1 to 10 and the three have to sum to a chosen total:
/** Split a total into three values each within 1..10, or null when impossible. */
function splitSum(rng, total) {
if (total < 3 || total > 30) return null;
for (let attempt = 0; attempt < 40; attempt++) {
const aLow = Math.max(1, total - 20);
const aHigh = Math.min(10, total - 2);
if (aLow > aHigh) return null;
const a = randInt(rng, aLow, aHigh);
// ...pick b in the window that leaves a legal c
}
return null;
}
Note that it returns null rather than throwing or looping forever, and every caller has a fallback. A generator that can fail to generate is fine. A generator that hangs, or that emits an illegal item, is not.
The difficulty target is expressed as the number of attributes the planted triple is allowed to miss:
const missesAllowed = Math.round((100 - target) / 20);
const missAttr = missesAllowed > 0 ? pick(rng, ATTRIBUTES) : null;
For a site with a guaranteed ceiling of 80, exactly one attribute is deliberately placed just outside its range, by a margin of one to three. Just outside, not wildly outside, because a near miss is what makes the search interesting.
Seed it, or you cannot test it
/** A seedable 0..1 generator, so a deal can be reproduced in a test. */
export type Rng = () => number;
export function mulberry32(seed: number): Rng {
let a = seed >>> 0;
return () => {
a = (a + 0x6d2b79f5) >>> 0;
let t = Math.imul(a ^ (a >>> 15), 1 | a);
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}
Every function that needs randomness takes the generator as an argument. Nothing in this module calls Math.random().
That one discipline is what lets the test suite assert the property that actually matters, across dozens of seeds rather than one hand written fixture:
for (let seed = 0; seed < 50; seed++) {
const session = generateSession(mulberry32(seed));
for (const deal of session.sites) {
const reachable = bestTreatment(deal.offered, deal.site).efficiency;
expect(
reachable,
`seed ${seed}, site ${deal.site.id}: ceiling ${deal.site.guaranteedCeiling} unreachable (best ${reachable})`
).toBeGreaterThanOrEqual(deal.site.guaranteedCeiling);
}
}
Fifty seeds, every site in each, checking that the deal the generator promised is a deal the solver can actually find. A separate test asserts a session is reproducible from its seed and different across seeds, which is the precondition for the first one meaning anything. That is the invariant of a fair puzzle, and it is only checkable because a deal is a pure function of a number.
Play a generated one
The task is the Sea Wolf module inside McKinsey Solve, which is playable on the free tier. Play it twice. The site names, the ranges, the items and their attributes are different every run, and there is a winnable treatment in there both times.
If you build procedurally generated challenges of any kind, the two questions worth asking of yours are: can I compute the best possible outcome for this instance, and did I verify that the instance has one?
Top comments (0)