Our scoring pipeline had one output, and for 31 tests it was the wrong one

CogniPrep scores simulated employer assessments. Every result used to go down one pipeline:

rawScore -> normalizeScore(populationStats) -> "you scored better than 68% of people"

That pipeline is correct, and it is well built. It is also a category error for 31 of our 165 tests, and nothing in the code knew the difference.

Two kinds of test wearing the same interface

An ability test has a right answer. Numerical reasoning, logical reasoning, a coding assessment: higher is better, and ranking you against a population is exactly what the instrument is for.

A trait instrument does not. A personality or strengths inventory is a set of bidirectional scales, reported per dimension against a norm group. "Prefers to work at a steady pace and take a cautious approach" at one end, "driven and seeks out new challenges, makes decisions quickly" at the other, with a marker somewhere between. There is no total. There is no good end. Which pole an employer wants depends entirely on the role.

One vendor's own peer reviewed audit says its games are "not meant to be won or lost". Another's trait report draws all eighteen dimensions as continua with written descriptions at both poles and no percentile anywhere.

Emit an ability percentile for one of those and you have done two bad things. You have told a candidate something false about the instrument, and you have trained them to "score high" on a test that has no high.

There is a third problem specific to some of these: several are ipsative, meaning each block is a forced ranking of options. The raw total is arithmetically identical for every candidate who completes it. You could not rank people on it even if you wanted to.

One file decides

export type ScoringKind = 'ability' | 'trait';

const TRAIT_GAMES: ReadonlySet<string> = new Set([
  'pm-balloon',
  'pm-hard-or-easy',
  'pm-money-exchange-1',
  'pm-money-exchange-2',
  'sv-personality',
  'balloon',
  // ...
]);

export function getScoringKind(gameId: string): ScoringKind {
  return TRAIT_GAMES.has(gameId) ? 'trait' : 'ability';
}

A set of ids and a lookup. The value is not in the code, it is in the fact that the decision exists in exactly one place and every consumer reads it from there, with the reasoning for each member recorded next to it.

The membership decisions are genuinely hard, which is why they are written down rather than being obvious:

A vendor is not a kind. One vendor's suite splits down the middle. Its cognitive games (digit span, arrows, a stop signal task, a tower puzzle) have correct answers and stay ability. Its social and risk games (a balloon risk task, a trust and fairness exchange) are bidirectional traits. Keying this by provider would have been wrong for half of them.

A situational judgement test looks like a questionnaire and is not one. It is presented as "what would you do", which reads like a preference survey, but it is keyed against an effectiveness standard. There genuinely is a better answer. It stays ability, and that is written into the file explicitly because the next person to read the list will assume otherwise.

The same construct under two brands is the same kind. Two different vendors ship a balloon risk task. They are the same bidirectional construct, so both are traits, regardless of how the two vendors describe their own products.

A future combined test is pre-decided. One vendor sells a blended assessment we have not built yet. The file says, in advance: if it is ever built, it must not go in this set. Its score would come from the ability sections only, with the personality sections surfaced as descriptive profile text and never folded into a percentile. Marking the whole thing a trait test would throw away a legitimate ability rank. Writing the decision down before the game exists costs one paragraph and saves the argument later.

What branches on it

The interesting thing about a distinction like this is how far it travels. Three places consume it, and the third is the one I would not have predicted.

The results screen:

const isTrait = scoringResult?.scoringKind === 'trait';

The feedback report:

isTrait: getScoringKind(data.session.gameId) === 'trait',

And a celebration toast:

// Trait games have no "best" - a higher score is not better - so they are
// excluded from the personal-best celebration.
if (saveResult.isPersonalBest && scoringKind !== 'trait') {
  notifySuccess('New personal best! 🎉', `You scored ${Math.round(rawScore)} on ${gameName}.`);
}

"New personal best" on a personality inventory is a small piece of UI and a complete misrepresentation of what the candidate just did. It says: this is a thing you can get better at, and you did. Neither half is true.

This is the general shape of these bugs. The main pipeline gets fixed because it is the thing you were thinking about. The confetti, the streak counter, the leaderboard, the "your best ever" line in the weekly email, those keep asserting the old model long after the main screen stopped. If a distinction is real, go and find every surface that implies its opposite.

Why not derive it

The obvious alternative is to let each test declare its own kind in the game library, next to its name and provider. We did not, for one reason: this is a claim about a real world instrument, not a configuration value. Each entry needs a justification that can be checked against the vendor's own documentation, and justifications belong together where someone auditing them can read them in one sitting.

The set is also the thing that gets audited when a vendor publishes something new about their instrument, and a single file is a much better target for that review than 165 scattered declarations.

Look at both kinds

Play an ability test and a trait test and compare the end screens. Every test on Watson Glaser and McKinsey Solve is an ability instrument, so you get a score and a percentile. All three Hogan inventories and three of the nine Thomas International instruments are traits, so you get a profile instead, with no rank and nothing to beat. SHL is the instructive one: seven ability tests and the OPQ, one suite, two pipelines.

Then go and look at whatever your product calls a score. Ask whether every input to it is measuring something where more is genuinely better. If some of them are not, the percentile is not a summary of your data. It is a claim about your data that your data does not support.