Partial credit for chained arithmetic, and the 120-line parser that does it

Some of our practice tests have chained questions. Question 2 asks for a figure derived from question 1's answer, question 3 builds on question 2, and so on down the set.

That structure is what makes them realistic, and it is also what makes naive marking useless. One arithmetic slip in question 1 poisons every downstream answer. Mark each item right or wrong and you tell a candidate they got four questions wrong, when what actually happened is that they made one mistake and then reasoned correctly from it for ten minutes.

Those are different problems with different fixes. A candidate who cannot do the arithmetic needs practice. A candidate who carried a bad number forward needs a habit, not practice. Feedback that cannot tell them apart is feedback that sends half its readers to the wrong remedy.

The data model: an answer knows how it relates to earlier answers

Each chained bank item carries the relationship between its own answer and the answers before it, as a small expression string:

{
  "questionId": 4,
  "dependsOn": [1, 2],
  "consistency": "{q1} / {q2} * 100",
  "correctAnswer": "43.2"
}

That one field is the whole trick, and it is worth noticing what it is not. It is not the derivation the candidate is supposed to perform (they are reading a chart, not being handed a formula). It is the invariant that must hold between answers if the reasoning was coherent, whether or not the inputs were right.

Score the same expression twice

const expectedFromOwn = evaluateExpression(trial.consistency, givenByQuestion);
if (
  given !== null &&
  expectedFromOwn !== null &&
  numericAnswerMatches(String(given), expectedFromOwn, CONSISTENCY_TOLERANCE)
) {
  consistent += 1;
}

Evaluate the expression twice: once substituting the keyed answers, once substituting the candidate's own earlier answers.

• Against the key, you learn whether the answer is right.
• Against their own earlier answers, you learn whether the answer follows from where they were.

The second is scored as its own metric, cascade_metric, and it is the most useful number the scorer produces. A candidate who slips once and stays coherent scores near 100 on it while scoring poorly on accuracy, and that pair of numbers is a diagnosis rather than a grade.

The tolerance is 2%, because candidates legitimately round intermediates.

One small guard that took a moment to get right:

const cascade_metric = chained === 0 ? accuracy_metric : share(consistent, chained);

With nothing chained in the set, there is no cascade to measure. Substituting a neutral 50 there would distort a short set in one direction or the other, so accuracy stands in instead. Neutral defaults are rarely neutral.

Detecting the rounding habit specifically

There is a third pass. When an answer is wrong, the scorer asks whether it would have been right if the previous step had been rounded:

if (!trial.correct && given !== null) {
  for (const places of [0, 1]) {
    const factor = 10 ** places;
    const rounded: Record<string, number> = {};
    for (const [key, value] of Object.entries(keyedByQuestion)) {
      rounded[key] = Math.round(value * factor) / factor;
    }
    const fromRounded = evaluateExpression(trial.consistency, rounded);
    if (fromRounded !== null && numericAnswerMatches(String(given), fromRounded, CONSISTENCY_TOLERANCE)) {
      roundingFlags += 1;
      break;
    }
  }
}

Whole numbers and one decimal place, because those are the two places a person actually rounds when they write an intermediate on paper instead of reusing the exact value.

The reason to detect this separately is that its fix is behavioural: reuse the logged result instead of retyping a rounded one. Telling someone to practise more percentages when their problem is a notepad habit wastes their evening.

No eval, and not because of style

Bank content is fetched JSON. Running fetched strings as code to perform long division would be an absurd risk for the convenience it buys, so consistency is parsed by a small evaluator that accepts exactly four things: digits and a decimal point, the four arithmetic operators, parentheses, and {placeholder} names.

The tokeniser is the interesting half, because it is where the safety lives:

if (ch === '{') {
  const end = expr.indexOf('}', i);
  if (end === -1) return null;
  const name = expr.slice(i + 1, end).trim();
  const value = vars[name];
  if (typeof value !== 'number' || !Number.isFinite(value)) return null;
  tokens.push({ kind: 'number', value });
  i = end + 1;
  continue;
}

A placeholder does not become an identifier to resolve later. It is looked up immediately and pushed as a number token, or the whole parse fails. By the time anything is evaluated there are no names left in the token stream, only numbers, operators and parens. There is no scope for an expression to reach, because there is no scope.

Everything returns null rather than throwing. A malformed expression in a bank is a content bug, and the right behaviour for a content bug at scoring time is for that one consistency check to be skipped, not for the candidate's whole result to fail to save.

The rest is a textbook precedence-climbing evaluator, and the whole module including comments is about 120 lines. Writing it took an afternoon, which is cheaper than the meeting you would otherwise have about whether new Function is acceptable just this once.

What this buys, in one sentence

The scorer reports accuracy, cascade consistency, rounding loss and reuse of logged results as four separate numbers, and a candidate reading them can tell whether their problem is arithmetic, coherence, or a notepad.

If your product grades chained work of any kind (multi-step maths, a form where later fields derive from earlier ones, a data pipeline test), the general move is the same: store the relationship between steps, not only the expected value of each step. Then you can evaluate it against what the user actually did, and stop counting one mistake four times.

There is more about the format this scorer serves at https://cogniprep.app/blogs/mckinsey-solve-what-the-invitation-length-tells-you, and the practice module itself is at https://cogniprep.app/games/mckinsey-solve. Get one early answer deliberately wrong, then stay consistent with it, and watch accuracy and cascade consistency come back as very different numbers.