Three of our six tests say "candidates report", the other three say "our estimate"

The UK train driver battery has a reporting model that is awkward for a practice app.

Each test is passed or failed on its own. RSSB recommends a national minimum for each, some train companies set a higher "enhanced" standard of their own, and there is no single enhanced standard. The industry standard that governs the process sets no pass marks, and none are published anywhere. The candidate gets an outcome and never a score, and every result goes on a national database that every train company can check.

So we built six of those tests, and every one of them ends on a screen that has to say something about how you did. Our scorers produce numbers. The real test produces a word.

One metric, three bands, and a sentence about where they came from

The compromise is a band. A sitting is classified as below the national standard, inside its range, or inside the range of an enhanced standard, stored as a single standardBand metric of 0, 1 or 2 so the feedback layer can read it without a new persisted field.

Every threshold behind those bands is ours. For three of the six tests, Group Bourdon and the two Trainability for Rules and Procedures tests, they are anchored on something real: candidates post their marks and their outcomes on railway forums, often with the year and the company. Each threshold in the module carries a comment naming the accounts it rests on, so a future edit has to argue with a source rather than with a number.

The feedback then says exactly that:

The real test gives you no score. You pass or fail it on its own, against either RSSB's recommended national minimum or a company's stricter enhanced standard, and every result goes on a national database that every train company checks. Pass marks are not published, so the band below comes from what candidates report passing with.

That sentence shipped on all six tests. Then the audit of the generated feedback asked the obvious question about the other three.

Nobody reports a mark for half of them

The WAFV, the TEA-Occ and the ATAVT are vigilance, attention and perception tests sat on a dedicated test system. Candidates write about them constantly. What they write is what the test felt like, how fast the tones came, how long the half hour was. Nobody posts a number, because the system does not show them one.

So for those three, "the band below comes from what candidates report passing with" was false. Not a hedge that was slightly too strong, and not a rounding of someone else's figure: a citation of evidence that does not exist.

The fix is a set and a second copy of every sentence that makes the claim:

/**
 * The tests whose bands rest on no candidate report at all. Their frame and
 * band sentence say the band is our estimate, because "candidates report"
 * would be untrue.
 */
export const TD_ESTIMATED_GAMES: ReadonlySet<string> = new Set(['td-wafv', 'td-tea-occ', 'td-atavt']);

export const TD_BAND_PHRASE: Record<TdBand, string> = {
  0: 'below what candidates report passing with',
  1: 'in the range candidates report for the national standard',
  2: 'in the range candidates report for an enhanced standard',
};

export const TD_BAND_PHRASE_ESTIMATED: Record<TdBand, string> = {
  0: 'below our estimate of the national standard',
  1: 'in our estimated range for the national standard',
  2: 'in our estimated range for an enhanced standard',
};

Not a boolean parameter threaded through the feedback functions, and not a conditional inside one string. Two complete sets of words, picked by id. The band logic, the thresholds and the stored metric did not change at all; only the provenance claim did.

The same substitution had to land in four places that each render it differently: the result feedback, the instruction screen each of those three games shows before you start, the provider hub page, and the article about the battery. A per-test detail line was wrong too, because it compared a count against "the range candidates report passing with" rather than against our own estimated range.

The test that enforces it in both directions

The interesting assertion is not that the three estimated tests say "our own estimate". It is that the other three must never say it:

it.each(IDS)('%s only credits candidate reports where some exist', (id) => {
  for (const level of ['weak', 'middling', 'strong'] as const) {
    const report = generateGameInsights(id, PERCENTILE[level], SITTINGS[id][level]).join(' ');
    const category = level === 'weak' ? 'low' : level === 'middling' ? 'average' : 'high';
    const feedback = generateClientFeedback('s', id, PERCENTILE[level], category, SITTINGS[id][level]);
    const text = [report, ...feedback.strengths, ...feedback.weaknesses].join(' ');
    if (TD_ESTIMATED_GAMES.has(id)) {
      expect(text, `${id} ${level}`).not.toMatch(/candidates report (passing|for)/);
      expect(text, `${id} ${level}`).toContain('our own estimate');
    } else {
      expect(text, `${id} ${level}`).not.toMatch(/our (own )?estimate/);
    }
  }
});

Both directions matter. A test that only checked for the weaker wording would pass if we gave up and called all six bands a guess, which would throw away three tests' worth of real evidence to make one rule simpler. The feedback is generated per sitting from synthetic weak, middling and strong states run through the real scorer, so the assertion covers the text a user actually sees rather than the template it came from.

See it

No account needed for either of these.

• Open the train driver hub and search the page for our own estimate. The sentence names the three tests it applies to, and the sentence before it says the other three come from dated candidate accounts. Both are in the same paragraph, under a heading that says our bands are not pass marks.
• Open the train driver cheating guide and search for candidates report passing with. The clause continues "or against our own estimate for the three tests where nobody reports a mark", which is the whole distinction in one sentence.

If you are counting occurrences in the HTML, every sentence on these pages appears twice, once in the markup and once in the serialised component payload. That is a different post.

A practice product is a chain of claims about somebody else's test. The thresholds are the easy part. Saying out loud which ones are inherited from evidence and which ones we made up is the part that changes what a user does with the number.