Shortcut judges: a cheap test your preference pairs should pass before you train on them

Femi Ogunsanya — Sun, 15 Nov 2026 10:55:12 +0000

I'm building evaluation adapters (small reward models) for Telegraph, a network that routes AI requests to miners and scores their answers. Disclosure: I'm competing in Telegraph Hackathon Season II, which has prizes.

A preference pair is a prompt, a better answer and a worse one. A reward model trained on pairs learns whatever separates the two answers. If something cheaper than the skill you want separates them, it learns that instead, and your accuracy number measures the shortcut.

The test

Before training, run rules that pick an answer without checking anything:

pick the longer answer
pick the answer with more words in common with the prompt (TF-IDF cosine)
pick the answer with more of an obvious feature (alarm words, a verdict word)

If any of them is far above 50%, fix the pairs first.

Two cases from this week

Another builder (AdversarialEval, in the Telegraph Discord) found a keyword-count judge got every one of their severity pairs right, because all the worse answers were downgrades. After adding inflations, the TF-IDF judge still won ~95%. They rebuilt the pairs so the two answers are identical except for the severity claim, drew wrong severities from the same distribution as true ones (either direction), and balanced which answer is longer. On 84 new pairs: length 50%, TF-IDF 50%, keywords 55%.
My own 300 weather-verification pairs: length was 49.7% overall but 17–88% within error families, because the verdict phrases differ in length and some families lean one way. TF-IDF was 78% overall. Most of that is legitimate (in a wrong-station pair, or a digit swap of the question's own forecast, not matching the question is the error), but my fill-value and window pairs leak.

What I'd do differently

write the worse answer by copying the better one and changing only the claim
balance verdict direction and length within every error family, not just overall
report the shortcut judges next to model accuracy in every run

Script (standard library only) and the per-family table: https://github.com/fexx301/oathcast-telegraph-s2/tree/main/shortcut-judges-v1

dev.to allows up to four tags. I'd use machinelearning, ai, llm, datascience.

DEV Community: Femi Ogunsanya

Shortcut judges: a cheap test your preference pairs should pass before you train on them