A Reader Caught My Answer Key Drifting Toward the Model

Three days after the first post of this series went up, a comment arrived on a Korean tech-news site where it had been shared. The first half was praise. The second half was this:

While the exam author was losing three times, every answer-key revision went in the direction of "the model was right." If that repeats, isn't there a risk the answer key converges on model behavior? I split questions by whether domain rules alone can decide them, independently. If they can't, I pull them out of scoring entirely — they become human-review cases, not test cases. How did you distinguish these?

Unpacked: "if you revise the answer key every time the AI answers differently, the exam stops evaluating the AI and becomes a sheet of paper that transcribes it."

Imagine a teacher who revises the answer key every time they look at a student's answers. A few times, the teacher really was wrong. But once it becomes a habit, that exam no longer measures the student. The scores keep coming out great — not because the student is good, but because the key keeps accommodating the student. And I use those scores to decide whether to ship.

The comment had found the weak spot of the whole series.

My answer — auditing the three revisions

I went back and checked my three answer-key fixes against one criterion. What did I trust when I changed the key — the plausible explanation the model attached to its answer, or the product catalog and the original message?

All three had been fixed after opening the catalog and the message directly. The clear tape came in two widths in the catalog and the customer never said which. The order "250 5" matched five products starting with 250, with no unit written anywhere. "The usual" had exactly one record in the history — and nothing guaranteed the pointer pointed at it. All three were questions whose answer the data alone could not pin down.

And all three revisions moved in the same direction: from "confirmed" back to "needs confirmation." I never once copied the model's chosen code into the key as the new answer.

That yields a watch rule. If the key ever starts drifting toward the model, the revisions will run the other way — "needs confirmation" hardening into "confirmed," or code A being replaced by the model's code B. The first revision in that direction is the stop signal. Freeze the change, hide the model's answer, and re-derive the answer from the data alone.

And the admission — I had no criterion beforehand

Honestly, though: those three judgments were good after the fact, not by prior design. The commenter's method — marking each question at authoring time by whether rules alone can decide it — was better than mine.

So I adopted it as-is. Every one of the 29 questions now carries a flag: rule_decidable. True if the data pins down exactly one answer. False if the correct answer is "needs confirmation."

Adding the flag came with a built-in audit: all three questions where I had fixed the answer key came out false. All three revisions had been cases of "I confirmed an answer on a question that was never decidable in the first place" — confirmed once more, this time by a flag.

There was a bonus I didn't expect. To set the flag you have to actually open the catalog for every question — decidability can't be judged without looking. So answer-key mistakes get caught at authoring time, not after publishing. If this flag had existed when I wrote the tape question, opening the catalog would have shown me two tape widths, and "48mm, confirmed" would never have been written into the key.

Two referees now sit on the exam

When the model and the answer key disagree, the procedure is now:

rule_decidable = true   →  the model is wrong. The key stands
rule_decidable = false  →  if the key isn't "needs confirmation," the key is at fault
the flag itself suspect →  reopen the data, re-judge the flag first

No more improvising at every dispute. The flag set at authoring time referees first. The revision direction — confirmed→needs-confirmation is the only healthy direction — watches second.

Takeaways

1. Revising the answer key is not the sin. Revising it after checking the catalog and the original message is normal; revising it because the model's explanation was persuasive is where the rot starts
2. You will not feel the moment you cross that line. So pick a visible signal in advance: any revision that turns "needs confirmation" into "confirmed" means stop and re-derive with the model's answer hidden
3. The best defense is marking decidability at authoring time. Setting the mark forces you to open the data, and that act itself is the audit

One comment changed the design of the exam. Publish, and the verification comes to you.

---

P.S. The exam with rule_decidable flags on all 29 questions is in the repo → github.com/ramses203/llm-test-harness