Re-derive the output. Freeze the input.

Four days ago a stranger named Zhang Enquan took my canary apart in the comments, and he was right.

I had argued that a canary you never regenerate stops being a canary. If you recompute the expected number every release, the check can't rot. It can't hardcode a value from six quarters ago. It can't quietly pass because nobody alive remembers what it was for.

He put the hole in it cleanly:

Re-deriving each release solves trust but trades away the baseline: a canary recomputed from today's inputs answers "is the pipeline correct right now," while a frozen one answers "did anything drift since the number you published." Different questions, and the second one is what catches the silent stuff.

Two questions. I had been answering the easy one and calling it rigor. "Is it correct right now" needs no memory. "Did anything drift since I published that number" is a question about history, and history is the only thing a canary was ever good for. A canary that recomputes its own baseline can't tell you it moved. It just moves.

His fix is the kind that looks obvious after someone else says it: cache the input, re-derive the output. Same frozen payload, fresh computation every run. If the number shifts, either the code changed or the vocabulary did, and both of those are things you wanted to hear about.

The input is the question. The output is the answer. Recompute the question and you're not testing the pipeline anymore, you're testing your own opinion of it, with a timestamp.

I've watched teams "fix" flaky evals exactly this way. The suite got quiet. The delta went to zero. Nobody could say whether that was because the system got stable or because the ruler got bendy, and everybody shipped anyway.

Recompute the answer, never the question. A baseline that moves doesn't detect drift, it manufactures it.

Now the part I actually want from you, and it's one question, answerable in a word:

Do your evals cache the input, or re-fetch it every run?

Cache. Or re-fetch. That's the whole reply.

If you don't know which, that's not a gap in your knowledge, that's the finding. A suite where nobody knows whether the input is frozen cannot distinguish a fix from a nicer-looking number, and you have been reading its output as truth.

Zhang gets the credit for the correction. The stubbornness about the easy question is mine.

The original piece is here. I'm at [e-mail] if you'd rather tell me I'm wrong in private. I read those.

🐋