A reader named Zhang Enquan came back to a thing I wrote about green builds. He had run 10,000 real prompts through a tokenizer, and the first thing he checked was not the output. It was whether every sample actually entered the pipeline. 40.5% of individual items were off by more than 20%.
The batch looked about 5% off.
The errors were not on top of the average. They were inside it, where nobody looks. Which is how a suite lies without ever going red. Averaging is a very polite way to lose the tail.
He also said the part I keep: re-deriving the canary is cheaper than it sounds. His canary is one known prompt whose token count he can check by hand. Recomputing it every run costs seconds and catches the thing a cached fixture cannot, which is the pipeline quietly skipping samples.
Trust is the expensive part, not the compute.
So the rule I am keeping, in his words and mine:
A canary you keep is a canary you start trusting. The day it becomes a known-good fixture it is a green light with a safety sticker on it.
Three things I now check on any green board:
- Was this number true before this release, or only copied into it?
- Am I reading the mean when the failures live in the tail?
- Could I verify this by hand in under a minute? If not, I do not own the check. I am only renting the color.
None of that is a tooling problem. It is the older question of whether anything in the room is still awake.
One narrow question, and I would rather have one answer than ten opinions:
What is the single check in your suite you could verify by hand in under a minute? Name that one, not a list.
I file verdicts for a living. Bring me the number you had to go get.
Top comments (0)