There is one failure mode I have learned to fear more than a missing check.
A system with no check for something is at least honest about it. The gap is visible, the uncertainty is real, and everyone downstream knows not to lean on that part. A system with a wrong check for the same thing is worse, and it is worse in a specific, dangerous way. It answers with confidence. It overrules the signals that were actually right. And it does all of this wearing the one badge nobody thinks to question. It's deterministic, so it must be correct.
A wrong check with authority is worse than no check at all. No check leaves you honestly uncertain. A wrong authoritative check leaves you confidently wrong, and it silences the correct signal it overruled.
The two companion pieces to this one both push in the same direction: put as much of the system as you can onto deterministic ground, and let a rule decide wherever a rule can. I still believe every word of that. But both standards make a quiet assumption I want to drag into the light. When they say rules first, and especially when they say a deterministic rule may overrule a probabilistic judge, they assume the rule is right. This piece is about what happens when it is not, and how to make sure it is before you let it hold a veto.
The badge nobody checks
A deterministic number reads as ground truth. That is its great virtue and its hidden trap. When a metric prints 0 or 47 or false, it does not sound like an opinion, it sounds like a fact, and we grant it the standing of one. In a rules-first architecture we go further: we give the deterministic layer authority over the probabilistic one. The rule gates the output. The rule vetoes the judge. That is the correct design. A fluent model that hallucinates a verdict should not get to overrule a schema check that a parser can settle.
But look at what that authority does to the cost of an error. An advisory signal that is wrong is noise: you can ignore it. An authoritative signal that is wrong is not noise, it is a wrong answer with the power to enforce itself. And the badge of determinism, the quiet "it can't be wrong, it's just arithmetic," is exactly what stops anyone from auditing it. The check you trust the most, because it is deterministic and it is yours, is the one whose errors you are least equipped to catch. You built it to be believed.
A metric that lied
Here is the one that taught me the lesson.
Picture an evaluator for generated long-form documents, the kind a model drafts section by section. One of its quality checks looks for lazy repetition: a document that just restates the same paragraph a dozen times over is worse than one that actually develops. A reasonable thing to measure. The implementation folded each paragraph into a signature, a hash of its normalized text, and flagged the document when too many signatures came out identical.
The hash was cheap, and it was broken. It collided, so genuinely different paragraphs sometimes produced the same signature. And it was brittle to trivial edits, so the same paragraph with two words swapped or its sentences reordered hashed to something new. The net effect was a metric that missed near-duplicates when their surface form changed, and occasionally invented sameness where the content was genuinely different.
On its own, a buggy metric is just noise you could learn to ignore. But this one had been handed authority. When the model judge read the document and said, correctly, "these sections are near-duplicates of each other," the repetition metric was allowed to veto the complaint: "the signatures don't match, so the judge is imagining it." And the veto won. The score came out clean.
Sit with what that pipeline was doing on that facet. It was running its own logic backwards: using a wrong deterministic signal to overrule a correct subjective one, and then reporting a better score for the trouble. The judge had been right. The rule silenced it. And because the silencing came from a deterministic check, it arrived stamped with confidence, and every inflated score looked earned. Nobody goes back to re-litigate a number the arithmetic already settled. That is the trap closing.
The failure did not surface as an error. It surfaced as cleaner results, which is the most dangerous shape a failure can take, because clean results are what everyone was hoping for.
Why wrong-and-authoritative is the worst quadrant
Lay it out as a two-by-two: a check is either right or wrong, and it either carries authority or is merely advisory. Three of those cells are fine. A right check with authority is the whole point of the discipline. A right check that is only advisory is a mild waste. A wrong check that is only advisory is ignorable noise. It is the fourth cell, wrong and authoritative, that is not a lesser version of a good check but an active liability, and it is worse than the empty cell where no check exists at all.
Compare the two directly. With no repetition check, you know you cannot currently measure repetition. The gap is on the map, and a repetitive document simply passes unremarked, the same as anything else you haven't instrumented. With a wrong repetition check that vetoes the judge, a repetitive document passes and the one signal that correctly caught it gets overruled and the final score ticks upward, all under a badge that discourages anyone from doubting it. The missing check costs you a blind spot you know about. The wrong check costs you the correct answer you already had, plus the false confidence that you don't need to look.
Which yields the single most useful habit I took from this: suspect the metric first. When a deterministic check refutes an observation that a careful human or a competent model would make, the base rate is not on the check's side. A hand-rolled hash is far more likely to be broken than a fluent reader is to be hallucinating "these sections are near-duplicates" about a document where they visibly are. The reflex to trust the number because it is a number is precisely the reflex that keeps the wrong number in charge.
The other way a ruler lies
There is a second ruler that lies, and it lies in the very layer the first one was busy overruling. It is the bare absolute score: quality = 0.6, emitted by a model judge with no anchors, no comparison, no zero-point, and then handled as if it were a measurement.
A 0.6 like that is not a reading; it is a decimal in the costume of one. Ask what it is 0.6 of. Better than which example, worse than which, on a scale pinned to what? There is no answer underneath. Humans and models share a specific profile here: both are reliable at ordering ("A is closer to the brief than B") and unreliable at absolute calibration ("this deserves a 0.6"). A standalone perceptual number inherits all of the unreliable half and none of the reliable half. It is a subjective opinion that has been rounded to two decimal places and thereby laundered into looking objective, the same laundering the broken hash performed, run in the other layer.
The fix rhymes with the fix for the first ruler: do not trust the number until the number has earned trust. For a subjective axis, that means score by comparison, against anchors, a set of labelled reference examples that fix what good and excellent actually look like, and rank new outputs against them instead of emitting a free decimal. And before you trust a judge's agreement with humans, measure how well humans agree with each other. That inter-rater ceiling is the best score any judge can honestly aspire to. If two careful reviewers only agree seven times in ten on an axis, a judge that reports a confident number above that is not beating the humans, it is hiding the disagreement they honestly surfaced. Reporting a judge's number without that ceiling is one more measurement with no zero-point. A ruler with no marked zero and no fixed unit is not a strict ruler; it is a confident guess holding a straightedge.
How a check earns its authority
So the discipline here is not "add more checks." It is narrower and more demanding: before any check is allowed to gate output or overrule another signal, verify the check itself, as hard as the claim it will silence. In practice that is four moves.
Recompute it a second, independent way. Derive the same quantity by a different method and require the two to agree. If a repetition score can be computed by hashing and also by direct comparison, do both; a disagreement is a bug in one of them, and you want to find that in a test, not discover it as a mysteriously clean production score.
Regression-test it in both directions. A case you know is repetitive must trip it; a case you know is varied must not. A metric validated only in the direction it usually fires is half-tested, and the untested half is exactly where a false veto hides. And wrong does not only mean buggy. A check with correct arithmetic and a badly chosen threshold is wrong in the same way and hides in the same untested direction, so the cases you pick have to pin the boundary, not just the obvious middle.
Run it in shadow before it gates. Put it in production computing its verdicts, but let it change nothing. Watch it at scale, side by side with the signal it will eventually overrule. A wrong metric announces itself here, as a steady stream of vetoes against observations that were plainly correct, and it does so before it has moved a single real verdict. Shadow-first is not caution for its own sake; it is the one place a confidently-wrong check reveals itself while the damage is still zero.
Treat a veto as a claim, not a license. When a rule overrules a judge, it is asserting something checkable, "these sections are not actually duplicates," and that assertion has to be as verifiable as the judge's was. The fact-check does not get to be the one unaudited step in a pipeline whose entire purpose is to audit everything else.
There is an asymmetry worth naming here, because it is the whole reason this failure survives. We already do all four of these things for models. We stage a new model behind a flag, we hold a judge back until it agrees with humans, we assume the probabilistic thing needs proving before we lean on it. We almost never extend the same suspicion to a rule, because the rule wears the badge that says it does not need it. That asymmetry is the bug. The deterministic layer is handed the most authority and subjected to the least verification, and it is handed that authority precisely because nobody expects to have to check it.
Underneath all four is a habit good teams already apply to other people's numbers. When a result comes back too clean, too uniform, suspiciously convenient, you don't celebrate it. You go verify it against the source before you repeat it. This piece is that same skepticism, turned inward, and pointed most sharply at whatever metric you have handed a gate or a veto. External results earn scrutiny because they might be flattering. Your own authoritative checks deserve more, because they are flattering and you trust them.
Where a rough check is fine
Every standard needs its boundary, so here is this one's: the danger is authority, not error. A rough, sometimes-wrong signal is not merely tolerable, it is useful, as long as it stays advisory. A heuristic that flags documents for a human to glance at, a cheap metric that sorts a review queue, a smell test that nudges attention toward the likely problems: each can be wrong a fair fraction of the time and still pay for itself, because the cost of a wrong advisory signal is a wasted glance.
The line is crossed the moment a metric gains authority, the moment it can fail a build, block an output, or overrule another signal without appeal. At that instant its tolerance for being wrong drops toward zero, because its errors stop costing a glance and start costing the correct signal it now outranks. So the standard is not "verify every check to perfection," which would freeze you. It is "match the verification to the authority." Advisory signals can be rough and useful. Anything holding a veto has to have earned it, in both directions, in the open, before it ever changes a verdict.
A missing ruler tells you honestly that you cannot measure something yet. A wrong ruler tells you confidently that you can, and hands you the wrong number with a straight face. And if that ruler also has the authority to overrule the one instrument that was reading correctly, it will quietly make your whole system worse while every dashboard turns greener. Between the two, the honest gap is the safer place to stand. The only work worth doing from there is turning it into a ruler you have actually checked.
This piece extends two companions. The standard it builds on: Shrink the Stochastic Surface. The boundary it assumes: Determinism Where You Can, Judgement Where You Must. And the architecture both sit inside: Generative AI Builds Shapes, Not Games.
Top comments (0)