Nine LLM reviewers score every article I publish. The gate is simple: zero blockers, and a mean score above my floor.
Yesterday it failed an article it had scored 9.1. Then it failed a second one three separate times. Neither reviewer ever disagreed with the writing — all three failures were the harness misreading its own reviewers.
A perfect score, filed as a blocker
The Checklist reviewer returned 10.0 and wrote this under BLOCKERS:
no biography/family/military/location/heritage; no political position;
the numbers are incident/operational facts, so the rule doesn't bite.
That is a clean bill of health. It failed the gate.
The parser dropped lines meaning "nothing here" with /^none/i. The prompt asks reviewers to explain why nothing is wrong, so they write the explanation instead of the word — and "no biography" is not "none".
Why it survives review: the regex is correct for the string it was written against. Nobody writes a test for the sentence a model didn't emit.
The fix wasn't a longer regex. Guessing every phrasing of "nothing is wrong" is unbounded. The prompts already state the contract — fully in scope means BLOCKERS: None and SCORE: 10 — so a 10 that also reports a blocker is self-contradictory:
if (score === 10 && blockers.length) {
improvements.push(...blockers); // demote, never drop
blockers = [];
}
Demoted, not deleted. If a reviewer ever scores 10 on something genuinely broken, the text still reaches a human.
The word that made good reviews look like outages
My runner treats a quota message as a transport failure, so a billing error can never be scored as a bad article. The pattern included a bare rate limit.
It was tested against the reviewer's entire output. My corpus is security articles. Reviewers say "rate limit" constantly:
· Discovery & Hook: USAGE LIMIT — SCORE: 8.0
· Voice & Agenda: USAGE LIMIT — SCORE: 9.5
· Quality: USAGE LIMIT — SCORE: 9.0
Three finished reviews, scores visible in the text that got thrown away. That article lost 4 of 9 reviewers and failed two batch runs before I looked. It reads as an account problem, which is exactly why it survived.
The tell separates the two cleanly: a real quota banner is the only thing on stdout. A review that merely discusses limits has a SCORE: line sitting right there.
An excluded reviewer voting anyway
One reviewer is informational — its rubric weights brand fit 40%, which structurally caps anything outside that niche. It was excluded from the gate score.
It was not excluded from blockers. So it vetoed through the back door — and what it files under BLOCKERS is its own arithmetic:
✗ Axis 1 — Content Quality: 8.5
✗ Weighted = 8.5 x 0.6 + 5 x 0.4 = 7.1
Four "blockers" on an article it had just called original and reproducible. A half-applied exclusion is not an exclusion — and a weighted composite is exactly the shape that hides one, because the arithmetic looks like a finding.
The pattern
Every one of these is the harness misreading agreement as disagreement, and all three fail in the same direction: quietly, toward rejection. A false pass is loud — something bad ships and you see it. A false fail looks exactly like a strict gate doing its job, so it can run for weeks while you assume the work is merely not good enough yet. It is the same asymmetry that makes a leaderboard wrong in the flattering direction, and the same reason measurement bias survives longest when it agrees with what you expected.
If you run an LLM as a judge, assert on the disagreements it cannot logically have: a perfect score with a blocker, a reviewer excluded from the score that still blocks, a transport error carrying a parsed result. Those are contradictions, and contradictions are testable without predicting a single word the model will say.
assert.equal(parse("SCORE: 10\nBLOCKERS:\n- none found here").blockers.length, 0);
assert.equal(parse("SCORE: 7\nBLOCKERS:\n- the claim has no date").blockers.length, 1);
Both directions. A one-sided test would have passed on all three bugs — my earlier fix for the opposite failure is what introduced the second one. That is the same lesson ground truth taught me that unit tests could not: a test written from the failure you already know about only ever proves you fixed that one.
More on the tooling behind this at github.com/ofri-peretz/eslint. The three defects above were found on 2026-08-11 across 39 gated articles.
What's the last false negative you found in your own tooling — and how long had it been running?
Top comments (0)