The setup was textbook. Four LLM judges on different base models. Double-blind pairs. Both presentation orders, to cancel position bias. Gold anchors seeded into the pool — samples where humans had already reached a verdict, including character-card copy a real user had flagged as "obviously AI" on sight, and community cards written by humans that had accumulated over a million conversations.
All the judges had to do was answer, for the anchors, "which of these two reads more human?" They scored 12%.
Failure one: not noisy — inverted
Twelve percent is the interesting kind of failure. A coin flip scores 50%; a noisy judge hovers around it. Scoring 12% means the panel was confidently wrong in a consistent direction: the copy humans called AI, judges praised as "grounded, textured, clearly human"; the human-written hits, judges dismissed as "formulaic, likely generated."
Failure two: agreement without accuracy
Pairwise agreement across the four judges was 86%. Before the experiment we'd planned to use majority voting for robustness. That plan died here: voting cancels independent random errors, but these errors weren't independent — the judges shared the same learned bias and converged on the same wrong answers. High consensus made the panel look more trustworthy while being reliably backwards. Separately, in about a third of the pairs, simply swapping presentation order flipped the verdict — position bias is a documented LLM-as-judge failure mode (the MT-Bench paper measured it systematically); we just met it again in our own data.
The bias, named. Reading the judges' rationales, two substitutions kept surfacing: they treated detail density as humanity, and ordinariness as fakeness. AI-polished copy — precise numbers, perfect structure, a hook per paragraph — read to them as "lived-in." Human copy — with its filler, lazy adjectives, and repetition — read as "generic." One detail stung: community-style stat-bar formatting in human cards was cited as "evidence of real community interaction"… and the same formatting in our imitation cards earned the same praise.
Failure three: prompting rescues one model family, not the rest
After the first round collapsed, we ran a dedicated calibration round — corpus expanded to 17 pairs, more heterogeneous judges — with an explicit debiasing preamble spelling out, among other things, that detail density is not evidence of humanity. A Claude-family judge jumped to 83% (the original four-judge setup had scored 12%): its bias lived at the prompt layer. DeepSeek, GLM and Qwen didn't move past 25–33% with the identical preamble; their prior sits deeper than instructions reach — they even cited the community stat-bar formatting our imitation cards copied as "evidence of real community interaction." Even the rescued judge only barely clears the calibration bar we now enforce, so "barely usable, single configuration" is the honest summary.
The rule we adopted
One sentence went into our engineering standards after this: an LLM judge that has not passed gold calibration (≥80% on human-consensus anchors) must not be used as an optimization target. Not for scoring, not for regression gates, not for leaderboards. Our current stack is three layers: a rule-based detector (phrase blacklists plus sentence-pattern quotas, calibrated against real user judgments — we once had to fix a rule that false-flagged human scenery writing, after which human samples scored clean) as the daily gate; human spot checks as ground truth; LLM blind review only for relative before/after comparisons of the same text.
And the honest boundary: we hold samples of carefully templated AI copy that dodge every pattern rule and sail past both the detector and the judges. The deepest layer of AI flavor lives in whole-text texture — uniform detail density, zero idle sentences, immaculate structure — and nothing automated currently catches it. What catches it is still a well-read human, reading aloud, frowning at the right moment.
Originally published on the Foreverse blog. We build Foreverse, an Android AI reading app; this calibration round is part of the public eval data behind the product's AI-flavor detection.
Top comments (0)