Our LLM Judges Called Human Writing "AI-Flavored" 88% of the Time

Foreverse — Sat, 25 Jul 2026 12:41:50 +0000

The setup was textbook. Four LLM judges on different base models. Double-blind pairs. Both presentation orders, to cancel position bias. Gold anchors seeded into the pool — samples where humans had already reached a verdict, including character-card copy a real user had flagged as "obviously AI" on sight, and community cards written by humans that had accumulated over a million conversations.

All the judges had to do was answer, for the anchors, "which of these two reads more human?" They scored 12%.

Failure one: not noisy — inverted

Twelve percent is the interesting kind of failure. A coin flip scores 50%; a noisy judge hovers around it. Scoring 12% means the panel was confidently wrong in a consistent direction: the copy humans called AI, judges praised as "grounded, textured, clearly human"; the human-written hits, judges dismissed as "formulaic, likely generated."

Failure two: agreement without accuracy

Pairwise agreement across the four judges was 86%. Before the experiment we'd planned to use majority voting for robustness. That plan died here: voting cancels independent random errors, but these errors weren't independent — the judges shared the same learned bias and converged on the same wrong answers. High consensus made the panel look more trustworthy while being reliably backwards. Separately, in about a third of the pairs, simply swapping presentation order flipped the verdict — position bias is a documented LLM-as-judge failure mode (the MT-Bench paper measured it systematically); we just met it again in our own data.

The bias, named. Reading the judges' rationales, two substitutions kept surfacing: they treated detail density as humanity, and ordinariness as fakeness. AI-polished copy — precise numbers, perfect structure, a hook per paragraph — read to them as "lived-in." Human copy — with its filler, lazy adjectives, and repetition — read as "generic." One detail stung: community-style stat-bar formatting in human cards was cited as "evidence of real community interaction"… and the same formatting in our imitation cards earned the same praise.

Failure three: prompting rescues one model family, not the rest

After the first round collapsed, we ran a dedicated calibration round — corpus expanded to 17 pairs, more heterogeneous judges — with an explicit debiasing preamble spelling out, among other things, that detail density is not evidence of humanity. A Claude-family judge jumped to 83% (the original four-judge setup had scored 12%): its bias lived at the prompt layer. DeepSeek, GLM and Qwen didn't move past 25–33% with the identical preamble; their prior sits deeper than instructions reach — they even cited the community stat-bar formatting our imitation cards copied as "evidence of real community interaction." Even the rescued judge only barely clears the calibration bar we now enforce, so "barely usable, single configuration" is the honest summary.

The rule we adopted

One sentence went into our engineering standards after this: an LLM judge that has not passed gold calibration (≥80% on human-consensus anchors) must not be used as an optimization target. Not for scoring, not for regression gates, not for leaderboards. Our current stack is three layers: a rule-based detector (phrase blacklists plus sentence-pattern quotas, calibrated against real user judgments — we once had to fix a rule that false-flagged human scenery writing, after which human samples scored clean) as the daily gate; human spot checks as ground truth; LLM blind review only for relative before/after comparisons of the same text.

And the honest boundary: we hold samples of carefully templated AI copy that dodge every pattern rule and sail past both the detector and the judges. The deepest layer of AI flavor lives in whole-text texture — uniform detail density, zero idle sentences, immaculate structure — and nothing automated currently catches it. What catches it is still a well-read human, reading aloud, frowning at the right moment.

Originally published on the Foreverse blog. We build Foreverse, an Android AI reading app; this calibration round is part of the public eval data behind the product's AI-flavor detection.

We Ran 360 Rounds of AI Novel Continuation. Models Fail Long Fiction in Exactly Three Ways

Foreverse — Sat, 25 Jul 2026 12:41:38 +0000

We have now run 360 rounds of novel continuation across nine models and two books — an 8.9M-character Chinese fantasy epic and the palace-intrigue classic Empresses in the Palace. Same protocol every time: continue from a fixed anchor, 20 consecutive rounds, each round's output appended back into a fixed 16k-token window, then double-blind review with two randomized letter mappings. Every long-run failure we observed fits one of three patterns.

Pattern one: the restart loop

The model ignores everything generated so far and restarts from the original text's ending — every single round. Gemini 3.1 Pro did this for 20 out of 20 rounds in the fantasy run: twenty phrasings of the same opening instant, a story that never takes its second step.

The root cause was ours, not the model's. Our context labels which passages are original prose and which are earlier AI continuations, and the explanation line said AI passages were "for plot continuity reference". Gemini read "reference" as "not canon" and skipped them all. Three ablations pinned it: old wording, 20/20 restarts; label with no explanation at all, still roughly 6 of 8 rounds restarting; explanation rewritten to state the passages are canonical events that must be treated exactly like original prose — zero restarts in 28 rounds across both books. One sentence of prompt wording was the entire difference between disqualification and mid-table.

Pattern two: the mid-run freeze

The model advances normally, then locks onto its own recent output and repeats it verbatim. Grok 4.5 froze in the middle windows of the fantasy run — one stretch repeats the same two paragraphs three times word for word — then partially recovered. On the palace novel it froze harder: both blind reviewers independently wrote "plot rewind" in their notes, and across twenty rounds the story clock never left the afternoon of the inciting incident. Kimi K2.6 showed a milder version, copying whole passages from its own earlier rounds.

This is the classic autoregressive fixed point: as the context fills with the model's own text, imitating itself becomes easier than advancing. It is also why one-shot writing benchmarks miss it entirely — the loop needs rounds to build up.

Pattern three: the ending rewind

The strangest one, and it hit the best prose mimic in the field. GPT-5.6 Terra's early and middle windows were, by both reviewers' judgment, the closest thing to the original author anyone produced — it even reproduced micro-habits like tilde-marked onomatopoeia and the author's trademark comma rhythm. Then in rounds 18 through 20 it rewound the entire plot back to the anchor point and replayed chapter one, reusing lines from its own round-2 output. Texture stayed perfect; the story folded back on itself.

The lesson we take from it: surface mimicry and long-range plot coherence are separate capabilities. A model can hold an author's voice flawlessly while losing track of where the story is. On the palace novel, for what it's worth, Terra never rewound — it took first-tier placement with a zero-incident run.

What survived twenty rounds

DeepSeek V4 Flash won the fantasy run outright — the only system both reviewers called "improving as it goes", still opening new plot arcs at round 20. Then it dropped to sixth-seventh on the palace novel, where its plain-spoken instincts fight the author's ornate register, and its sibling V4 Pro — criticized in fantasy for writing "like a finer-penned author" — took first place, because a finer pen is exactly what that book is written with. GLM 5.2 was the only model in the top three of both genres, with one persistent quirk: half-width quotation marks in Chinese prose, in every run, on both books.

No universal ranking survives contact with a second genre. If your app locks you to one model for a whole book, that is a real cost; per-paragraph model switching on continuation is not a luxury feature, it is how you route around each model's failure pattern before it locks in.

Originally published on the Foreverse blog. We build Foreverse, an Android reader that lets you continue any novel with AI — every direction becomes a branch, the original text stays intact. The eval data behind our model choices is public.

DEV Community: Foreverse