AI detectors are everywhere now — in classrooms, in hiring, and in freelance contracts that let clients reject work when a tool flags it. The pitch is simple: the detector can tell whether a human or a model wrote a text.
So we ran a small, fully disclosed test: four human-written texts and four model-written texts, all between roughly 110 and 160 words, through one detector.
One of the human texts came back 99% AI.
That text was the opening of Wikipedia's article on photosynthesis.
The setup
The detector: the Clever AI Detector — the same widget that powers the free detector on our own site. We didn't run this to promote it or to attack a competitor; we ran it on the tool our readers actually use, and we're reporting what it did, including the false positive.
The AI texts (4): a social-media marketing blurb, a product description, an internal policy email, and an opinion-essay intro. All generated by MiMo V2.6 — the model used in RewriteAI's editorial workflow — at default sampling settings, in a single unedited pass, specifically for this test. Disclosing that is the same policy we tell writers to follow.
The human texts (4): chosen to span different registers of real writing, all license-safe to quote:
- Wikipedia (CC BY-SA) — the photosynthesis intro
- arXiv — the "Attention Is All You Need" abstract (Vaswani et al., 2017)
- Jane Austen — Pride and Prejudice opening (Project Gutenberg, public domain)
- Henry David Thoreau — the "I went to the woods" passage from Walden (Project Gutenberg)
Nothing was edited, translated, or smoothed. Every text went in exactly as shown, within the widget's stated 80–3,000-word range. All scores below were recorded on October 17, 2026.
The results
| Text | True origin | Score | Correct? |
|---|---|---|---|
| Social-media marketing prose | AI | 99% AI | Yes |
| Product description | AI | 91% AI | Yes |
| Policy announcement email | AI | 90% AI | Yes |
| Essay intro (social media & teens) | AI | 96% AI | Yes |
| Wikipedia: photosynthesis intro | Human | 99% AI | No - false positive |
| arXiv: Transformer abstract | Human | 12% AI | Yes |
| Austen: Pride and Prejudice opening | Human | 12% AI | Yes |
| Thoreau: Walden passage | Human | 1% AI | Yes |
Two clean findings and one uncomfortable one:
- Every AI text was flagged — 90% to 99%. For ordinary, single-pass model output on everyday topics, the detector works as advertised.
- Three of four human texts were correctly identified — 1% to 12%, including two century-old literary passages.
- One human text was called 99% AI — with the same confidence the detector applied to our most obviously machine-flavored marketing sample. That is not a rounding error.
Why the Wikipedia paragraph fooled it
This is the part I find most interesting from a modeling standpoint: detectors don't inspect provenance. They model the statistics of the text — roughly, two properties:
- Perplexity — how "surprising" the word choices are
- Burstiness — how much sentence length and structure vary
Encyclopedic prose is engineered to be low-perplexity and low-burstiness: neutral register, standard constructions, terminology used in its most statistically expected sense, uniform sentence length, content words distributed with unusual evenness.
Those are precisely the properties a language model produces by default — and precisely what detectors learn to flag.
In other words, the detector likely wasn't detecting "AI authorship" so much as detecting "textbook register." Wikipedia shares its statistics with machine text. Austen and Thoreau don't. The arXiv abstract — human, dense, but written with a distinctive authorial hand — landed at 12%.
This is the same failure mode Liang et al. documented at scale when detectors systematically mis-flagged non-native English writers: careful, constrained prose reads as machine prose, and the writers least able to defend themselves absorb the penalty.
What this means if you build with (or against) these tools
- A score is never a verdict. A 99% attached to a human paragraph proves the percentage describes the detector's model, not your laptop.
- Register is the trap. Academic abstracts, encyclopedic entries, corporate documentation, and careful non-native English all sit statistically close to machine text. Expect noise there.
- Cross-test before believing. One detector, one opinion. 99% here and 20% there is worth exactly neither number — it's a reason to read the text instead.
- The safe workflow is review, not screening. Read the text, verify the claims, judge it. That is what Google's publisher guidance reaches too.
Eight texts is a tiny sample — this is an illustration, not a benchmark, and it cannot produce an accuracy rate. But it can show, concretely, the failure mode the research literature warns about.
Full write-up, methodology, and all eight scores: our full false-positive test on RewriteAI
Working with detector output in your own pipeline? RewriteAI also has RewriteAI's AI humanizer, built around the same premise: review the output, don't just trust a score.
Top comments (0)