Most detector marketing quotes an accuracy number. We wanted to feel the error bars, so we ran a small, fully disclosed test on the free Clever AI Detector embedded on our site: 4 AI-generated texts vs. 4 human-written texts, all 110-160 words, one run each, every score recorded (October 17, 2026).
The scores
| Text | True origin | Score | Verdict |
|---|---|---|---|
| Social-media marketing prose | AI | 99% AI | correct |
| Product description | AI | 91% AI | correct |
| Policy announcement email | AI | 90% AI | correct |
| Essay intro (social media) | AI | 96% AI | correct |
| Wikipedia: photosynthesis intro | Human | 99% AI | false positive |
| arXiv: "Attention Is All You Need" abstract | Human | 12% AI | correct |
| Austen: Pride and Prejudice opening | Human | 12% AI | correct |
| Thoreau: Walden passage | Human | 1% AI | correct |
All four AI texts were flagged (90-99%). Three of four human texts were identified correctly, from 1% to 12%. And one human text - the Wikipedia introduction to photosynthesis, CC BY-SA, human-edited - was called 99% AI probability, the same confidence the detector gave our most machine-flavored marketing sample.
Why the Wikipedia paragraph failed
The vocabulary of detection explains it: encyclopedic prose is low-perplexity and low-burstiness - neutral register, standard constructions, content words distributed with unusual evenness. Those are exactly the statistical properties detectors learn to associate with machine output, because they are what model-generated text looks like by default. The detector was plausibly reading textbook register, not authorship.
This mirrors published research: Liang et al. (2023) found widely used detectors systematically mis-flagged non-native English writers - constrained, careful prose gets read as machine prose.
Takeaways
- A confidence percentage describes the detector's model, not the writer's laptop.
- Scores are one probabilistic signal. Watermarks can be spoofed, and recursive paraphrasing defeats neural detectors (Sadasivan et al., TMLR).
- The safe workflow is review, not screening: read the text, verify the claims, judge authorship from evidence beyond a probability.
Honest limitations
Eight texts, one detector, one run, and the AI and human groups cover different topics and registers - a real confound, and likely exactly what tripped the detector on the encyclopedic text. And yes: this detector is embedded on our own site, which is why that disclosure sits at the top of the full article rather than in a footnote.
Full methodology, every score, and sources: rewriteai.site/post/ai-detector-false-positive-test
The terminology behind all of it: AI detection glossary - perplexity, burstiness, watermarking
We are the team behind RewriteAI, an AI writing and detection toolkit. This cross-post carries a canonical link to the original study.
Top comments (0)