Perplexity, Burstiness, and AI Text Detection: An NLP Glossary

Wajahat Ali Shahani — Sat, 17 Oct 2026 20:12:06 +0000

What do "perplexity," "burstiness," and "watermarking" actually mean when people throw them around in AI-detection debates? We wrote a plain-English glossary of the core NLP vocabulary, grounded in the primary papers. Here is the short version of the terms that matter most, with why each one is in the conversation.

Perplexity

Perplexity measures how "surprised" a language model is by a text: the average probability it assigns to each next token, inverted. Low perplexity means the text reads as highly predictable to a model. Because model-generated text tends to be fluent and statistically bland, detectors and researchers often treat very low perplexity as a machine-likeness signal. The Google ML Glossary defines it as "a measurement of how well a probability model predicts a sample."

Caveat: careful, edited human prose - legal writing, textbook prose, some Wikipedia intros - also scores low on perplexity. Low perplexity is a register signal, not proof of authorship. (We saw exactly this failure mode in our 8-text detector test: a human Wikipedia paragraph scored 99% AI.)

Burstiness

Burstiness describes variation in sentence length and structure across a text - the rhythm of a human draft: some terse sentences, some long winding ones. Classical work by Benedetto, Caglioti, and Loreto (Physical Review Letters, 2002) showed that compressed-length features of human language vary in characteristic ways that simple generated text does not. Modern detectors use stylistic features in this family; when people say detectors look for "human rhythm," burstiness is usually what they mean.

Tokenization

Models do not read words; they read subword units produced by tokenizers (BPE, WordPiece). Detectors that operate on token probabilities inherit everything about how the tokenizer splits text - which is partly why detector behavior shifts across languages and domains.

Watermarking

Green et al. (2023) proposed embedding statistical "green lists" of plausible next tokens during generation, so watermarked text carries a detectable signature. Kirchenbauer et al. (arXiv:2301.10226, ICML 2023) extended this to bit-encoded payloads. Watermarking is a provenance technique, not a detector of human text - and its signals can be attacked: Sadasivan et al. showed paraphrasing (including recursive paraphrasing) can substantially reduce watermark detectability (arXiv:2303.11156, TMLR).

Membership inference & zero-shot detection

Zero-shot detectors (Liang et al., arXiv:2304.02819, Patterns) estimate the likelihood a token sequence came from a model, without detector-specific training. The same paper found widely used detectors disproportionately mis-flagged non-native English writers - constrained prose reads as machine prose.

GLTR & the original detector toolkit

GLTR (Gehrmann, Strobelt, Rush; ACL 2019 demo, arXiv:1906.04043) visualizes token-rank distributions to help humans spot machine text. It is a teaching tool as much as a detector, and its paper is a clean entry point into the whole feature family.

The honest takeaway

Every term above describes a statistical property of text, and every detector is a model built on such properties. Confidence percentages describe the model, not the writer's laptop. The practical workflow is review, not screening.

Full glossary with all citations: Perplexity, Burstiness, and AI Text Detection: an NLP glossary

The test data behind the caveats: 8 Texts, 1 Detector

Cross-posted from RewriteAI with a canonical link to the original.

We Ran 8 Texts Through an AI Detector, and a Human Paragraph Scored 99% AI

Wajahat Ali Shahani — Sat, 17 Oct 2026 14:40:09 +0000

Most detector marketing quotes an accuracy number. We wanted to feel the error bars, so we ran a small, fully disclosed test on the free Clever AI Detector embedded on our site: 4 AI-generated texts vs. 4 human-written texts, all 110-160 words, one run each, every score recorded (October 17, 2026).

The scores

Text	True origin	Score	Verdict
Social-media marketing prose	AI	99% AI	correct
Product description	AI	91% AI	correct
Policy announcement email	AI	90% AI	correct
Essay intro (social media)	AI	96% AI	correct
Wikipedia: photosynthesis intro	Human	99% AI	false positive
arXiv: "Attention Is All You Need" abstract	Human	12% AI	correct
Austen: Pride and Prejudice opening	Human	12% AI	correct
Thoreau: Walden passage	Human	1% AI	correct

All four AI texts were flagged (90-99%). Three of four human texts were identified correctly, from 1% to 12%. And one human text - the Wikipedia introduction to photosynthesis, CC BY-SA, human-edited - was called 99% AI probability, the same confidence the detector gave our most machine-flavored marketing sample.

Why the Wikipedia paragraph failed

The vocabulary of detection explains it: encyclopedic prose is low-perplexity and low-burstiness - neutral register, standard constructions, content words distributed with unusual evenness. Those are exactly the statistical properties detectors learn to associate with machine output, because they are what model-generated text looks like by default. The detector was plausibly reading textbook register, not authorship.

This mirrors published research: Liang et al. (2023) found widely used detectors systematically mis-flagged non-native English writers - constrained, careful prose gets read as machine prose.

Takeaways

A confidence percentage describes the detector's model, not the writer's laptop.
Scores are one probabilistic signal. Watermarks can be spoofed, and recursive paraphrasing defeats neural detectors (Sadasivan et al., TMLR).
The safe workflow is review, not screening: read the text, verify the claims, judge authorship from evidence beyond a probability.

Honest limitations

Eight texts, one detector, one run, and the AI and human groups cover different topics and registers - a real confound, and likely exactly what tripped the detector on the encyclopedic text. And yes: this detector is embedded on our own site, which is why that disclosure sits at the top of the full article rather than in a footnote.

Full methodology, every score, and sources: rewriteai.site/post/ai-detector-false-positive-test

The terminology behind all of it: AI detection glossary - perplexity, burstiness, watermarking

We are the team behind RewriteAI, an AI writing and detection toolkit. This cross-post carries a canonical link to the original study.

DEV Community: Wajahat Ali Shahani