You paste a paragraph into an AI detector. A little spinner does its thing for maybe two seconds, and then you get a verdict: "98% likely AI-generated."
What actually happened inside that black box?
Most people treat these tools like they're running some kind of deep neural wizardry that sniffs out machine prose through sheer intelligence. But the two numbers doing the heavy lifting, perplexity and burstiness, are not mysterious at all. They are borrowed directly from statistical natural language processing, and both have been around since long before anyone worried about ChatGPT flooding the internet with synthetic text.
I spent a few weeks recently digging through research papers, running text samples against five different detection APIs, and talking to people who actually build these systems. The takeaway was not what I expected. A lot of what we assume about how AI detection works is either oversimplified or just wrong. So here is what I found.
Where the Numbers Come From
Perplexity did not start its life as an AI detection metric. NLP researchers have been using it for decades to evaluate how well a language model predicts text. The core question is dead simple: given a sequence of words, how surprised would a particular model be by each next word?
Imagine I type "The cat sat on the." A decent language model will assign a very high probability to "mat" as the next token. It will give a moderate probability to "floor." It will assign a near-zero probability to "watermelon." The less expected the actual continuation, the higher the perplexity score.
At some point researchers noticed a pattern: human writing consistently produces higher perplexity than machine generated text. The logic behind this is almost too obvious. Language models are trained to predict the most probable next token. They optimize for low perplexity by design. Humans do not think in probability distributions. We make weird leaps. We combine words in unexpected ways. We get distracted mid sentence and never finish the thought.
Burstiness came later. It measures how much sentence structure and length vary across a passage. Human writing tends to bounce around. A long, meandering sentence full of subordinate clauses will sit right next to a two word fragment. Then another sprawling sentence that runs on for thirty words before finding its period. This is normal human rhythm. AI text, especially from earlier models, tends to settle into a consistent cadence. Sentences hover around similar lengths. Paragraphs follow the same structural template. The rhythm is almost too clean.
So the theory goes: high perplexity plus high burstiness equals human. Low on both equals machine. Simple enough, until you actually start testing it.
The Calculation Is Trickier Than It Looks
Here is the part most explainer articles skip. Different detectors calculate these metrics in different ways, and that is why the same text can return wildly different scores depending on which tool you use.
Some detectors measure perplexity against GPT-2. Others use their own internally fine-tuned models. A few run the text through multiple reference models and average the results. The specific model you choose as a baseline matters enormously because perplexity is always relative. A paragraph that looks "surprising" to GPT-2 might register as completely ordinary to Claude. There is no absolute perplexity score floating out there in the universe waiting to be discovered. It is always a comparison.
The math goes roughly like this. First, the detector tokenizes your text. Then, for each position in the sequence, it asks the reference model: what probability would you have assigned to this specific token given everything that came before it? It computes cross-entropy across all positions, exponentiates, and out comes the perplexity number.
Lower perplexity means the text tracks closely with what the model would have predicted, which suggests machine generation. Higher perplexity means more unpredictability, which suggests a human wrote it.
But here is the problem that does not get talked about enough. Perplexity is deeply content dependent. Technical documentation, legal contracts, and academic writing naturally produce lower perplexity scores than creative fiction or casual conversation regardless of who or what wrote them. A human drafted terms of service agreement might clock a lower perplexity than an AI generated poem. The metric confuses "predictable structure" with "non human origin," and those two things are very much not the same.
Burstiness has a parallel blind spot. Say you are writing a technical tutorial. You explain a concept, show a code example, explain the next concept. Your sentences probably follow a fairly predictable rhythm because that is what makes a tutorial readable. That structured cadence can trigger low burstiness flags even though the writing is entirely human. Meanwhile, an AI model explicitly prompted to "drastically vary your sentence length throughout this response" can produce text that scores high on burstiness without any actual human involvement.
The metrics work directionally. They are useful signals. But treating them as definitive is like judging a restaurant entirely by its Yelp star rating without reading a single review.
The Dimensions Nobody Mentions
Most public discussion of AI detection begins and ends with perplexity and burstiness. I get why. Those are the two metrics OpenAI mentioned when they briefly released their own detector, and that framing stuck.
But the research literature points to at least half a dozen other signals that modern detection systems incorporate, and the more sophisticated platforms are getting more transparent about this.
Vocabulary diversity is a big one, typically measured through type-token ratios or hapax legomena counts, which track how many words appear exactly once in a passage. AI models, especially ones running at conservative temperature settings, tend to reuse vocabulary more frequently than human writers do. A paragraph where "important" shows up six times instead of rotating between "crucial," "significant," "essential," and "vital" is raising a subtle flag that most readers would never consciously notice.
Then there are syntactic pattern markers. Certain sentence constructions appear disproportionately in AI generated text. Not because the model cannot produce variety, but because its training data skews heavily toward particular rhetorical patterns. Corporate blog posts and academic papers dominate the training corpora, and those genres have their own stylistic fingerprints. The "not only, but also" construction is a classic example. It shows up far more often in formal written English than in spontaneous human speech, and AI models have absorbed that distribution.
Discourse level coherence is another dimension gaining attention. Humans maintain thematic threads across paragraphs in ways that are genuinely hard to formalize but relatively easy for a trained classifier to notice when they are missing. AI text can be perfectly coherent at the local level. Each sentence follows logically from the one before it. But the long range structural integrity that characterizes sustained human argumentation is often absent. The text drifts. It circles back to the same points without developing them. It lacks the kind of argumentative arc that a human writer builds almost instinctively.
My point is not that any single one of these signals is definitive. My point is that boiling AI detection down to two numbers misses almost everything interesting about how these systems actually work. It would be like evaluating a car's performance by only looking at horsepower and top speed while ignoring torque, weight distribution, aerodynamics, and a dozen other factors that determine how the thing actually drives.
The Practical Side: Why Rewriting Is Not a Button Press
If detectors measure specific things, the obvious question is whether you can just adjust AI output to score better on those metrics. The short answer is yes, sort of. The longer answer is that every adjustment tends to break something else.
Bumping up perplexity by inserting unusual word choices makes text sound unnatural in a different way. You know the kind of writing I mean. Every third adjective was clearly pulled from a thesaurus by someone who has never used that word in actual conversation. The vocabulary is technically varied but the effect is the opposite of natural.
Manually varying sentence length to increase burstiness produces text that feels choppy rather than rhythmic. You can spot it in the wild pretty easily. A short sentence. Then another short sentence. Then a very long sentence that tries to compensate for all the short ones by cramming in subordinate clause after subordinate clause until the reader forgets where the sentence started. This is not how human sentence variation works. Human variation has a logic to it. Short sentences land hard points. Long sentences build momentum. The rhythm serves the meaning rather than fighting against it.
Increasing vocabulary diversity without adjusting the underlying argument structure creates writing that is lexically varied but intellectually flat. You swapped out some words but you did not add any new ideas. The detector might give you a better score but any careful reader will notice that something feels off.
What the more capable tools in this space have figured out is that effective rewriting requires simultaneous adjustment across multiple dimensions, not a one dimensional perplexity bump. You need to vary sentence structure, diversify word choice, restructure paragraphs at the logical level, inject appropriate emotional tone, and add concrete specific details of the kind that language models tend to gloss over. None of this is a single transformation. It is a set of edits that pull in different directions and need to be balanced against each other.
The platforms that offer paragraph level analysis rather than a single document wide score end up being substantially more useful in practice. Real text has problems that cluster. Your introduction might scan as completely human while paragraph four triggers every flag in the system. A single aggregate score tells you there is a problem somewhere. A paragraph by paragraph breakdown tells you exactly where to focus, which turns an intimidating task into something manageable.
What This Means If You Work With AI Text
If you are a developer building products that involve AI generated content, whether documentation, marketing copy, or anything user facing, a few things are worth keeping in mind.
Do not trust single metric detection scores. Any tool that gives you a percentage and a green or red light without explaining what it measured or how is hiding the information you actually need. Ask what model they are measuring against. Ask what dimensions they analyze beyond perplexity. If they cannot answer those questions, you are looking at a black box that is probably making decisions based on surface level statistical correlations that sometimes align with AI generation and sometimes do not.
Context matters more than any individual metric. The same paragraph that looks suspicious in a creative essay might be entirely normal in a technical specification. Human writing spans an enormous range of styles, and detectors calibrated on one slice of that range will misfire on others. This is not a theoretical concern. It happens constantly, and the consequences range from annoying to genuinely harmful depending on what hangs on the detection result.
If you are trying to improve AI generated text, treat it as a multi dimensional editing problem rather than a single pass transformation. This is where tools that combine detection and rewriting become genuinely useful. Not as a magic button that turns machine text into human prose. That framing oversells what the technology can do and sets up expectations that no current tool can meet. But as an iterative editing workflow where you identify specific problems, address them individually, and verify the results, the approach works.
The ecosystem around this problem is maturing fast. A year ago most tools offered either basic detection or simple paraphrasing with nothing in between. Now there are platforms that break down perplexity, burstiness, and vocabulary diversity into separate per paragraph scores with specific rewriting suggestions for each dimension. Some combine this analysis with multiple distinct rewriting strategies. Sentence restructuring, vocabulary replacement, paragraph reorganization, tone injection, detail supplementation. Each addresses a different aspect of what makes text read as machine generated, and the combination is more powerful than any single approach alone.
If you want to understand this space, run experiments. Take a passage you wrote yourself, grab something ChatGPT generated, and manually edit another AI passage for ten minutes. Feed all three into a tool that gives you per paragraph multi dimensional results. The patterns you notice will teach you more about how these systems work than reading about it ever will.
I went into this research expecting to confirm some things I already believed. Instead I came out with a much messier and more interesting picture of what these metrics can and cannot tell us. Perplexity measures predictability. Burstiness measures structural variation. Both are useful. Both are easy to misinterpret. And the space between what they measure and what people think they measure turns out to be where most of the interesting problems live.
I write about AI text analysis and the tools around it fairly regularly. A lot of the experiments I mentioned in this piece were run on EvalHub, which is one of the platforms that separates perplexity, burstiness, and vocabulary diversity into individual paragraph level scores. That granularity made it easier to see where specific patterns clustered rather than staring at a single document wide number and guessing.
Top comments (0)