There is a note at the top of GPTZero's own explainer about perplexity and burstiness. It reads: "As of autumn 2023, GPTZero no longer uses perplexity and burstiness for its AI detection because we migrated to a deep-learning based architecture."
That page is still up, still the best short explanation of the two terms on the internet, and still the thing everyone links when they explain how AI detectors work. Including, until recently, me.
Quick recap for anyone who has not had to care. Perplexity is how surprised a language model is by your next word. If you wrote the word it would have predicted, perplexity is low. Burstiness is how much your sentence construction varies across a passage. Long compound sentence, short flat one, fragment. Low variation, low burstiness. The theory was that generated text sits low on both, so low plus low means machine.
It is a good theory. It is the theory the entire public conversation still runs on, three years after the company that popularised it moved off it.
Turnitin says something adjacent on its FAQ page, and the phrasing is careful in a way worth copying: its model "is not explicitly programmed to evaluate specific signals such as 'burstiness,' 'perplexity,' or other individual metrics sometimes referenced in public discussions." The next sentence: "Instead, it learns statistical patterns from our training data."
Do not over-read that. Elsewhere on the same page Turnitin says its classifiers are trained to detect differences in word probability, and perplexity is a function of word probability. So the accurate version is narrow: the model does not compute a burstiness score and a perplexity score and threshold them. It is not a claim that word probability is irrelevant. I have watched people quote the first sentence as if it were the second, and it is not a distinction you can hand-wave in a meeting where somebody's degree is on the table.
What Turnitin does publish about mechanism is more useful than either statistic. Sentences are extracted and segmented into overlapping sections. Each segment gets a value between 0 and 1. Every qualifying sentence inherits its segment's score, and because segments overlap, one sentence can carry several scores, which are pooled, then aggregated into the document score.
That paragraph explains a thing the perplexity story cannot. Turnitin says that in documents of only a few hundred words the prediction is "mostly 'all or nothing'", because there is a single segment and no opportunity to overlap. And the consequence, in their words: text that is a mix of AI-generated and original content could be flagged as entirely AI-generated.
So the length of the input changes the shape of the output, structurally, before any question of what the model believes. Which is worth knowing next time somebody pastes one paragraph into a free checker and treats the result as a reading on their thesis.
The practical fallout of all this is mostly negative and I think that is fine. Lists of forbidden AI words are poor explanations of a deep classifier, because the classifier was never assembling a list. It learned combinations that nobody at the vendor can necessarily name individually, which is also why Turnitin admits individual predictions may not be explainable feature by feature. If the internal decision surface is not a list of habits, then no list of habits describes it.
Two things that survive all of the above and are worth writing down.
First, whatever the number is, ask what it is a percentage of. Document confidence, sentence confidence and share-of-eligible-prose are three different quantities and they get displayed in the same little coloured box.
Second, version and date belong with the score, always. A vendor can update a model and change a classification without a single character of the prose changing. A screenshot of a percentage with no product name, no date and no language is not reproducible by anyone, including the vendor.
I work on HumanPen, which is how I ended up reading more of these vendor pages than is good for a person. The thing I did not expect going in: the vendors are consistently more careful in their own documentation than the internet is when quoting them. The overclaiming is almost all downstream.
If you know of a current detector that publishes per-version behaviour in a way you can actually diff, drop it below. I have not found one.
Top comments (1)
That narrow distinction matters. A detector can move away from hand-thresholded perplexity and still learn word-probability patterns inside a classifier. The bad failure mode is when the score turns into disciplinary evidence without that caveat attached.