This was originally published on All Agents Considered.
I have no doubt you already know that last week Substack just attached an AI detector to every article on the platform.
The new “Scan for AI” feature lets readers analyze a post, Note, comment, or reply with Pangram, which estimates how much of the writing is human, AI-assisted or written by AI.
And the percentage it returns might looks precise, like a speed camera telling you exactly how fast you were going.
But Pangram did not watch the person type, check their browsing history, or find a hidden signature left by ChatGPT or Claude. It looked at the finished text and decided what the writing resembled.
That distinction sits at the center of the argument happening across the Substack, professor, and AI research subreddits.
Some people think these detectors generate little more than a random number. Others treat the score like a confession.
Both sides are missing the point.
Pangram is pretty good at identifying obvious AI writing, and it has more behind it than the early tools that made AI detection a laughing stock. In independent testing from the University of Chicago, it identified writing from widely available AI models with low error rates under the conditions studied.
But it is not perfect. Some model output and repeated paraphrasing can evade it, and human writing can still be classified incorrectly.
A low AI score does not prove that no AI was used, just as a high score does not prove an algo wrote the text. Pangram is a serious pattern detector, not a yes-or-no authorship test. It cannot magically settle the question of authorship. So today I am going a bit out of the way with the main topics of AAC to show you the truth behind Pangram and AI detectors.
In this edition I’ll cover how Pangram is trained, what its score measures, why it differs from older detectors, and where it fails, without pretending the tool can prove with 100% certainty who wrote a piece a text.
In this article:
Why older AI detectors earned so much distrust
How Pangram learns to identify AI patterns
What its percentage does and does not mean
The strongest evidence that it works
The known ways it can fail
How to use a result without accusing the wrong person
Why Everyone Hates AI Detectors
Early AI detectors often leaned heavily on two ideas called perplexity (not the tool) and burstiness.
Perplexity asks how surprised a language model is by the next word. AIs are trained to predict the next word, that’s literally written in the DNA of all LLMs. On the other hand, our human writing patterns are less predictable just because our writing tends to be messier.
Burstiness asks whether sentence lengths and structures vary. Humans may write one clipped sentence, then an overgrown paragraph, followed by a shorter sentences. AI models usually settle into a steady rhythm (like the Rule of 3’s).
You don’t need me to point out that these “AI detection” techniques are fundamentally flawed. Imagine judging whether bread came from a factory by checking whether every slice has the same thickness. That clue may help in some cases, but it doesn’t tell you exactly who baked the loaf.
A safety manual is predictable because it needs to be. A legal form repeats itself because consistency matters. A non-native writer may use simpler sentence patterns.
A poet may produce language so unusual that the detector has little familiar ground beneath it.
This is why old detectors produced absurd headlines. In 2023, Ars Technica reported that one detector labeled part of the US Constitution as likely AI-generated. A 2023 academic review of detection tools also found serious reliability problems, especially after text was edited or paraphrased. Pangram argues that those two measures alone fail as a basis for detection.
Thus an online myth was born: AI detectors don’t work.
Before we go into why Pangram is truly different, we really have to get this out of the way. Pangram is not the only trained detector in 2026. GPTZero describes a multiclass model for human, mixed, polished, generated, and paraphrased text. Originality says its current detector uses a trained transformer model built from human and generated examples.
Pangram Does Not Search for an Em Dash
Pangram is a trained classifier.
Researchers train it on many documents labeled by source: human-written or AI-generated. From those examples, the detector learns which combinations of patterns tend to distinguish one group from the other.
Think of it like teaching someone to recognize counterfeit passports rather than giving them a checklist of what a document should contain. Side note: this reminded me of the game Papers, Please.
Pangram says its flagship detector uses a transformer model adapted to classify sequences of text. Its technical explanation and original research paper describe a system that learns from human writing and model-generated writing. Basically they created an AI that can detect AIs.
It does not call OpenAI to ask whether ChatGPT wrote the paragraph. It does not recover the prompt. It does not compare the post with a giant database of every answer an AI company has produced. There’s literally no need for that.
Pangram learns what different kinds of writing tend to look like, then it analyzes the text you feed it against that map.
This should also clear up the em dash frenzy.
Pangram can show human-readable supporting clues such as em dashes, lists, headings, stock phrases, Markdown, and groups of. But Pangram says those visible clues are separate from the flagship detector. They help a reader inspect the text and show that they are not the inputs driving the main model.
To better put this into perspective, imagine a prisoner just escaped from incarceration. Deleting every em dash is like removing their striped shirt before a facial-recognition scan. The prisoner changed something visible without necessarily changing what the system recognizes.
Pangram Trains on Twins
The most interesting difference is how Pangram constructs its training set.
Suppose we trained a detector using personal diaries as the human examples and software landing pages as the AI examples. It might appear accurate while learning the wrong lesson. Instead of detecting AI, it could learn that feelings are human and product features are synthetic.
Pangram tries to prevent that shortcut with what it calls synthetic mirroring.
For each human document, it creates an AI counterpart about the same subject, in a similar tone and style, carrying similar information. A human book review is compared with an AI book review of the same material. A human essay is paired with its AI twin.
The detector must find differences between twins rather than compare a shopping list with a sales page.
Pangram’s model card says its human training material covers essays, reviews, books, creative writing, news, scientific papers, Wikipedia, and general web text. Its AI side is generated in-house to create those matched comparisons. Then it makes the exam even harder.
Pangram scans large collections of known human writing and looks for passages the detector wrongly calls AI. Researchers add those mistakes back into the training data, create AI mirrors for them, and train again.
This technique is called hard-negative mining.
There’s a famous story from World War II based on the statistician Abraham Wald. The military engineers were asked to examine aircraft that returned from battle and map where they had been hit. In most cases, the wings and fuselage were covered in bullet holes, so the obvious response was to reinforce those areas.
Wald argued that the military was looking only at surviving aircraft, but if those planes could return despite being hit in the wings and fuselage, then those areas were not the most vulnerable. That’s how he figured out the places with few bullet holes (like the engines) needed more protection.
Hard-negative mining follows a similar principle. The most useful evidence often comes from the cases a system mishandles. So instead of repeatedly training a detector on obvious AI writing, researchers study the difficult human passages Pangram (initially) falsely identified as AI and feed those mistakes back into the training process.
Pangram’s Open Model Is Not the Substack Model
Pangram has released public code, data links, and model weights through its EditLens repository. The project accompanies a paper accepted at ICLR 2026 on measuring degrees of AI editing.
That is meaningful transparency. It lets researchers inspect and reproduce part of the method without trusting a marketing page.
What it doesn’t reveal is the complete system running inside Pangram’s commercial product.
The public repository contains two smaller research models and instructions for training them. Pangram’s Open Pangram announcement explicitly says the release is a research baseline and should not be used to enforce AI policies in schools or workplaces.
The current commercial model described by Pangram is version 3.3. Its model card documents a larger production system with later training and scanning changes. Substack publicly confirms the Pangram integration, but its help page does not name the exact model version or threshold it uses.
What the Percentage Means
We can all agree that teal writing has made binary detection obsolete.
A writer may create the first draft, ask Claude to tighten one section, run the result through Grammarly, dictate a new ending, and then rewrite every sentence by hand. Which words belong to whom?
Pangram’s EditLens research tries to measure that grey area. Researchers begin with human text, ask models to edit it at different intensities, and compare each edited version with the original. A classifier then learns to estimate the degree of editing from the final text.
The full EditLens paper states the important limitation of its method. When the detector checks your article, it doesn’t receive your original draft. It only sees the final text.
The estimate is like a restorer examining a painting and guessing how much has been retouched. They may recognize signs of later work, but they can’t count the extra brushstrokes.
For long documents, Pangram breaks the text into overlapping windows, checks each window, then makes a finer pass near uncertain boundaries. Its 3.2 model card describes an approximate resolution of 50 words.
Pangram can then describe shares of a document as Human, Lightly AI-assisted, Moderately AI-assisted, or AI. A result such as 30% AI refers to the share of classified text segments assigned to that category.
It does not mean:
There is a 30% chance the writer used AI
A machine typed exactly 30% of the words
Pangram found the original prompt
The writer contributed only 70% of the thinking
Ideas, wording, structure, research, and final judgment are different contributions. No detector can reduce all of them to one authorship percentage by reading the final copy.
This is where Substack’s interface creates a social problem. A percentage looks like a measurement, but underneath it’s just a classification.
The Evidence That Pangram Works
The strongest independent evidence I found comes from the 2025 University of Chicago working paper Artificial Writing and Automated Detection.
The researchers gathered 1,992 verified human passages across blogs, novels, résumés, essays, business writing, and other genres. They produced matched AI versions using GPT-4.1, Claude Opus 4, Claude Sonnet 4, and Gemini 2.0 Flash.
On medium and long passages in that test, Pangram produced essentially zero false positives and false negatives. It outperformed GPTZero, OriginalityAI, and an open RoBERTa detector in the same experiment.
That should clear up the myth that “AI detectors never work”.
But this should also be taken with a grain of salt. The paper was, after all, a working paper. The test used a controlled set of matched human and synthetic documents. It evaluated the Pangram service available in May 2025 and it did not test every Substack voice, mixed editing workflow, new model, humanizer, or unusual format.
Pangram’s current numbers come from Pangram. Its 3.3 model card reports a 0.01% false-positive rate for long-form creative writing, 0.02% for academic writing, 0.04% for multilingual how-to articles, and 0.49% for poetry.
It also reports a 1.5% false-negative rate on a random Chatbot Arena set.
Those are impressive company-reported results, but they shouldn’t be treated as promises about the next text you scan.
The clearest warning comes from a competing test. In February 2026, GPTZero published its own benchmark and reported that GPTZero 4.3b beat Pangram 3.2 on its datasets.
That does not prove GPTZero is better. It proves the winner changes with the test set, model version, threshold, and person running the test.
I made the same argument in my breakdown of why AI benchmarks fail real agent workflows. A leaderboard compresses the conditions into one number, but conditions change and can be picked arbitrarily.
Where Pangram Messes Up
Pangram’s own documentation says it works best on long-form prose written in complete sentences, like a Substack post.
Its 3.3 limitations warn about bullet lists, technical instructions, tables of contents, references, templates, equations, headers, and footers. Poetry has a much higher reported false-positive rate than long creative prose.
Short text creates another problem. Just as a smoke detector cannot identify much from a few molecules in the air, Pangram can not correctly classify a short text.
Then there are false negatives.
A May 2026 preprint titled Base Models Look Human To AI Detectors found that output from base language models often looked human to both Pangram and GPTZero. Instruction-tuned chat models were easier to identify than base models. Researchers also used repeated paraphrasing to make AI text appear more human to the detectors.
That suggests detectors may be especially good at recognizing the habits produced by chat-model training, rather than detecting one universal substance called “AI writing.”
The Atlantic found another crack. A reporter sent ChatGPT and Claude output through the Walter Writes humanizer, then watched Pangram label the result human. Pangram has since described better humanizer detection in version 3.3, so one test does not freeze the system forever, but it does show the shape of the fight.
The detector learns the disguise, and the disguise keeps changing. It’s just like with police figuring out a new way to transport illicit substances, then traffickers come up with something new and the wheel keeps spinning.
This arms race creates an important asymmetry:
A result of “AI” can be a false accusation
A result of “Human” can be successfully disguised AI
Neither side provides proof.
Pangram CEO Max Spero told The Atlantic the detector should never be the final arbiter. The company setting limits on its own tool carries more weight than a critic attacking it from outside.
How to Read a Pangram Result
A Pangram result is closer to weather radar than a fingerprint.
Radar detects conditions associated with a storm. It can tell you when the signal is strong, where to look, and whether you should carry an umbrella. A fingerprint connects a person to an object through physical evidence, and Pangram has no such access to the writing process.
I also don’t use Pangram as a moral scorecard. I write about technical subjects, and I use AI heavily to structure my thoughts and explain difficult concepts to less technical readers. My first question when I read something is much simpler: Did I get anything valuable from it?
If an article teaches me something new, gives me a useful idea, or makes a complicated subject easier to understand, I’m satisfied with the time I spent reading it; whether AI helped produce it comes second.
I take the same approach to Duolingo, where I’m currently learning German. Duolingo has openly described using generative AI to create and validate course content. I also know that more than 1,000 volunteers helped create dozens of its original language courses.
I have serious ethical problems with that history. I personally believe that Duolingo owes an enormous debt to the community that helped build it, and embracing automation does not erase that debt. But I cannot reverse the company’s decisions, and I still want to learn German, so I keep on using the app and embrace the technology for what it can do.
That is roughly how I approach AI-assisted writing. I care about the quality of the thinking, the usefulness of the result, and whether the article rewards my attention. A detector score may tell me something about how the prose was produced, but it cannot decide whether the prose was worth reading.
With that in mind, I use four rules when reading a Pangram result:
Check the material. Is it long-form prose, or a short Note, poem, list, technical manual, reference section, or template? Pangram’s own model card says the detector is intended for long-form writing in complete sentences and warns that lists, instructions, reference sections, templates, and dense equations are more susceptible to false positives.
Read the label as resemblance. Pangram is a trained classifier. It studies patterns in human and AI writing and estimates which group a new passage most closely resembles. Pangram describes the process as turning the text into a numerical representation and passing it through a classifier. It does not observe who typed the words or watch the document being created.
Do not mistake the score for a reconstruction of the writing process. A high result does not tell me which ideas came from a model, how much work the writer contributed, or whether AI only helped with editing. A low result does not prove that no AI was involved. The detector sees the finished prose, not the conversation, research, prompting, or revision behind it. This is also why my best agent workflow is mostly files. Files preserve the trail between research, drafting, and editing, even when AI is part of the process.
Ask what question I actually want answered. If I want to know whether the prose resembles AI-generated writing, Pangram provides a useful signal. If I want to know whether the article is accurate, original, insightful, or worth reading, I have to judge those things myself. A strong idea does not become worthless because AI helped express it, just as a completely human-written article does not become valuable merely because a person typed every word.
I apply the same discipline when AI gives me a technical answer because accepting AI’s first answer without checking it replaces judgment with convenience. I do not reject an answer because a model produced it, and I do not trust it merely because it sounds polished. I ask whether it is accurate, useful, and worth my attention.
A Pangram result can describe patterns in the writing, but it cannot tell me whether I learned something. That would be dystopian.
My Verdict On Pangram
Pangram is different from the crude detectors people remember.
Its classifier learns from paired human and AI documents.
Its synthetic mirrors reduce topic shortcuts.
Its hard-negative mining feeds human false positives back into training.
Its calibration favors avoiding false accusations.
EditLens attempts to model mixed editing rather than forcing every document into a human-or-machine box.
Independent researchers found that this approach performed extremely well on their tested medium and long passages.
Those are facts.
Pangram still sees only the finished text. It cannot observe the actual writing process.
Its published research does not reveal every part of the production system, and its accuracy varies by domain. Short and formulaic formats remain particularly difficult.
New model types and paraphrasing can also evade detection. Not to mention independent benchmarks do not always agree with Pangram’s own results.
And even the percentage can be misleading. Readers may mistake it for a probability or a record of the writing process, but it is neither.
Those limitations are facts too.
The reasonable conclusion lies somewhere between “all detectors are useless” and “the algorithm knows the absolute truth”.
Pangram has earned the right to be treated as evidence, but it hasn’t earned the right to be treated as a verdict.
Top comments (0)