Wikipedia has been running a public, adversarial AI-detection program since December 2023. It has a field guide of tells, a quiz with an answer key, and a documented list of indicators the editors decided are useless. I went looking for the code-side equivalent and could not find one, even though the code-detection literature keeps landing on the same conclusions the Wikipedia editors published in prose.
The field guide is the artifact worth reading. Wikipedia:Signs of AI writing opens by telling you what it is not:
This list is descriptive, not prescriptive; it consists of observations, not rules. The patterns listed here are also only potential signs of a problem, not the problem itself.
And then the line I would tape to the monitor of anyone building an "AI code detector":
Please do not merely treat these signs as the problems to be fixed; that could just make detection harder.
That is the trap in every style-based detector. The tells decay when models change, and they are also things humans do. The guide puts "perfect grammar", "bland or robotic prose", and prose that mixes clinical and emotional register under a section called Ineffective indicators, with the note that a person writing casually in a technical field may trip all three.
They published the error rates, including their own
The caveats section is where this stops being a writing-style page and starts being an eval page. Three numbers:
A 2025 study found human ability to distinguish LLM text from human text was no better than random chance. A separate study on German theses put humans at 57% recognition on AI texts and 64% on human texts, which is a coin flip with extra steps. A preprint cited in the same section had expert annotators hit 90% on AI-generated text while misidentifying 3.3% of human-written text as AI.
Then the guide turns on the tools. GPTZero and Pangram "perform better than random chance" and still carry non-trivial error rates, and are documented as degrading under paraphrasing, markup changes, and spacing changes. The guide's own example of a robustness problem is the RAID benchmark, which exists precisely because detector scores do not survive distribution shift.
The operational consequence is stated flatly: a high detector score is not a valid criterion for speedy deletion. Which is the correct posture, and it is the posture most code teams have not taken. I have watched people get accused of shipping AI code on the strength of a detector percentage with no false-positive number attached to it.
The quiz is a calibration test, and the scoreboard is public
Wikipedia:AI or not quiz is ten articles, human or AI, with answers at the bottom of each page. The page says some entries had obvious tells stripped (access dates, for example) and that the page history spoils the answer, so you have to read the text.
The quiz has a public scoreboard where editors post their results. I pulled the wikitext and counted the self-reported scores: 145 entries, mean 7.37 out of 10, median 8. Sixteen people scored 10/10 and two scored 2/10. The 6/10 and 7/10 buckets hold 45 of the 145.
Read that scoreboard with the caveats it deserves. Self-reported, self-selected, and people who bomb a public quiz are less likely to post their score, so 7.37 is probably optimistic. The interesting signal is in the comments. A recurring one is that the misses were false positives on human-written text. One editor wrote that three of their four misses were articles they wrongly called AI. Another said "I might be too suspicious." A third noted that every article they misidentified was written before 2020.
That is the failure mode of style-based detection in one thread: the cost lands on people who did nothing wrong, and the tells are correlated with era rather than authorship.
The tells expire, and they say so
This is the part I did not expect. The guide has a section for historical indicators, meaning signs that used to work and no longer do. Em dash overuse is listed there with the date range 2022 to September 2026. Outdated access-date parameters, knowledge-cutoff statements, and abrupt cutoffs are in the same bucket.
Meanwhile the page names the signs it currently treats as most common in new AI-generated text: Markdown syntax leaking into wikitext, AI-written edit summaries, hedging disclaimers about source availability, and canned emphasis on attribution and media coverage.
That list is a snapshot with a shelf life, and the editors know it. In an IEEE Spectrum interview, WikiProject AI Cleanup cofounder Ellia Lebleu describes the shift directly: by 2025 the signs had become more structural, with the rule of three and negative parallelisms mattering as they accumulate rather than individually.
She also names the signal that has nothing to do with prose. If someone writes five articles of several thousand words each in five minutes, you have a reason to suspect AI. That is not a style tell. That is a rate observation, and rate observations do not decay when a model updates its phrasing.
Where they stopped trusting text and started checking provenance
Two more things from that interview matter for code.
On citations: hallucinated references have become rarer because retrieval-augmented generation can find real sources, but the more common failure is a real source whose content does not support the claim. She calls that "extremely commonplace." This is the text equivalent of a coding agent citing a real function that does not do what the diff assumes it does.
On their own AI verifier: another editor built a tool that walks the sources on a page and checks whether they match the claims, and Lebleu flags the obvious problem. Since the verifier is itself a model, there is no guarantee it is right every time. So it gets used to filter obvious cases at scale, and a human still owns the call.
They also moved the durable evidence out of the prose entirely. The guide's Signs of human writing section leads with the age of the text relative to the ChatGPT launch on November 30, 2022. An edit made before that date cannot be AI-generated, and the editors treat that as conclusive rather than suggestive. The other two durable signals are syntax and whether the author can explain their own editorial choices.
A timestamp. A checkable claim. An author who can answer questions about the work. None of those are style.
The code side landed in the same place
This is not a text problem that code avoids. AICD Bench is the widest code-detection benchmark I have seen: 2 million examples, 77 models across 11 families, 9 languages, including reasoning models. It sets up three tasks, binary classification under distribution shift, model family attribution, and fine-grained classification across human, machine, hybrid, and adversarial code.
The result the authors report is that detector performance "remains far below practical usability, particularly under distribution shift and for hybrid or adversarial code." That is the same finding as the Wikipedia caveats, measured on code, with a much bigger sample.
The earlier ICSE 2025 study is blunter about the tools: the existing AI-code detectors "all perform poorly and lack sufficient generalizability to be practically deployed." Their best classifier, trained on AST-derived features and code embeddings, reached an F1 of 82.55, and that is a trained classifier on a matched distribution, not an off-the-shelf checker pointed at your monorepo.
There is also a family-attribution line of work, LLM code stylometry for authorship attribution, which maps to the guide's section on model differences. Each model has a recognizable idiolect, so what marks GPT-5 output is not what marks Claude output. Practical upshot: a single heuristic calibrated on one model's output is stale the week the next model ships.
What survives is provenance, and you can get it today
The signal that holds up is not how the code reads. It is what you recorded about how it got there.
Here is the two-tier rubric I would run on a PR, in order. Tier one is evidence, and it is cheap to collect:
- Does the commit carry an AI attribution trailer, and does your team's convention require one? Trailer counts are a policy measurement, not a truth measurement, and the per-repo spread is wide, so report per-repo numbers instead of a fleet average.
- What is the arrival rate? Five PRs from one author in an hour is a rate observation, the code equivalent of five articles in five minutes.
- Can the author explain the change without re-reading it? This is the guide's "ability to explain one's own editorial choices", and it is the single highest-signal question in a review, because it is hard to fake on a diff you did not write.
- Does the cited thing exist and do what the code assumes? This is the source-verification step. A comment that references a function whose behavior changed last week is the code version of a real citation that does not support the claim.
Tier two is style, and it is for triage only: comment density, formatting that looks like a chat response, identifier naming that flips register mid-file, a docstring that restates the signature. These are weak individually and they decay. Never let a tier-two signal be the only reason you say something.
Calibrate your own detection before you accuse anyone
The Wikipedia quiz is a calibration instrument, and you can copy the mechanic in an afternoon. Take ten merged PRs from the last year, five that you know came from a human writing by hand and five you know came from an agent, strip the branch names and the commit trailers, and shuffle them. Then have the reviewers who will be doing the accusing score them blind, and count the false positives separately from the misses.
The scoreboard on the Wikipedia quiz tells you what to expect. Most people land at 7 or 8 out of 10, and the misses cluster on human-written text. If your team's reviewers land there too, you now know that "this looks AI-written" is a claim with a measurable error rate, and it deserves the same skepticism as any other claim.
That number also changes what you do with the finding. A miss means you reviewed some AI code as if a human wrote it, and the review still happened, so the cost is low. A false positive means you told a colleague their work looks machine-made, and if you were wrong, you spent trust on nothing. The asymmetry is the whole argument for putting provenance first.
What this means for review tooling
If you are evaluating AI review tools, the question to ask is whether the tool reports provenance or infers style. CodeRabbit, Greptile, Kodus, and Copilot's PR reviewer all surface metadata next to their findings, and the useful versions tell you what they read to reach a conclusion, so you can check it. The useless versions hand you a confidence percentage with no false-positive rate attached, which is the same thing the Wikipedia editors decided was not enough to delete a page.
The reason this matters for the review-volume problem is that detection keeps getting proposed as the answer to it. If you can tag the AI-written PRs, the argument goes, you can route them somewhere different. That plan depends on the tag being right, and both the text-side and the code-side evidence say the style-based version of the tag is wrong often enough that people stop trusting it. Provenance-based tagging is narrower, because it only covers work that declared itself, and it is the part that stays true after the next model release.
So the honest version of the pipeline is boring. Record what generated the change when you know it. Ask the author to explain the diff. Check the citations. Run style heuristics as triage, never as a verdict. And publish your own false-positive rate somewhere, because the one program that has been doing this for three years at scale published theirs, and it is the reason I believe anything else on the page.
Top comments (0)