The prompt was "Simplify word choices as if written by a non-native speaker." The essays were written by US eighth graders. Seven detectors had been reading that set almost perfectly, misclassifying an average of 5.19% of them as AI. After the prompt, 56.65%.
Nobody changed who wrote anything. The arguments were the same arguments, by the same kids. Only the vocabulary moved.
I keep meeting this study as a single number. It gets cited as "the one where detectors flagged 61% of essays by non-native writers," which is true, and which I think is the least interesting thing in it. The paper is on arXiv and the interesting part is that the authors did not stop at comparing two populations. They ran the manipulation in both directions.
Direction one. They took 91 human-written TOEFL essays. The average false positive rate across seven detectors was 61.22%, all seven unanimously agreed on 18 of them, and 89 of the 91 were flagged by at least one. Then they asked ChatGPT to "Enhance the word choices to sound more like that of a native speaker." Average false positive rate fell to 11.77%. The unanimous count went from 18 essays to 1.
Direction two is the one I quoted at the top, and it is the stronger evidence because it runs the other way. Take writing the detectors handled well, make the vocabulary plainer, watch the classification break.
Two directions, one mechanism. What the classifiers are reacting to lives in the surface language, and lexical range is a large part of it. That is not a demographic signal leaking into the model. There is no field in there for where you grew up. It is a text feature that happens to correlate hard with a population.
If you build classifiers this will feel familiar and slightly grim. It is the standard shape of a proxy variable, arrived at honestly, by a model that was never given the protected attribute and learned something adjacent to it from the data. The part that makes this one worse than usual is where the output lands. A recommender ranking your feed badly costs you a boring afternoon. This one arrives in a meeting where a person is deciding whether you cheated.
The same paper has a third result that I would put on the first slide of any procurement deck. They generated essays with ChatGPT-3.5, which the detectors caught reliably, then applied one self-edit prompt: "Elevate the provided text by employing literary language." Detection fell from 100% to 13%.
Put the three results next to each other and you get a measurement instrument with two failure modes pointing in opposite directions. Plain vocabulary from a human reads as machine. Ornate vocabulary from a machine reads as human. A single prompt moves it 87 points in the direction that helps whoever is trying to cheat. The same axis penalises whoever has the smallest working vocabulary, and that is not usually the person cheating.
Caveats, because this paper is old in AI years and the number gets flung around without them. It is 91 essays from one forum plus 88 from the ASAP dataset. The two groups differ in age as well as in language background, which is a real confound. The prompts rewrite more than vocabulary, so calling this a clean vocabulary experiment is too strong. It is seven products as they existed in early 2023, and detectors have shipped many versions since. A vendor pushing back on the generalisation is not obviously wrong to.
None of that touches the structural point, which is that an aggregate accuracy figure can sit on top of a much worse number for one subgroup and look fine. You cannot recover the subgroup number from the aggregate. Somebody has to measure it separately. So if you are ever in the room when one of these gets evaluated, the question is not what the accuracy is. It is what the error rate is on the population that will actually be judged by it, and whether anybody holds that figure at all.
The bad reading of this study is that you should feed your writing to a model and ask for fancier words. Look at what that does. Your paper is now a bit less like you, on the theory that a classifier prefers it, and you have accepted that the classifier gets to define good writing. Also the paper measures the 2023 tools. Nobody can promise you a number on the one your institution runs next semester, and I would treat any product that does promise you one as having told you something important about itself.
I do work in this area, on HumanPen, so read that last paragraph knowing it argues against my own commercial interest in saying anything more confident.
If you have a subgroup error breakdown for any current detector, from a vendor or from your own testing, I would genuinely like to see it. I have looked and I keep finding the 2023 paper.
Top comments (0)