Paul Graham posted a test this week that has nothing to do with sentence structure. Slop gives itself away, he said, when the diction doesn't match the idea, when something completely ordinary gets delivered with the excitement of someone announcing a discovery.
That's a different axis than everything else I've been reading about detection this month. Pangram scores a pattern: token by token, sentence by sentence, does the shape of this text statistically resemble a machine's output. Sloan, the human version I ran into on DEV.to, did the same thing by ear instead of by classifier, GPTZero as a second opinion. Both are measuring the same thing: surface. PG's test measures a relationship. What's actually being said, against how much weight the delivery is putting behind saying it.
Run my own flagged pieces through it and they'd pass clean. What got them flagged wasn't excitement outrunning substance, it was named data points and short paragraphs doing real argumentative work, plainly. Pangram's classifier and PG's ear would disagree with each other on the same writing. That's worth sitting with. The tool built to formalize the intuition doesn't actually agree with the intuition once you test them against the same text.
Pascal's comment on the last piece fits here too. Rephrasing his own English for flow, after 6 to 20 hours of writing the argument himself, isn't a case of ordinary ideas dressed up as brilliant ones. It's someone's real thinking, translated. Nothing in that process produces the register mismatch PG is describing. A classifier flagged it anyway, at 97% confidence, on a post from 2017.
The other thing PG's test doesn't need is a platform. Chris Best's whole pitch for shipping Pangram into Substack is that reader intuition doesn't scale, you need a tool doing this at the volume a feed operates at. PG is quietly arguing the opposite: the tell was always available to anyone reading carefully, you just have to know what you're listening for. That's closer to Josh Puckett's complaint in "In Defense of Writing," that readers already discount slop on sight, than it is to anything Substack shipped this week.
I don't think that makes the tool pointless. Most readers aren't reading carefully, most of the time, and a feed moving fast enough rewards that. But it does mean the two approaches are solving different versions of the problem. One scales an ear. The other replaces it.
I know which one caught real writing and called it fake, and I know which one would have let it through. Diction that matches the idea isn't something a classifier trained on hard negatives is built to notice. It's something you have to actually read for.
Top comments (3)
Graham's test is a sharper version of the question from your last post. Diction-idea mismatch isn't a separate signal from "can the author defend what they wrote," it's what that question looks like from the reader's side. Someone who can't say why the enthusiasm is earned usually can't defend the underlying claim either. Same failure, different angle.
The 2017 example is the tell on the tell, though. A classifier reading register instead of proportionality flags genuine excitement about a modest idea the same way it flags padding. Your test needs a reader who already knows how big the idea is, which is exactly the part that doesn't scale.
Mike, "the tell on the tell" is exactly right and it's worse than a scaling problem, it's a domain problem. A reader outside their own expertise can't tell earned excitement from padding either way. A surprising result in a field they don't know will read as ordinary, because they don't know enough to be surprised by it. PG's test and a classifier fail in mirror-image directions on the same piece of writing: the classifier flags real writing that happens to be statistically regular, PG's test flags real writing that happens to be about something the reader can't judge.
Which puts your line from the other thread, can the author defend it line by line in an interesting spot. It doesn't need the reader to already know the domain going in, just to ask the right follow-up. Does that actually hold, or does a good defense still require enough shared context that an unfamiliar reader can't judge the defense either?
It doesn't fully hold, and I think the domain problem reproduces one level up instead of dissolving. A reader who can't tell earned excitement from padding in a field they don't know also can't reliably tell a good defense from a confident-sounding one in that same field, because judging the defense requires the same background the original text needed.
What the defense does change is the shape of the failure. PG's test fails silently: the reader misjudges and never knows it. A defense that gets challenged and can't hold up produces a visible tell, incoherence, contradiction, an inability to answer a specific follow-up, that doesn't require domain expertise to notice even if it requires domain expertise to fully evaluate. So the unfamiliar reader still can't confirm a good defense is correct, but they have a much better shot at noticing a bad one collapse under a direct question, which is a real improvement even though it's not the full answer either of us would want.