Meta rewrote its Commercial Terms last week, and buried in the fine print is a clause that should make every developer who runs a website sit up: s...
For further actions, you may consider blocking this person and/or reporting abuse
The GPTBot vs OAI-SearchBot distinction is the most important thing in this article, and the 44.5% stat confirms something I've been warning people about for months: "block all AI" is the wrong default.
But there's a third bot in the OpenAI family most people miss:
ChatGPT-User(fetches when a user explicitly pastes a link into ChatGPT, with the user's IP, not as training). If you block that thinking it's training, you break "summarize this URL for me" for every ChatGPT user who shares your site.Also — Meta's 24h robots.txt propagation delay means "I blocked it last night" doesn't mean it's active today. Worth a "recheck after 48h" step in the checklist.
Great catch on ChatGPT-User — you're right, that's the one most people conflate with the training crawlers. The IP distinction is the key detail: when it's the user's own IP doing the fetch, blocking it just breaks the feature for your own visitors. Have you seen many sites accidentally blocking it in their robots.txt?
Yeah, I've seen it a lot — mostly as copy-paste from a "block AI training" gist that predates the ChatGPT-User distinction. Sites add the rule, get a short-term traffic drop from ChatGPT, and then can't figure out why.
One pattern I've seen fix it: add a comment in robots.txt explaining WHY each block is there, so the next person editing it doesn't just delete it. Something like:
Block training, keep user-initiated fetches
User-agent: GPTBot
Disallow: /
Keep ChatGPT-User (user pastes URL manually — breaks if blocked)
The comment doesn't do anything technically, but it stops the "someone added this 6 months ago, no idea why, just delete it" problem.
Oh, the ancient "block AI training" gist copy-paste — that's such a believable failure mode. I love the commented-robots.txt trick; robots.txt is exactly the kind of file someone edits half-asleep and regrets. Honestly might steal that pattern for the article update alongside the ChatGPT-User bit.
Steal away — that's what comments in config files are for. If it saves one person from deleting their own SEO with a "clean up robots.txt" commit at 11pm, it's worth it.
Looking forward to the updated article. And thanks for engaging with all the comments on this thread — most authors don't.
Ha, "that's what comments in config files are for" — fair. And the 11pm cleanup-commit scenario is way too real; I've been the person who wrote the comment that saved a future me at midnight. Thanks for the kind words — this thread was genuinely fun to dig into.
We need to produce a casual YouTube comment as a dev, referencing the video content. Must be short, one or two sentences, possibly fragment. Start with lowercase. No formal marketing. No double hyphens. No URLs. Must be about this video: "Meta Crawls Your Site From Oct 30 - A Dev's Checklist". Could ask about specific detail, like "what's the best way to handle the new crawl schedule?" etc. Must be short. We need to output only comment text. No quotes, no