Two thirds of the sites blocking GPTBot are also blocking the crawler that sends them traffic. I measured it.
Blocking GPTBot is a reasonable decision. OpenAI documents it as the crawler that gathers training data, you get nothing back for it, and plenty of publishers have decided that trade isn't worth making. Fine.
The problem is that GPTBot is not the only OpenAI token in your robots.txt, and the other two do completely different jobs:
-
GPTBot— training corpus. Fetches content to train future models. Sends you nothing. -
OAI-SearchBot— builds the search index behind ChatGPT's browsing. This is what makes your page eligible to be cited. -
ChatGPT-User— the fetch that happens while a person waits for an answer. A citation here is a link a human can click.
Block the first and you've opted out of training. Block all three and you've opted out of being found, which is a different decision that a lot of people appear to be making by accident.
How I measured it
I wanted actual numbers rather than vibes, so:
- Frame: Tranco top-1M (the research-grade domain ranking), downloaded 2026-11-07.
-
Sample: the 100 highest-ranked domains per ccTLD —
.com,.de,.fr,.es,.it,.jp,.nl. Deterministic, so anyone can reproduce it. -
Fetch:
robots.txtover HTTPS directly,www.then bare, redirects followed, 12s timeout. -
Verdict: for path
/, using RFC 9309 group selection — the exact user-agent group if one exists, otherwise*, never both — with longest-match precedence andAllowwinning ties. -
Read successfully: 528 of 700. The rest were unreachable (137) or served an HTML page where
robots.txtshould be (35).
The numbers (n=528)
| Measure | Share |
|---|---|
Block GPTBot
|
30.5% |
Block OAI-SearchBot
|
16.5% |
Block ChatGPT-User
|
20.3% |
| Block at least one traffic-bringing AI crawler | 20.5% |
Block GPTBot but allow both traffic crawlers
|
10.2% |
| Block an AI traffic crawler while still allowing Googlebot | 17.2% |
| Blanket-block Googlebot too | 3.4% |
The line that matters: of the 161 sites blocking GPTBot, 107 — 66.5% — also block a traffic-bringing crawler. Only 10.2% of the whole sample made the distinction and blocked training while keeping discovery open.
And 17.2% block an AI crawler that could cite them while carefully keeping Googlebot allowed. Those sites clearly care about being found. They just don't appear to know that User-agent: * plus a Disallow catches a lot more than Google's competitors.
It splits by market, and not how I expected
| Market | Read | GPTBot |
Any traffic crawler |
|---|---|---|---|
Netherlands (.nl) |
73 | 37.0% | 31.5% |
France (.fr) |
89 | 36.0% | 25.8% |
English (.com) |
64 | 34.4% | 29.7% |
Italy (.it) |
68 | 33.8% | 16.2% |
Germany (.de) |
81 | 30.9% | 12.3% |
Spain (.es) |
80 | 23.8% | 15.0% |
Japan (.jp) |
73 | 17.8% | 13.7% |
Germany is the interesting one. German sites block GPTBot at nearly the same rate as .com (30.9% vs 34.4%) but block traffic crawlers at less than half the rate (12.3% vs 29.7%). That is a market making the deliberate version of the decision. Japan barely blocks anything.
The Netherlands is the opposite — the highest blocking rate in the sample on both counts.
The honest limitation
These are high-ranking domains. They are not small business websites, and the figures should not be read as a picture of the small-business web. Large publishers have legal teams with opinions about training data; a plumber's site has whatever the theme shipped. A second sample of lower-ranked domains would be needed before claiming anything about the long tail — I haven't run it yet.
Why "which line decided it" is the whole problem
When I started checking individual sites I kept hitting the same thing: the owner had no idea their file blocked what it blocks. Not because they misread it, but because robots.txt precedence is genuinely unintuitive:
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
A lot of people read that as "block everything, but let Google in." It is — but it also blocks every AI crawler, including the two that could send readers. There's no error, no warning, nothing in Search Console. The file is doing exactly what it says and not at all what was meant.
Worse, the groups don't combine. Under RFC 9309 a crawler obeys the group matching its own token, and falls back to * only if no such group exists. So:
User-agent: Google-Extended
Disallow: /
...does nothing whatsoever to Googlebot — different token, different group. And a named empty group beats a restrictive wildcard:
User-agent: *
Disallow: /
User-agent: GPTBot
That allows GPTBot everything. The group exists, it has no rules, so nothing is disallowed. People write this believing it's a block.
So I open-sourced the checker
Every number above came out of a matcher I needed anyway, so it's on GitHub under MIT:
npx github:orenhacham-lab/ai-crawler-access nytimes.com
Zero dependencies, no build step, one network request. For every crawler it prints the verdict and the exact line that produced it — which turned out to be the only part people actually find convincing, because "your file blocks ChatGPT" gets argued with and "line 14, Disallow: /, in the * group that applies to you because you have no OAI-SearchBot group" does not.
As a library:
import { checkRobots, summarize } from 'ai-crawler-access'
const result = await checkRobots('example.com')
console.log(summarize(result)) // training vs discovery, separated
The crawler dataset is separate and CC BY 4.0, as JSON and CSV. Every row's token spelling, purpose and robots.txt behaviour comes from the operator's own documentation, with the URL and the date it was read in the row — nothing inferred from log files. Tokens I've seen in the wild but can't find documentation for are listed separately as candidates, because "we saw this" and "the operator says this" are not the same claim and shouldn't sit in one table.
- Code: https://github.com/orenhacham-lab/ai-crawler-access
- Dataset: https://www.gotopseo.com/api/public/ai-crawlers (no key, no signup, CORS open)
- Hosted checker: https://www.gotopseo.com/en/ai-crawlers
Corrections to the dataset are the contribution I want most — particularly an operator doc page for a token currently sitting in candidates.
What I'd actually do with this
If you want out of training but still want to be cited, the file is explicit and boring:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
Named groups for every token you have an opinion about. Don't rely on * to express a nuanced position — it can't, and the 66.5% figure above is what happens when people try.
Two caveats worth saying out loud. Google-Extended and Apple's equivalent are control tokens that crawl nothing by themselves; disallowing them changes how already-crawled content may be used, and both operators document that it does not remove you from search results. And a handful of agents have operators who document that robots.txt may be ignored — for those, your file states a request, not an outcome.
Top comments (1)
The 66.5% is best read as an intent-effect gap with no feedback channel: the file parses, the syntax is valid, and nothing anywhere reports the semantic verdict. That's why the natural artifact is the evaluator you built, pointed at one file instead of 528: a deploy-time table of the 30 bot tokens that matter, each with its effective verdict for /. If OAI-SearchBot flips to Disallow, the build says so that day, not after a traffic audit. One design note on why this keeps recurring: any policy stated by enumeration is incomplete, so every site is implicitly choosing which way it fails on tokens that don't exist yet. Disallow-by-list means tomorrow's training crawler starts allowed; allow-by-list means tomorrow's citation crawler starts blocked. The wildcard line is where that choice actually lives, and most files made it by accident.