Two thirds of sites blocking GPTBot are also blocking the crawler that brings them traffic

orenhacham-lab — Sat, 07 Nov 2026 12:03:30 +0000

Blocking GPTBot and blocking OAI-SearchBot are not the same decision. Most robots.txt files treat them as if they were, and the cost of that confusion is invisible: nothing in your analytics will ever tell you about the visitor who never arrived.

GPTBot collects pages to train a model. Deciding you don't want that is a legitimate editorial choice with no effect on whether anyone can find you.
OAI-SearchBot builds the index behind ChatGPT's answers.
ChatGPT-User fetches your page while a person is waiting for an answer — the fetch that can cite you and send a click.

Block the first and you opt out of training. Block the other two and you have removed yourself from the place your customers are now asking their questions.

So I measured how often that happens

Method. I took the Tranco top-1M list — the standard research ranking — and sampled the 100 highest-ranked domains for each of seven markets by ccTLD: .com, .de, .fr, .es, .it, .jp, .nl. 700 domains, measured on 2026-11-07. I fetched each robots.txt directly and evaluated the homepage path with a strict RFC 9309 matcher. 528 sites served a readable file; 137 were unreachable and 35 served HTML instead.

Results, across those 528 sites:

	share
block `GPTBot` (training)	30.5%
block `OAI-SearchBot` (ChatGPT's search index)	16.5%
block `ChatGPT-User` (live answer fetch)	20.3%
block at least one traffic-bringing AI crawler	20.5%
block an AI traffic crawler while allowing Googlebot	17.2%

The number I did not expect: of the 161 sites that block GPTBot, 66.5% also block a crawler that would have brought them a visitor. Only 10.2% of all sites made the narrow choice — block training, keep discovery.

And 17.2% block an AI crawler that brings traffic while still welcoming Googlebot. They are not opting out of search. They are opting out of the half of search that grew.

The limitation, stated plainly: these are high-ranking sites, not small businesses. The figures describe the leading sites in each market and should not be read as a picture of small-business websites. A separate sample would be needed for that claim.

The four rules a robots.txt matcher usually gets wrong

I wrote the matcher because the answer I wanted was not "blocked" but "blocked, because of this line, in this group". These four are where implementations fall over, and each is pinned by a test:

1. Group selection is exact, not a prefix. A group for Google-Extended does not govern Googlebot. Prefix matching is the classic bug that reports a site as blocking Google when it blocks one Google product.

2. * is a fallback, not an addition. If any group names the crawler, the * group is ignored entirely — not merged, not consulted for paths the specific group is silent about:

User-agent: *
Disallow: /

User-agent: GPTBot

That file allows GPTBot everywhere. Read it as "both apply" and you invert it.

3. Longest match wins, and Allow breaks the tie. Disallow: /blog/ plus Allow: /blog/public/ allows /blog/public/x.

4. An empty Disallow: allows everything. It is the documented way to say "no restrictions". Treat it as a zero-length prefix — which every path matches — and you have reversed the file's meaning completely.

The code

I open-sourced the checker. Zero dependencies, no build step, MIT:

npx github:orenhacham-lab/ai-crawler-access nytimes.com

  BLOCKED  OAI-SearchBot   OpenAI · search  ← disallow: / (line 266, agent group)

The line number is the point. A boolean gives a reader no reason to trust it over re-reading their own file.

import { checkRobots, summarize } from 'ai-crawler-access'

summarize(checkRobots(robotsText, '/'))
// { allowed, blocked, trafficBlocked: ['OAI-SearchBot'], blocksTrainingOnly: false }

Repo: https://github.com/orenhacham-lab/ai-crawler-access

The crawler list is also published as JSON, CSV and a JSON Schema under CC BY, so you can consume it without installing anything.

Corrections are the contribution it most wants — if an operator documents a token I have wrong, open an issue with the link.

Disclosure: I build Go Top SEO, and there is a hosted version of this check with a UI and no signup at gotopseo.com/en/ai-crawlers. The library above is MIT-licensed and needs nothing from me to run. This article was drafted with AI assistance; the measurement, the code and the conclusions are mine.

DEV Community: orenhacham-lab

Two thirds of sites blocking GPTBot are also blocking the crawler that brings them traffic

So I measured how often that happens

The four rules a robots.txt matcher usually gets wrong

The code