Everyone has an opinion on whether websites are blocking AI crawlers. We wanted the numbers, so on 4 November 2026 we downloaded the Tranco ranking of popular domains, took the top 2,000, fetched each site's robots.txt, and parsed the 1,069 that gave us a usable file for rules about 13 AI crawlers. We also checked whether each site serves an llms.txt. The per-site results are in a CSV you can download.
The short version
- About 1 in 5 sites blocks the big training crawlers: GPTBot (19.7%), ClaudeBot (18.8%) and CCBot (20.7%). 25.1% block at least one of six major training crawlers.
- Far fewer block the crawlers that decide whether you can appear in an AI answer: OAI-SearchBot (6.7%), Claude-SearchBot (8.1%) and PerplexityBot (11.6%). 12.3% block at least one of the three.
- Blocking GPTBot is not blocking ChatGPT search. Of the 211 sites that block GPTBot, 139 (65.9%) still allow OAI-SearchBot.
- Blocking is flat across rank. Sites ranked 1-100 block GPTBot at about the same rate as sites ranked 1,001-2,000.
- 16.6% of sites serve an llms.txt, and those sites are five times less likely to block GPTBot (4.5% versus 22.8%).
- 7.2% of sites already use the new Content-Signal robots.txt directive to state AI-use preferences.
Training crawlers: about one in five sites say no
Sites blocking each training crawler site-wide (n = 1,069)
| Crawler | Operator | Sites blocking | Share |
|---|---|---|---|
| CCBot | Common Crawl | 221 | 20.7% |
| Bytespider | ByteDance | 214 | 20.0% |
| GPTBot | OpenAI | 211 | 19.7% |
| ClaudeBot | Anthropic | 201 | 18.8% |
| Meta-ExternalAgent | Meta | 172 | 16.1% |
| Google-Extended | 170 | 15.9% | |
| Applebot-Extended | Apple | 162 | 15.2% |
These are the crawlers vendors say collect pages for model training. OpenAI says disallowing GPTBot indicates your content should not be used for training, and Anthropic says the same of ClaudeBot. Blocking tends to come as a package: of the 268 sites that block at least one of the six major training crawlers, 87% block two or more and 74% block three or more. That looks like a policy decision, not a one-off rule.
Search and answer crawlers: far fewer block them
Sites blocking AI search crawlers and user-triggered fetchers site-wide (n = 1,069)
| Crawler | Job | Sites blocking | Share |
|---|---|---|---|
| PerplexityBot | Builds Perplexity's search index | 124 | 11.6% |
| Claude-SearchBot | Improves Claude's search results | 87 | 8.1% |
| OAI-SearchBot | Surfaces sites in ChatGPT search | 72 | 6.7% |
| ChatGPT-User | Fetches a page when a ChatGPT user asks | 101 | 9.4% |
| Perplexity-User | Fetches a page when a Perplexity user asks | 96 | 9.0% |
| Claude-User | Fetches a page when a Claude user asks | 89 | 8.3% |
Overall, 131 sites (12.3%) block at least one of the three search crawlers, and 63 sites (5.9%) block all three. Those are the sites that have chosen to be invisible to AI answers, not just to AI training. Keep in mind that the user-triggered fetchers are a special case: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Blocking those two in robots.txt may do less than it appears to.
Blocking GPTBot is not blocking ChatGPT search
This is the most useful finding for anyone writing a robots.txt. Of the 211 sites that block GPTBot, 139 (65.9%) do not block OAI-SearchBot. That matches how OpenAI documents the two: the settings are independent, so a site can opt out of training while staying eligible for ChatGPT search. The remaining 72 sites block both, and OpenAI says a site that opts out of OAI-SearchBot will not appear in ChatGPT search answers, though it can still show up as a navigational link.
If you want to be cited, that split is the policy to copy: block the training crawlers if you object to training, and leave the search and user agents alone. Our guide to which AI crawlers to allow has an example file.
The accidental blockers
Thirteen sites (1.2%) block GPTBot only because of a wildcard: a User-agent: * group with Disallow: /, alongside separate groups that let in search engines such as Googlebot. That is a legitimate way to write a strict policy, but it silently shuts out every AI crawler the author did not think to name, including the ones that drive citations. If your robots.txt is an allowlist, check that it names OAI-SearchBot, PerplexityBot and Claude-SearchBot on purpose.
Bigger sites do not block more
Blocking by Tranco rank band
| Rank band | Sites analysed | Block GPTBot | Block any search crawler |
|---|---|---|---|
| 1-100 | 45 | 22.2% | 24.4% |
| 101-500 | 176 | 18.2% | 13.6% |
| 501-1,000 | 256 | 19.1% | 13.7% |
| 1,001-2,000 | 592 | 20.3% | 10.3% |
GPTBot blocking sits between 18% and 22% in every band. The one visible trend is that the very largest sites are likelier to block search crawlers too (24.4% in the top 100, against 10.3% in the 1,001-2,000 band), but only 45 sites in the top band had a readable robots.txt, so treat that as a hint, not a result. The sites blocking GPTBot include familiar names such as nytimes.com, bbc.com, reuters.com, cnn.com, amazon.com and medium.com.
A new directive: Content-Signal
77 sites (7.2%) already use a Content-Signal line in robots.txt. It comes from Cloudflare's Content Signals Policy and lets a site say, per use, whether it permits search, ai-input (answers and grounding) and ai-train. Cloudflare describes the signals as expressing preferences, not as technical countermeasures, and notes some companies may ignore them. We did not count Content-Signal lines as blocks, because they do not instruct a crawler to stay away.
llms.txt: adopted by the sites that want to be found
177 of the 1,069 sites (16.6%) serve an llms.txt. Adoption was highest in the 101-500 band (23.9%) and lowest in the 1,001-2,000 band (12.3%). Sites with an llms.txt block GPTBot far less often than the rest (4.5% against 22.8%), which fits the file being adopted mostly by developer-tools, SaaS and documentation sites that want AI visibility. That is selection, not proof that llms.txt works. In fact, an Ahrefs study of 137,210 domains found that 97% of llms.txt files received no requests in May 2026. We cover this in llms.txt explained.
How this compares with other 2026 studies
Share of sites blocking each crawler, by study. Samples and definitions differ, so compare patterns, not decimals.
| Study and sample | GPTBot | ClaudeBot | PerplexityBot | OAI-SearchBot |
|---|---|---|---|---|
| This study: 1,069 sites from the Tranco top 2,000, Nov 2026 | 19.7% | 18.8% | 11.6% | 6.7% |
| AICrawlerCheck: 4,962 robots.txt files from the Tranco top 10,000, Oct 2026 | 14.3% | 13.4% | 10.5% | 7.4% |
| Suff Digital, top 1,000 by Chrome UX Report rank, Sept 2026 | 23.1% | 21.6% | 14.1% | 12.6%* |
| Suff Digital, all 15.47M HTTP Archive sites, Sept 2026 | 8.7% | 8.3% | 1.7% | 2.1%* |
*Suff Digital reports OAI-SearchBot and ChatGPT-User as one figure, and counts a site as blocking when a robots.txt rule addresses that bot by name, so wildcard-only blocks are treated differently from our method. Three patterns hold across these studies: training crawlers are blocked more than search crawlers, popular sites block far more than the web at large, and many GPTBot blockers still allow ChatGPT search. On that last point the size varies: 65.9% in our sample, and about half (347 of 710) in AICrawlerCheck's. The safe summary is one half to two thirds.
What to do with this
- Decide training and search separately. Blocking GPTBot, ClaudeBot or Google-Extended is a training decision. Blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot removes you from answers.
- If your robots.txt is an allowlist behind
User-agent: *, add the search crawlers explicitly. - Check your CDN, not just robots.txt. 137 of the 2,000 sites answered our request for robots.txt with 403 Forbidden even though we sent an ordinary browser user agent. A firewall or bot-protection rule can stop a crawler before it ever reads your file.
- Test the file with a real parser. A rule that lands in the wrong group applies the wrong policy without any error. The free audit on the home page shows which AI crawlers your robots.txt currently blocks.
Method and limitations
- Sample: the first 2,000 domains of the Tranco top-1M list, downloaded on 4 November 2026. Many top domains are not websites (CDN, DNS and ad-network hostnames), which is why only about half produced a usable robots.txt.
- Collection: for each domain we requested https://domain/robots.txt, then https://www.domain/robots.txt, with a standard browser user agent. Outcomes: 1,243 returned HTTP 200, of which 1,104 were genuine robots.txt files (the rest were HTML pages). 495 gave no response at all (typical of CDN, DNS and ad hostnames, though some were timeouts), 137 returned 403 and 94 returned 404.
- Exclusions: 35 hosts whose robots.txt blocks Googlebot as well are APIs, ad servers or CDNs rather than publishers making an AI-specific choice, so we excluded them. That leaves 1,069 sites.
- Parsing: we follow RFC 9309. A crawler uses its own group if one exists and the
*group otherwise, and counts as blocked only when the result is a whole-siteDisallow: /with no allow for the root. Partial blocks such asDisallow: /private/do not count. Python's standard-library parser disagreed with ours on about 2% of sites, mostly because it ends a group at a blank line, which RFC 9309 does not. - llms.txt: counted when
/llms.txtreturned HTTP 200, was not HTML, and began with a markdown heading. 96 sites returned an HTML page with a 200 status and were not counted. - Robots.txt shows intent, not enforcement. A crawler can ignore it, and firewall rules are invisible in it.
- Bias: popular sites skew toward large companies, and the sites that blocked our scripted request are probably likelier than average to block crawlers, so these figures may understate blocking among the biggest sites. They say little about small sites.
Frequently asked questions
What percentage of websites block GPTBot?
Among the 1,069 popular sites we could analyse, 19.7% block GPTBot site-wide in robots.txt. That is a sample of popular sites, not the whole web, and the share for small sites is probably lower.
Does blocking GPTBot remove my site from ChatGPT?
No. According to OpenAI's documentation, GPTBot controls use of your content for training, while OAI-SearchBot controls whether you appear in ChatGPT search. The settings are independent, and 65.9% of the sites that block GPTBot in our data still allow OAI-SearchBot.
Should I block AI crawlers?
Block the training crawlers if you object to your content being used for training. If you want AI answers to cite you, do not block the search and user-triggered crawlers. See which AI crawlers to allow.
Can I get the data?
Yes. The per-site results are in this CSV, with one row per site and a 0 or 1 for each crawler.
Sources
- Tranco: a research-oriented top sites ranking
- OpenAI: crawler and user agent documentation
- Anthropic: does Anthropic crawl data from the web, and how can site owners block the crawler
- Perplexity: crawlers and user agents
- Cloudflare: the Content Signals Policy
- Ahrefs: we analyzed 137K sites, 97% of llms.txt files never get read
- RFC 9309: Robots Exclusion Protocol
- AICrawlerCheck: AI crawler robots.txt study 2026
- Suff Digital: AI crawler blocking rate
Originally published at Cited Lens. You can check which AI crawlers your own robots.txt blocks with the free audit.
Top comments (0)