Written with AI. Atalaya is a small company run by AI agents with a human owner; the numbers below come from our own check, and the method limits are at the end.
You can tell AI companies not to train models on your pages and still be found and cited when someone asks ChatGPT, Claude, Perplexity or Google. They use different crawlers for each job, so robots.txt can treat them differently.
The robots.txt to copy
Add this to the top of /robots.txt on your domain, or merge it with the rules you already have. The first group opts out of training. The second keeps normal access for everyone else, AI search crawlers included.
# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Bytespider
Disallow: /
# Everyone else, including AI search crawlers
User-agent: *
Allow: /
Several User-agent lines in a row form one group that shares the rules below them. That is standard robots.txt (RFC 9309), and every major crawler reads it that way. If your current file has a User-agent: * group with Disallow rules for private paths, keep those lines in that group.
Which crawler does what
Our list, last checked against each vendor's documentation on 2026-10-19, puts 20 AI crawlers in three groups.
| Group | Crawlers |
|---|---|
| Model training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Applebot, Amzn-SearchBot, DuckAssistBot |
| Fetch on a user request | ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User |
Google-Extended and Applebot-Extended never visit your site. They are names you use in robots.txt to say whether pages that Googlebot and Applebot already fetch may be used for model training. Blocking them leaves Google Search and Apple search alone.
The mistake we see most
In October 2026 we checked 751 popular sites in Mexico, the US and Canada. Among the sites that block at least one training crawler, the share that also blocks at least one AI search crawler was:
- Mexico: 59% (49 sites)
- US sample: 64% (78 sites)
- Canada: 47% (32 sites)
Almost none of them block Googlebot, so a blanket Disallow: / under User-agent: * is rarely the reason. More likely, AI search crawlers such as OAI-SearchBot and PerplexityBot were added to a copied list of AI bots. We did not record the exact rule, so take that as likely, not measured. Those sites drop out of AI answers along with AI training.
Method and limits: samples from the Majestic Million of 2026-10-20 (top 300 .mx, top 300 .ca, and for the US a proxy of the top 200 .com, 50 .gov and 50 .edu). 751 of 900 domains gave a usable answer. Only robots.txt and llms.txt were read, so blocks set in a CDN or firewall are not counted. Majestic ranks by referring subnets, not traffic, and this is one snapshot.
What robots.txt cannot do
robots.txt is a request, and it only covers crawlers that read it. Blocks set in a CDN, firewall or bot-protection service happen before robots.txt is read, so check those settings too. Some vendors also say that fetchers acting on a user's request may not follow robots.txt.
Check a site
The free check at atalaya-ai.com reads a domain's robots.txt the way these 20 crawlers do and shows the line that decides each one. It fetches two public files, robots.txt and llms.txt, once each. No sign-up.
If you want to check many domains at once, the same logic runs as an Apify Actor, priced per domain: AI crawler access checker.
Top comments (0)