If your robots.txt has no Disallow lines that apply to them, every AI crawler can already read your site. The ones that matter are GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, and the Google-Extended control token for Gemini and AI Overviews. Blocking any of them means that assistant cannot quote your pages. Everything below is the detail behind that sentence.
The full list, and what each one actually does
Checked 3 August 2026. Operators add and rename tokens without announcement, so treat this as a snapshot and re run the check on your own file rather than trusting any list, including this one.
- GPTBot OpenAI · General crawling for OpenAI, including model training · Crawler
- OAI-SearchBot OpenAI · The search index behind links shown in ChatGPT · Crawler
- ChatGPT-User OpenAI · A single fetch when a user asks ChatGPT to open your page · User triggered
- ClaudeBot Anthropic · Crawling for Claude · Crawler
- Claude-User Anthropic · A fetch made on behalf of a person using Claude · User triggered
- PerplexityBot Perplexity · The Perplexity index · Crawler
- Perplexity-User Perplexity · A fetch triggered by a user question · User triggered
- Google-Extended Google · Whether Gemini and AI Overviews may use what Googlebot fetched · Control token
- Googlebot Google · Google Search, and therefore AI Overviews at all · Crawler
- Bingbot Microsoft · Bing, which is the index behind Copilot · Crawler
- Applebot Apple · Siri and Spotlight · Crawler
- Applebot-Extended Apple · Whether Apple may use that content to train its models · Control token
- CCBot Common Crawl · An open dataset used as training input by many labs · Crawler
- meta-externalagent Meta · Meta AI training and products · Crawler
- Amazonbot Amazon · Alexa and Amazon shopping assistants · Crawler
- Bytespider ByteDance · TikTok search and Doubao · Crawler
Google-Extended and Applebot-Extended are not crawlers
Nothing ever arrives at your server calling itself Google-Extended. It is a token you put in robots.txt that answers a policy question: may the content Googlebot already fetched be used for Gemini and for grounding AI Overviews. Applebot-Extended works the same way for Apple.
This matters because it makes them safe to reason about. Disallowing Google-Extended costs you nothing in Google Search. Your rankings, your indexing and your crawl budget are untouched, because the fetch that already happened was Googlebot's. What you lose is presence in the answers Gemini generates and in AI Overviews. That is a real cost and it is a clean, separable one, which is more than can be said for most decisions in this area.
The reverse is worth stating too. If Googlebot is blocked, Google-Extended is irrelevant. A page nobody fetched cannot be used for anything.
Crawlers and user triggered fetchers are different questions
GPTBot crawls the web on OpenAI's schedule. ChatGPT-User fetches one page because a person in a chat asked for it. Anthropic and Perplexity draw the same line with Claude-User and Perplexity-User.
The distinction is the whole basis of the nuanced position many publishers want: refuse bulk crawling for training, allow the fetch that happens when a real person is reading you right now. That is expressible in robots.txt, and it is the configuration most businesses arrive at once they understand the two are separable.
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Allow: /
The rule that silently breaks most robots.txt files
A crawler obeys exactly one group. It picks the group whose user agent token is the longest match for its own name, and it then ignores every other group in the file, including the wildcard.
That single rule is behind most of the accidental blocks we see. Consider this file:
User-agent: *
Disallow: /private/
User-agent: GPTBot
Disallow: /
Every crawler except GPTBot stays out of /private/. GPTBot is blocked from everything, and is also free to crawl /private/, because the wildcard group does not apply to it at all. Now consider the more common accident:
User-agent: GPTBot
Allow: /
User-agent: *
Disallow: /
The author intended to welcome GPTBot. What they actually published blocks every other crawler on the internet, Googlebot included.
Within the group that applies, the longest matching path wins, not the first line in the file. Allow: /blog/ beats Disallow: / for a URL under /blog/. Our checker implements both rules, group selection by longest matching token and then longest match path evaluation, which is why it sometimes disagrees with a quick read of the file.
The two configurations worth copying
Allow everything, which is what a business that wants to be recommended should publish:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
That is it. No per bot allow lines are needed, because permission is the default and every crawler falls into the wildcard group.
Refuse training, keep retrieval, which is the publisher position:
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Read that second file carefully before deploying it. It is a real editorial position, not a safe default, and it does cost you some AI visibility on purpose.
So should you block them?
There is an honest case on both sides and anyone who tells you otherwise is selling something.
Block, if your revenue depends on the visit itself. News, recipes, reference content and anything monetised by advertising all share a shape: an assistant that answers the question has taken the transaction and left you the cost of producing the answer. Training crawlers take content for a product you are not paid for and are not credited in. Blocking is a coherent business decision and plenty of large publishers have made it.
Allow, if you sell something the answer leads to. For software, services, professional practices and most business to business companies, being named in an answer is the point. A buyer asking an assistant which tool to use is further down the funnel than a buyer typing a keyword into a search box, and being absent from that answer is a lost sale with no compensating upside. Nobody bought less software because ChatGPT was allowed to read the documentation.
The uncomfortable middle is a business that is both, and there the crawler and fetcher split above is usually the honest answer.
robots.txt is a convention, not a fence
Every major operator publishes documentation stating that their crawlers respect robots.txt. Independent research has repeatedly found fetches from paths that a robots.txt disallowed, sometimes from agents identifying themselves differently or from addresses not published as belonging to the operator.
Both things are true at once and it is worth being precise about the consequence. robots.txt is a request that well behaved software honours. It has no enforcement behind it. If a block is a preference, robots.txt is the right tool and it is sufficient. If a block has to hold, enforce it where it can be enforced: firewall rules, bot management at your CDN, or authentication in front of the content. Use robots.txt alongside that to state the intent clearly, because an operator that finds no stated preference has been given no preference to ignore.
How to check your own file
Reading robots.txt top to bottom is exactly how the group precedence mistake survives review. Evaluate it the way a crawler does, once per user agent.
Our AI crawler checker fetches your live robots.txt and does this for seven named bots, reporting which assistants are shut out by name rather than telling you something is wrong somewhere. It runs on a free account with no card. If the result surprises you, the fix is one commit and it takes effect on the crawler's next visit.
Two things worth doing in the same sitting. Confirm the crawlers can actually read the page once they arrive, because most AI crawlers do not run JavaScript and a client rendered site can be wide open and still effectively empty. Then check whether your brand is currently named in AI answers at all, which is the question all of this was ever in service of.
We wrote this article to the same standard our engine scores: the answer in the first sixty words, sections bounded by headings a model can chunk on, a table where a table is clearer than prose, and every claim either verifiable in a published document or marked as a snapshot with a date on it. You can run that check on this page.
Sources
- Every token in the table above comes from the operator's own documentation. Check these rather than us, because they change and we are a snapshot.
- OpenAI, bots and user agents, for GPTBot, OAI-SearchBot and ChatGPT-User.
- Anthropic, does Anthropic crawl the web, and how can site owners block it, for ClaudeBot and Claude-User.
- Perplexity, bots, for PerplexityBot and Perplexity-User.
- Google, common crawlers and the overview of Google crawlers, for Googlebot and Google-Extended.
- Apple, about Applebot, for Applebot and Applebot-Extended.
- Common Crawl, CCBot.
- IETF, RFC 9309: Robots Exclusion Protocol, for the group selection and longest match rules described above.
FAQ
Does blocking GPTBot remove my site from ChatGPT?
It stops OpenAI fetching new pages from your site, so anything you publish from that point on cannot be quoted. It does not erase what a model already absorbed during earlier training, and it does not remove pages on other sites that describe you. The effect is a slow fade rather than a disappearance.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls pages for OpenAI generally. OAI-SearchBot exists to build the search index that surfaces links inside ChatGPT. If you want to be cited with a clickable link but you do not want your content used for training, OAI-SearchBot is the one to allow and GPTBot is the one to disallow.
Does Google-Extended affect my Google Search rankings?
No. Google-Extended is a control token, not a crawler. It governs whether content Googlebot already fetched may be used to ground and train Gemini and AI Overviews. Disallowing it changes nothing about ordinary Search indexing or ranking.
Do AI crawlers actually obey robots.txt?
The major operators publish documentation saying their crawlers do, and independent research has repeatedly found fetches from disallowed paths. robots.txt is a request, not a fence. If a block has to hold, enforce it at the edge with firewall or bot management rules and use robots.txt to state the intent.
How do I check which AI crawlers are blocked on my site?
Fetch your robots.txt and evaluate it once per user agent, applying real group precedence rather than reading it top to bottom. Our free AI crawler checker does exactly that against seven named bots and tells you which assistants are shut out.
Originally published at https://visibility100x.com/blog/ai-crawlers-gptbot-claudebot-perplexitybot/
Top comments (0)