Originally published on Sitecheckly.
AI crawlers serve different purposes. Blocking GPTBot targets training access, while blocking OAI-SearchBot affects ChatGPT search. The same distinction applies to ClaudeBot and Claude-SearchBot. Google-Extended controls Gemini training, not Google Search or AI Overviews. Choose robots.txt rules by purpose rather than blocking every AI-related user agent.
Why does blocking training differ from blocking AI search?
In the survey, 9.2% of sites blocked training crawlers while still allowing AI search crawlers. That is a useful distinction if you want to limit training access without closing off search discovery.
The November 2026 Sitecheckly survey covered the Tranco top 10,000, using Tranco list 7462X. We could analyze 4,934 of 10,000 domains. The rest were mostly CDN, DNS and API hosts, sites that ask crawlers to stay away in robots.txt, or sites that blocked automated requests. The training-versus-search finding uses the 4,546 sites with a readable robots.txt. See the survey methodology.
Treat your crawler policy as separate decisions:
- Training: May an operator crawl your content for model training?
- Search: May a search crawler access your pages for search-related use?
- User-requested access: May a service fetch a page when someone explicitly asks for it?
A training opt-out is not the same as a search opt-out. Start with the outcome you want, then choose the matching user-agent groups.
Which AI crawler user agents should you know?
This AI crawlers list separates training, search and user-requested fetching. Google-Extended is included because it belongs in the policy decision, even though it is a control token rather than a separate fetching crawler.
The allow and block columns describe the whole-site patterns shown below. They are alternatives, not instructions to apply both.
| User-agent token | Operator | Purpose | Allow rule | Block rule |
|---|---|---|---|---|
| GPTBot | OpenAI | Model training | Allow all paths | Disallow all paths |
| OAI-SearchBot | OpenAI | ChatGPT search | Allow all paths | Disallow all paths |
| ChatGPT-User | OpenAI | Fetching on a user's request | Allow all paths* | Disallow all paths* |
| ClaudeBot | Anthropic | Model training | Allow all paths | Disallow all paths |
| Claude-SearchBot | Anthropic | Search | Allow all paths | Disallow all paths |
| PerplexityBot | Perplexity | Search | Allow all paths | Disallow all paths |
| Google-Extended | Gemini training control, not a separate crawler | Allow all paths | Disallow all paths |
For OAI-SearchBot vs GPTBot, the key difference is search versus training. ChatGPT-User is separate from both: it fetches content in response to a user's request. OpenAI notes that robots.txt rules may not apply to these user-initiated requests, so a ChatGPT-User block is not a guaranteed access restriction. See OpenAI's bot documentation.
For Anthropic, use the distinct tokens for distinct decisions: ClaudeBot is for training, while Claude-SearchBot supports search.
PerplexityBot is Perplexity's search crawler. Do not classify it as a training crawler just because it is associated with an AI service.
How do you allow or block an AI crawler in robots.txt?
Use a user-agent group to name the crawler, then add the path rule. In these examples, the root path covers the whole site under that robots.txt policy. The group and path matching rules come from RFC 9309.
To block AI training access through GPTBot:
User-agent: GPTBot
Disallow: /
To allow AI search access through OAI-SearchBot:
User-agent: OAI-SearchBot
Allow: /
These groups can coexist. Blocking GPTBot does not require blocking OAI-SearchBot.
Allow patterns for the listed tokens
Each group below expresses an allow-all policy for its named token. Select the groups that match your intended policy. The ChatGPT-User caveat above still applies.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Block patterns for the listed tokens
These are the equivalent disallow-all groups. This is a reference, not a recommended file to paste wholesale. Blocking every listed search crawler would work against a policy intended to preserve AI search access.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
For a training-blocked, search-allowed policy, select the block groups for GPTBot, ClaudeBot and Google-Extended. Select the allow groups for OAI-SearchBot, Claude-SearchBot and PerplexityBot. Make the user-requested access decision separately.
Check group selection and path precedence
Do not assume that the last line in the file wins. Under RFC 9309's matching rules:
- A crawler uses the groups matching its product token. Matching groups for that token are combined.
- The wildcard group is the fallback when no specific user-agent group matches. Its rules are not automatically added to a matching specific group.
- Within the applicable rules, the longest matching path wins.
- When equivalent allow and disallow rules match, the allow rule should be used.
For example, this group blocks GPTBot from the site except for the more specific public path:
User-agent: GPTBot
Disallow: /
Allow: /public/
An allow rule is permission under the robots.txt policy. It is not a promise that a service will crawl, use or cite the page. Keep access eligibility separate from the goal of earning a citation.
Does Google-Extended affect Google Search or AI Overviews?
No. To opt out of Gemini training through this control, use:
User-agent: Google-Extended
Disallow: /
Google-Extended is a robots.txt control token. It is not a separate crawler making requests under that name. Blocking it does not block Google Search or AI Overviews; Googlebot is the relevant Search crawler. See Google's crawler documentation.
Do not substitute a Googlebot block when your intended decision is only about Gemini training. Those controls address different purposes.
How can you check a page after changing crawler rules?
Start with the published robots.txt, not just the copy in your editor. Compare its user-agent groups with the policy you meant to publish.
Use this checklist:
- Training policy: Are GPTBot, ClaudeBot and Google-Extended treated as intended?
- Search policy: Are OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed or blocked deliberately?
- User-requested access: Have you considered ChatGPT-User separately, including its enforcement limitation?
- Rule matching: Does the page's path match a more specific exception?
- Google policy: Have you kept the Google-Extended decision separate from Googlebot access?
Then run the free, no-sign-up Sitecheckly AEO audit on the page URL. Each run audits one page: the URL you enter, not the whole site.
For context beyond crawler access, read the answer engine optimization explainer. Use it to separate the access question from the broader work of making a page useful in search answers.
What to do next
- Write down your training, search and user-requested access decisions.
- Publish only the groups that express those decisions.
- Check the published rules against the specific page you want assessed.
Frequently asked questions
Is ChatGPT-User the same as OAI-SearchBot?
No. ChatGPT-User fetches content in response to a user's request; OAI-SearchBot supports ChatGPT search. See OpenAI's bot documentation.
Does allowing an AI search crawler guarantee a citation?
No. An allow rule permits crawling under that robots.txt policy; it does not guarantee that a service will crawl, use or cite the page.
What happens if two robots.txt path rules conflict?
The most specific matching path wins. If equivalent allow and disallow rules match, the protocol says the allow rule should be used. See RFC 9309.
Want to see how your own site does? Run a free audit at sitecheckly.com.
Figures in this article come from Sitecheckly's own audit data; see the methodology.
Top comments (0)