DEV Community

Cover image for AI Crawlers in robots.txt: Should You Block or Allow Them?
Amrendra kumar
Amrendra kumar

Posted on

AI Crawlers in robots.txt: Should You Block or Allow Them?

#ai #llm #seo

Disclosure

I'm a developer who works with React, Next.js and AI agents, and I offer related SEO and AI services, so I have a business interest in this topic. I used AI tools to help with research and drafting, and I reviewed the content before publishing. Bot names and rules come from official docs and can change, so check them before you deploy.

Quick Summary

Block training crawlers, allow search crawlers. For most sites, the best setup is to disallow bots that collect data for model training (like GPTBot and ClaudeBot) and allow bots that power AI search (like OAI-SearchBot and Claude-SearchBot). This keeps your content out of training sets but still lets AI tools cite you.

Key Highlights

  • AI crawlers fall into three jobs: training, search, and user-triggered fetch.

  • Each bot has its own User-agent name, so you can block one and allow another.

  • robots.txt controls crawling. It does not hide pages or protect private data.

  • Blocking OAI-SearchBot removes you from ChatGPT search answers.

  • Who should read this: site owners, bloggers and developers. Who can skip: sites that already have a bot policy.

How AI Crawlers Work (and Why robots.txt Matters)

AI bots and web crawlers are automated programs that fetch your pages. Some collect text for LLM training data. Others fetch pages so that an AI search tool can display your link in a result.

This is the crawling that you already know from Googlebot, with a difference: the purpose varies depending on which bot. Hence, AI crawler management requires more than one rule.

Your robots.txt file is at the root of your site and uses robots.txt instructions such as User-agent, Allow and Disallow to indicate to each bot which resources it may fetch.

Should You Block AI Crawlers?

Block them only if you have a clear reason. There are three common ones:

  • You don't want your writing used for model training.

  • Bots cause server load or bot traffic you can't afford.

  • Your content is paid or licensed.

Blocking everything has a cost. If you want AI search visibility, such as getting cited in ChatGPT, you need to let the search crawlers in. Decide per bot, not per category of "AI."

GPTBot vs OAI-SearchBot: Can You Separate Training from Search?

Yes. OpenAI says each setting is independent. According to OpenAI's docs, OAI-SearchBot surfaces sites in ChatGPT search, and sites that opt out will not appear in ChatGPT search answers.
Bot Company Job Block it if...
GPTBot OpenAI Training You don't want training use
OAI-SearchBot OpenAI Search results You don't want ChatGPT search visibility
ChatGPT-User OpenAI User-triggered fetch OpenAI notes robots.txt may not apply here
ClaudeBot Anthropic Training You don't want training use
Claude-SearchBot Anthropic Search You don't want Claude search visibility
Claude-User Anthropic User-triggered fetch You don't want user-requested fetches
Google-Extended Google Gemini AI use You want to opt out of Gemini use

So yes, you can block GPTBot and still allow ChatGPT search.

Google-Extended robots.txt: What It Does

Google-Extended is a robots.txt token, not a separate crawler. Google documented it as a control over whether your content contributes to training of Gemini models and grounding.

It has no effect on your Google Search ranking or indexing. Blocking it will not make you invisible to Google Search.

Google also says that robots.txt controls crawling, not indexing. To keep a page out of search results, use noindex (the page must be crawlable for Google to see the tag), or put it behind a login.

ClaudeBot robots.txt: The Anthropic Example

Anthropic uses three user agents. ClaudeBot collects content that may be used for training. Claude-SearchBot improves search result quality. Claude-User fetches pages when a person asks Claude about them.

This split lets you opt out of training and stay visible in search, the same pattern as OpenAI.

Which AI Crawlers Should You Allow?

Allow AI search crawlers if you want traffic and citations. Most blogs and business sites gain from this.

  • Allow: OAI-SearchBot, Claude-SearchBot, PerplexityBot

  • Usually block: GPTBot, ClaudeBot, CCBot, Google-Extended (if you oppose training use)

  • Leave to the platform: ChatGPT-User, Claude-User (user-triggered)

If you want to optimize content for AI search, also keep pages fast, server-rendered, and easy to read. Bots that can't see your text can't cite it.

How to Block AI Bots Using robots.txt

Add this to https://yourdomain.com/robots.txt. It blocks training bots and allows search bots:


\# Block AI training crawlers


User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: CCBot

Disallow: /

User-agent: Google-Extended

Disallow: /

\# Allow AI search crawlers

User-agent: OAI-SearchBot

Allow: /

User-agent: Claude-SearchBot

Allow: /

User-agent: PerplexityBot

Allow: /
Enter fullscreen mode Exit fullscreen mode

# Everyone else (including Googlebot)

User-agent: \*

Allow: /

Sitemap: https\://yourdomain.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

To block every AI bot, switch the search bots to Disallow: /. OpenAI says search changes can take about 24 hours to apply.

Check That It Works

  1. Open /robots.txt in your browser and confirm the rules.

  2. Check server logs for bot names like GPTBot and OAI-SearchBot.

  3. Verify real bots using the IP lists the vendors publish (for example, OpenAI's gptbot.json).

If you use Cloudflare or another CDN, check that your firewall isn't blocking bots you allowed. CDN and firewall bot blocking often overrides robots.txt.

Common Mistakes to Avoid

  • Treating robots.txt as security. It is a polite request. Bad bots can ignore it.

  • A blanket Disallow: / under User-agent: *. This can block Googlebot too.

  • Blocking search bots by accident and then wondering why AI tools never cite you.

  • Using robots.txt instead of noindex. A blocked page can still appear in results if other sites link to it.

  • Never checking logs. You can't manage what you don't measure.

For related reading on how Google treats AI-made content, see my post on Google's September 2026 spam update.

FAQ

Does robots.txt affect ChatGPT search?

Yes. Allowing or blocking OAI-SearchBot decides whether your site can appear in ChatGPT search answers.

Can I block GPTBot but allow ChatGPT search?

Yes. Disallow GPTBot and allow OAI-SearchBot. They are independent settings.

Will blocking Google-Extended hurt my Google rankings?

No. Google says it doesn't affect Search inclusion or ranking.

How do I control AI crawlers on my website?

Use one User-agent block per bot in robots.txt, then confirm with server logs.

Does robots.txt hide my page from Google?

No. It controls crawling only. Use noindex or authentication to keep pages out of results.

How do I get cited by ChatGPT?

Allow OAI-SearchBot, publish clear and accurate content, and make sure pages load without heavy JavaScript.

Conclusion

You don't have to make the same decision for every AI crawler. If you don't want your website content used for AI model training, consider blocking the relevant training crawlers. If you want your content to be discoverable in AI-powered search results, allow the search crawlers you want to support.

Start by reviewing your website's robots.txt file and checking whether its rules match your SEO goals. Test your configuration before publishing, and remember that robots.txt is a crawler directive, not a security mechanism.

Need help improving your website's technical SEO or AI search visibility? Contact me or explore my SEO and Content Strategy services.

Top comments (0)