Two years ago, most web developers only thought about one crawler: Googlebot. Today, there are at least half a dozen AI-specific crawlers hitting your site, and how you handle them directly affects whether your content appears in AI-generated answers.
This guide covers every major AI crawler active in 2026, what they do, and how to configure your site to work with (or block) each one.
The AI Crawler Landscape
Here are the crawlers you need to know about:
GPTBot (OpenAI)
-
User-agent:
GPTBot - Operator: OpenAI
- Purpose: Crawls the web to improve and train models behind ChatGPT and the OpenAI API. Also feeds ChatGPT's real-time browsing feature.
- IP ranges: Published at openai.com/gptbot-ranges.json
- Documentation: platform.openai.com/docs/gptbot
GPTBot is the highest-impact AI crawler. If you block it, your content is unlikely to surface in ChatGPT responses.
ChatGPT-User (OpenAI)
-
User-agent:
ChatGPT-User - Operator: OpenAI
- Purpose: Used specifically when a ChatGPT user triggers real-time web browsing during a conversation. Unlike GPTBot, this is user-initiated.
This is a separate user-agent from GPTBot. Blocking GPTBot does not block ChatGPT-User, and vice versa.
ClaudeBot (Anthropic)
-
User-agent:
ClaudeBot - Operator: Anthropic
- Purpose: Crawls content for Claude's training data and retrieval systems.
- Respects: robots.txt directives
Claude is increasingly used for research and analysis. ClaudeBot ensures your content can appear in Claude's knowledge base.
PerplexityBot
-
User-agent:
PerplexityBot - Operator: Perplexity AI
- Purpose: Feeds Perplexity's answer engine, which provides cited, real-time answers to search queries.
- Behavior: Aggressive crawler with high crawl rates. Respects robots.txt.
Perplexity is one of the fastest-growing AI search engines. Its answers always include source citations, making it valuable for traffic.
Google-Extended
-
User-agent:
Google-Extended - Operator: Google
- Purpose: Specifically for Gemini (formerly Bard) and Google's AI features. Separate from Googlebot, which handles traditional search indexing.
This is an important distinction. Blocking Google-Extended does not affect your Google Search rankings -- it only controls whether your content is used by Gemini and AI Overviews.
Amazonbot
-
User-agent:
Amazonbot - Operator: Amazon
- Purpose: Powers Alexa answers and Amazon's AI features.
- Documentation: developer.amazon.com/amazonbot
Often overlooked, but relevant if voice search via Alexa matters for your audience.
Other Notable Crawlers
| User-agent | Operator | Purpose |
|---|---|---|
Bytespider |
ByteDance | Training data for TikTok's AI features |
Applebot-Extended |
Apple | Apple Intelligence and Siri |
cohere-ai |
Cohere | Enterprise AI model training |
Meta-ExternalAgent |
Meta | Meta AI features |
Configuring robots.txt: Three Strategies
Strategy 1: Allow All AI Crawlers (Recommended for Most Sites)
If you want maximum AI visibility:
# AI Crawlers - Allow All
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: Applebot-Extended
Allow: /
# Standard crawlers
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
Strategy 2: Selective Access
Allow AI crawlers to access public content but protect certain sections:
User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /api/
Disallow: /admin/
Disallow: /premium/
User-agent: ClaudeBot
Allow: /blog/
Allow: /docs/
Disallow: /api/
Disallow: /admin/
Disallow: /premium/
User-agent: PerplexityBot
Allow: /blog/
Allow: /docs/
Disallow: /api/
Disallow: /admin/
Disallow: /premium/
Strategy 3: Block All AI Crawlers
If you want to opt out entirely (note: this reduces your AI search visibility to near zero):
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Beyond robots.txt: HTTP Headers
For more granular control, you can use HTTP headers:
# Nginx: Block AI training but allow browsing
location / {
add_header X-Robots-Tag "noai, noimageai" always;
}
Or in your HTML <head>:
<meta name="robots" content="noai, noimageai">
These directives are newer and not universally supported yet, but adoption is growing.
Verifying Your Configuration
After making changes, you need to verify that your configuration is actually working. Common mistakes include:
- A wildcard
Disallow: /rule that overrides specific AI crawler rules - CDN or WAF settings that block crawlers at the network level before robots.txt is even read
- CMS plugins that inject their own robots.txt rules
The fastest way to check is to use GEOScore's AI Crawler Access Checker. Enter your URL and it will show you exactly which AI crawlers are allowed and which are blocked, parsing your actual robots.txt file. For a comprehensive audit across all 11 GEO signals, the full scanner at geoscoreai.com provides a detailed report.
If you need to generate a properly formatted robots.txt from scratch, the AI Robots.txt Generator walks you through the options.
Crawl Budget Considerations
AI crawlers can be aggressive. If you are running a small server, you might need to manage crawl rates:
# Slow down specific crawlers
User-agent: PerplexityBot
Crawl-delay: 10
User-agent: Bytespider
Crawl-delay: 30
Note that Crawl-delay is not part of the official robots.txt standard and not all crawlers respect it. For guaranteed rate limiting, implement it at the server or CDN level.
Monitoring AI Crawler Activity
Check your server logs to see which AI crawlers are actually visiting:
# Find AI crawler hits in Nginx logs
grep -E "GPTBot|ClaudeBot|PerplexityBot|Google-Extended|ChatGPT-User" \
/var/log/nginx/access.log | \
awk '{print $1, $14}' | sort | uniq -c | sort -rn
This gives you a clear picture of crawl volume by crawler type and can help you spot issues like blocked crawlers that should be allowed.
The Practical Impact
Sites that properly manage AI crawlers see measurable differences in AI search citations. A study from early 2026 found that websites blocking GPTBot were cited 73% less often in ChatGPT responses compared to similar sites that allowed it.
The key takeaway: your robots.txt is no longer just a technical file. It is a strategic decision about your visibility in the next generation of search. Review it, configure it intentionally, and verify it works.
Top comments (0)