With the rise of generative search, webmasters face a crucial dilemma:
- On one hand: You want real-time AI search engines (like Perplexity, ChatGPT Search, and Claude) to crawl your pages and provide direct citations and backlink referrals to your product.
- On the other hand: You may want to restrict aggressive background data scrapers from consuming server bandwidth or training foundation models without attribution.
In 2026, standard robots.txt files with simple User-agent: * rules are no longer sufficient. Here is how to configure granular crawler access for modern AI bots.
🤖 Understanding the Major AI Crawlers
Not all AI bots serve the same purpose. Here is the distinction between search crawlers and training scrapers:
| Crawler | User-Agent | Primary Purpose | Best Practice |
|---|---|---|---|
| ChatGPT Search | OAI-SearchBot |
Real-time search queries in ChatGPT | Allow |
| Perplexity | PerplexityBot |
Real-time source retrieval & citations | Allow |
| Claude Search | ClaudeBot |
Anthropic Claude real-time web citations | Allow |
| Apple Intelligence | Applebot-Extended |
Apple AI web search indexing | Allow |
| OpenAI Training | GPTBot |
Foundation model training data | Configurable |
| ByteDance Scraper | Bytespider |
Aggressive TikTok/ByteDance scraper | Block/Rate-Limit |
📄 Recommended robots.txt Configuration Template
Here is a battle-tested template that allows high-value AI search crawlers while protecting your sensitive routes and blocking heavy scrapers:
# Standard Search Engines
User-agent: Googlebot
User-agent: Bingbot
Allow: /
# Generative Search Bots (Enable AI Citations)
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ClaudeBot
User-agent: Applebot-Extended
Allow: /
Allow: /llms.txt
Allow: /docs/
Disallow: /api/
Disallow: /admin/
# Restrict Aggressive Unlicensed Scrapers
User-agent: Bytespider
User-agent: CCBot
Disallow: /
# Sitemap Location
Sitemap: https://yourdomain.com/sitemap.xml
🛠️ Free Developer Tools for Testing & Generation
Instead of writing complex syntax by hand and risking unintended crawler lockouts, you can use these free utilities:
- 🤖 AI Robots.txt Generator: Interactive UI to toggle permissions for 15+ AI crawlers with one click.
- 🔍 AI Crawler Tester: Test whether specific AI bots (like PerplexityBot or GPTBot) are currently permitted or blocked on your live URLs.
- 📋 AI Crawlers Directory: Up-to-date registry of verified AI user-agents and IP ranges.
All of these tools are free and run client-side on GEOKit.
💡 Summary
Fine-tuning your robots.txt ensures your website remains discoverable in the AI search era without sacrificing your server resources. Take 5 minutes to audit your site today!
Top comments (0)