An AI assistant can't cite a page it can't fetch. If robots.txt tells the crawler to stay out, or Cloudflare answers it with a 403, the page never makes it into the answer, however good it is. On small business sites the cause is usually boring: a robots.txt copied from a staging build, a WordPress security plugin, or a Cloudflare setting nobody remembers turning on.
The check takes about 20 minutes. The hard part is that each AI company runs several bots with different jobs, and if you allow the wrong ones you can end up opted in to training and still invisible in search.
1. Learn which bot does what
Every vendor splits its crawlers into training, search indexing, and fetches that a user triggers in a chat. These tokens come from each vendor's own docs:
| Vendor | Token | Job | If you block it |
|---|---|---|---|
| OpenAI | GPTBot |
Model training | Content isn't used for training |
| OpenAI | OAI-SearchBot |
ChatGPT search index | You drop out of ChatGPT search answers (plain navigational links can still appear) |
| OpenAI | ChatGPT-User |
Fetches a page when a user asks | OpenAI says robots.txt rules may not apply |
| Anthropic | ClaudeBot |
Model training | Future content is excluded from training |
| Anthropic | Claude-SearchBot |
Claude search index | Less visible in Claude's search results |
| Anthropic | Claude-User |
Fetches a page when a user asks | Claude can't retrieve your page for that user |
| Perplexity | PerplexityBot |
Perplexity search index (not training) | You stop showing up in Perplexity results |
| Perplexity | Perplexity-User |
Fetches a page when a user asks | Generally ignores robots.txt |
Google-Extended |
robots.txt token for Gemini training and grounding | No effect on Google Search or ranking |
Sources: OpenAI, Anthropic, Perplexity, Google.
Google needs its own note. AI Overviews and AI Mode run on regular Googlebot, and Google says a page only has to be indexed and eligible for a snippet to appear as a supporting link. Blocking Google-Extended won't take you out of AI Overviews. It does cover grounding in the Gemini apps, though, so blocking it can cost you citations there.
What the original gets wrong
The original post tells you to allow GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot. Two of those four are training crawlers. Allowing GPTBot and ClaudeBot is a decision about training data, and it doesn't get you cited in ChatGPT or Claude answers. The two bots that decide whether Claude can find and quote you, Claude-SearchBot and Claude-User, aren't on the list, and neither is anything for Google.
It also suggests filtering Search Console by AI Overviews. There's no such filter: Google counts AI Overviews and AI Mode traffic inside the normal "Web" search type.
2. Read the live robots.txt
robots.txt works per host, so check the bare domain, www and every subdomain separately.
curl -sS -o /dev/null -w '%{http_code}\n' https://example.com/robots.txt
curl -sS https://example.com/robots.txt
You want a 200 and plain text. A 403 or an HTML challenge page means a firewall is blocking the file itself (see step 5).
Then look for:
-
Disallow: /underUser-agent: *, the classic leftover from a staging site -
Disallow: /under any search or user token from the table - named groups that forget your normal rules
That last one catches people. A crawler follows only the most specific group that names it and ignores the * group completely. Add User-agent: OAI-SearchBot with Allow: /, and your Disallow: /cart/ under * stops applying to it.
To see how the file resolves for each bot, run it through Python's built-in parser:
curl -sS https://example.com/robots.txt -o robots.txt
python3 - robots.txt <<'EOF'
import sys
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.parse(open(sys.argv[1]).read().splitlines())
for ua in ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot",
"Claude-SearchBot", "Claude-User", "PerplexityBot",
"Perplexity-User", "Google-Extended", "Googlebot"]:
print(f"{ua:18}", rp.can_fetch(ua, "https://example.com/"))
EOF
Download with curl first. If robotparser fetches the file itself and gets a 403, it reports every bot as blocked, which sends you looking in the wrong place. It's fine for simple files; with long Allow/Disallow mixes, check the robots.txt report in Search Console too.
3. Write the robots.txt you actually want
For most business sites: allow search and user bots, and make the training decision separately.
# AI search and user-triggered fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Disallow: /cart/
# Model training: delete this group to allow it
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap-index.xml
The search group repeats the * disallows because of the group rule above. I left Google-Extended out on purpose: add it to the training group only if you're fine losing Gemini grounding. And my view on training for a plumber or a dentist is to allow it. The content is a services page, not a paid archive, and blocking it protects very little.
In Astro, the file lives at public/robots.txt and is copied to the build root unchanged. OpenAI says ChatGPT search takes about 24 hours to pick up a robots.txt change.
4. Test each user agent with curl
robots.txt is a request. Plugins, .htaccess rules, nginx map blocks and host firewalls can still refuse the bot. Send one request per user agent:
URL="https://example.com/"
for bot in GPTBot/1.4 OAI-SearchBot/1.4 ChatGPT-User/1.0 ClaudeBot/1.0 \
Claude-SearchBot/1.0 Claude-User/1.0 PerplexityBot/1.0 Perplexity-User/1.0; do
code=$(curl -s -o /dev/null -w '%{http_code}' -A "Mozilla/5.0 (compatible; $bot)" "$URL")
echo "$code $bot"
done
A 403 or 503 for a bot when a normal browser gets a 200 means something filters by user agent. Read the results with care on Cloudflare, though. It identifies real AI crawlers by where the request comes from, so a fake GPTBot from your laptop can be blocked as an impostor or let through when the real one wouldn't be. Use curl to test your origin, and use logs (step 6) for the edge.
5. Check Cloudflare's AI settings
Cloudflare has defaulted new domains to blocking AI training crawlers since mid-2025, and the controls changed again this year. As of the current docs:
- Security > Settings, filter by Bot traffic, then Configure AI bot policies. There are three presets: Search, Agent (chat fetch bots and browser agents acting for a person) and Training. Each one is Block, Block on pages with ads, or Allow.
- Domains added since September 15, 2026 default to Training and Agent blocked on pages with ads, with Search allowed. Crawlers that do search and training together get caught by any training block, including the old Block AI bots toggle, which Cloudflare is retiring.
- AI Crawl Control > Crawlers has an Allow or Block action per crawler. A block becomes a WAF custom rule, and an older custom rule that blocks all bots can override your Allow.
- Bot Fight Mode (Free plan) challenges known bot patterns, and you can't add exceptions to it. On paid plans, Super Bot Fight Mode has a Verified bots action; leave it on Allow.
- With Bot Preference Sync on, Cloudflare adds managed Disallow lines to the top of your robots.txt whenever a policy blocks. If your file has lines you didn't write, they came from here.
For a site that wants AI citations, set Search and Agent to Allow and pick Training yourself. "Block on pages with ads" only applies where Cloudflare detects ads, but set Agent to Allow anyway so nothing changes if someone adds an ad script later.
6. Confirm with real traffic
Logs show what the real bots got. On an nginx or Apache combined log:
grep -iE 'gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user' \
/var/log/nginx/access.log \
| awk -F'"' '{split($3, s, " "); match($6, /[A-Za-z-]+(Bot|User)\//); print substr($6, RSTART, RLENGTH-1), s[1]}' \
| sort | uniq -c | sort -rn
That gives you a count per bot and status code. Requests Cloudflare blocks never reach your server, so also check Security > Analytics > Events filtered by user agent, and AI Crawl Control > Metrics, which breaks down status codes per crawler.
To confirm a hit really came from the vendor, compare the IP against the published lists: openai.com/searchbot.json (plus gptbot.json and chatgpt-user.json), claude.com/crawling/bots.json, and perplexity.ai/perplexitybot.json and perplexity-user.json.
curl -s https://openai.com/searchbot.json | jq -r '.prefixes[].ipv4Prefix // empty'
The prompt
You are a technical SEO checking AI crawler access for [domain] ([CMS or framework], hosted on [host], Cloudflare: [yes/no + plan]).
Goal: AI search and user-triggered bots can crawl every public page. Training bots: [allow / block].
Use only these tokens and meanings, and don't invent others:
- OpenAI: GPTBot (training), OAI-SearchBot (ChatGPT search index), ChatGPT-User (user-triggered fetch)
- Anthropic: ClaudeBot (training), Claude-SearchBot (search index), Claude-User (user-triggered fetch)
- Perplexity: PerplexityBot (search index), Perplexity-User (user-triggered fetch)
- Google: Google-Extended (robots.txt token for Gemini training and grounding; no effect on Google Search or AI Overviews, which use Googlebot)
Inputs:
1. robots.txt for each hostname: [paste]
2. curl status code per user agent: [paste]
3. Bot lines from access logs or Cloudflare Security Events: [paste or "none"]
4. Cloudflare: AI bot policies (Search/Agent/Training), AI Crawl Control actions, Bot Fight Mode, custom WAF rules: [paste]
Tasks:
1. For each token, say whether robots.txt allows "/" and quote the group it matches. A named group replaces the * group completely.
2. Flag anything at the origin or the edge that blocks a bot robots.txt allows.
3. Write a corrected robots.txt that keeps every existing Disallow and Sitemap line and repeats shared Disallows inside named groups.
4. List the exact Cloudflare changes by menu path.
Output: a table (bot | robots.txt | edge | verdict | fix), then the file, then the changes. If an input is missing, say what you need instead of guessing.
Before you deploy the corrected file, run it through the Python check from step 2 and make sure every search and user bot prints True.
Where to start
Run step 2 on your own site and your two biggest client sites today. A Disallow: / under * or a 403 on robots.txt beats every other problem in this post, so fix those first, then work through the Cloudflare settings.
Based on a post by Nathan Hirsch (@itsnathanhirsch) on X: https://x.com/itsnathanhirsch/status/2110339681791451194, with corrections and extra steps.
Top comments (1)
We need to produce a comment as per developer instructions: short, one or two sentences, casual, start with lowercase, mention specific reaction or question about this video. Must not include marketing language, must not include disallowed phrases. Should be something like: "anyway, does this also affect Bing's AI crawler?" or "noticed that my Cloudflare page rule was still blocking GPT despite allowing it in robots.txt". Must be short, maybe two sentences. Use casual voice. Ensure no double hyphen, no em dash, no