Blocking GPTBot Isn't Blocking ChatGPT Search: A robots.txt for the 3 Kinds of AI Crawlers

There's a robots.txt snippet going around titled something like "block all AI bots". You paste it, you feel better, you move on. The problem is that most of these lists mix two very different kinds of bots in one block: the ones that collect pages for model training, and the ones that fetch pages so an AI search product can show and cite you. Block the second kind and you can drop out of ChatGPT search or Perplexity answers, which for a small SaaS is often the opposite of what you wanted.

The vendors actually split these into separate user-agent tokens now. So you can say "don't train on my stuff" and "yes, show me in search" in the same file. This post is the config version: which token does what, a robots.txt you can adapt, the matching rule that breaks most hand-written files, and how to check what bots really get from your server.

The 3 kinds of AI crawlers

Roughly, every AI bot falls in one of three buckets:

1. Training: collects content that may be used to train models.
2. Search / answers: indexes pages so the product can surface and link them in AI answers.
3. User-triggered: fetches a page right now because a user asked a question or pasted a link.

Here's how the big vendors map to those, going by their own docs:

Vendor	Training	Search / answers	User-triggered
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Perplexity	none listed	PerplexityBot	Perplexity-User
Google	Google-Extended (control token)	Googlebot	n/a here
Apple	Applebot-Extended (control token)	Applebot	n/a here
Common Crawl	CCBot	n/a	n/a

A few details from the docs that matter when you write rules:

• OpenAI's crawler page says each setting is independent: you can allow OAI-SearchBot to appear in search results and disallow GPTBot for training. Sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers (they can still show up as navigational links). It also says changes take about 24 hours to kick in for search.
• ChatGPT-User is not used for automatic crawling, and OpenAI says robots.txt rules may not apply to it because a user started the request. It's also not what decides search inclusion, that's OAI-SearchBot.
• Anthropic's help article describes the same three-way split. Blocking Claude-SearchBot or Claude-User may reduce your site's visibility when people search through Claude. Unlike OpenAI and Perplexity, Anthropic says all three of its bots honor robots.txt, the user-triggered one included. They also say you need the rule on every subdomain you want to opt out, and that blocking their IPs isn't a reliable opt-out because then they can't read your robots.txt.
• Perplexity's bot docs say PerplexityBot is for surfacing and linking sites in search results and is not used to crawl for AI foundation models. Perplexity-User generally ignores robots.txt, since a user requested the fetch.
• Google-Extended and Applebot-Extended don't crawl anything themselves. They are tokens you put in robots.txt to say how content already fetched by Googlebot or Applebot may be used. Per Google's crawler list, Google-Extended doesn't affect inclusion or ranking in Google Search, but note it covers more than training: it also controls grounding in Gemini Apps and Grounding with Google Search on Vertex AI. So "block Google-Extended" isn't purely a training opt-out, it's also a "don't use me to ground Gemini answers" switch. Decide if you're fine with that.
• Apple's Applebot page says pages that disallow Applebot-Extended can still be in search results.

The one thing never to do: block Googlebot (or Applebot) to keep your content out of Gemini or Apple Intelligence. That takes you out of normal search, and the AI opt-out has its own token anyway.

A robots.txt that blocks training and keeps AI search

Adapt the paths to your app. The idea is three groups: training bots out, search bots in (minus private stuff), everyone else on your normal rules.

# 1) Training: opt out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# 2) AI search / answer bots: allowed, private paths still private
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /app/
Disallow: /api/

# 3) Everyone else, incl. Googlebot and Bingbot
User-agent: *
Disallow: /admin/
Disallow: /app/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Several User-agent lines stacked on top of one set of rules is valid. RFC 9309 defines a group as one or more user-agent lines followed by rules, so you don't have to repeat Disallow: / five times.

If you want the opposite policy (in training data is fine, you just want everything open) you don't need AI groups at all. A plain User-agent: * with your private paths covers every bot. That's what we do on postingdude.com: all 23 bots in our checker are allowed, because we want to be cited in AI answers.

The matching rule that breaks hand-written files

This is the part most people get wrong. A crawler does not read your * group and then add its own group on top. Per RFC 9309, it finds the group(s) that match its token and obeys those. Only if no group matches does it fall back to User-agent: *.

So this file:

User-agent: *
Disallow: /admin/
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /

means OAI-SearchBot is allowed on /admin/ and /api/ too. Its own group matched, so the * rules are ignored for it. That's why the example above repeats the private paths inside group 2. Every time you add a named group, copy your private Disallow lines into it.

Two other rules from the RFC worth knowing:

• Inside a group, the longest matching path wins. Allow: / plus Disallow: /admin/ means /admin/ is blocked.
• If your robots.txt returns a 4xx (like 404), crawlers may crawl everything. If it returns a 5xx or times out, crawlers must assume everything is disallowed. A robots.txt route that 500s once in a while is a quiet way to vanish.

Crawlers can cache robots.txt for up to 24 hours, so a fix isn't instant either.

Gotcha: the robots.txt you wrote isn't always the one being served

If you're behind a CDN or a host with bot features, check what's actually served, not what's in your repo.

Cloudflare's AI bot docs are a good example of how much can happen outside your file:

• It has separate AI bot policies for Search, Agent and Training, each with Block, Block on pages with ads, or Allow.
• Per those docs, from September 15, 2026 new domains get defaults where Training and Agent bots are blocked on pages that show ads, and Search stays allowed. Their "Agent" bucket includes chat fetch bots, which sounds like the ChatGPT-User / Claude-User kind of traffic.
• Its managed robots.txt feature prepends Cloudflare's own lines in front of your existing file. So the live file can have rules you never wrote.

And firewall or bot-fight rules work at a different layer. Your robots.txt can say Allow: / all day while a WAF rule hands the real bot a 403 or a challenge page. Perplexity's docs even include a section on allowing their bots through Cloudflare and AWS WAF by user agent plus their published IP ranges, which tells you how often this happens.

Also: robots.txt is voluntary. It states a preference. Bad scrapers ignore it, and if you need to actually stop something, that's a firewall job, not a robots.txt job. And listing /admin/ in a public file tells everyone it exists, so private pages need real auth, not a Disallow line.

How to check what bots really get

1. Fetch the live file from outside and diff it against your repo.

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt
diff <(curl -s https://example.com/robots.txt) public/robots.txt

You want a 200 (or a 404 if you truly want no rules). A 5xx here is the "everything disallowed" case from above. Do it for www, the apex and every subdomain you care about, each host has its own robots.txt.

2. Smoke-test with the bot's user agent.

curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" https://example.com/pricing

That UA string is the example from OpenAI's docs (version number may change). If you get a 403 or a challenge page, some rule is matching on the user agent. Treat a 200 as "probably fine", not proof. Bot rules on CDNs often verify real bots by IP, so a fake UA from your laptop isn't the same request as the real bot from its real network. It can be blocked as an impostor while the real one passes, or the other way.

3. Read your access logs. This is the real answer. Count requests per bot (nginx/Apache combined format):

for b in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User CCBot Googlebot bingbot; do
  printf "%-18s %s\n" "$b" "$(grep -ci "$b" /var/log/nginx/access.log)"
done

Then look at status codes for one bot. Lots of 403s for a search bot you meant to allow is your answer:

awk -F'"' '/OAI-SearchBot/ { split($3, a, " "); print a[1] }' /var/log/nginx/access.log | sort | uniq -c

If you're on a platform without raw logs, the CDN's firewall or bot analytics view usually shows the same thing (which rule fired for which user agent).

4. Check the IP is the real bot. Anyone can send GPTBot in a header. OpenAI publishes IP ranges per bot (for example https://openai.com/searchbot.json), and so does Perplexity. Quick check:

import ipaddress, json, urllib.request

ip = ipaddress.ip_address("203.0.113.7")  # IP from your log
data = json.load(urllib.request.urlopen("https://openai.com/searchbot.json"))
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]]
print(any(ip in n for n in nets))

Google, Apple and Common Crawl document reverse DNS instead: run host <ip>, check the name ends in the vendor's domain (googlebot.com, google.com or googleusercontent.com for Google, applebot.apple.com for Apple, crawl.commoncrawl.org for CCBot), then run host on that name and confirm it points back to the same IP.

A faster first pass

Reading robots.txt by hand gets annoying once there are named groups, wildcards and a CDN prepending lines. I built a free AI Crawler Checker on Posting Dude for exactly this (disclosure: it's ours, no signup). You paste a domain and it reads the live robots.txt and shows, bot by bot, whether 23 crawlers are allowed or blocked, which group applied and the exact rule line that decided it. It groups them as training, search/answers (with Googlebot, Bingbot and Applebot as a baseline) and user-triggered, and it tells a missing robots.txt apart from an unreachable one.

For a real example, here's what it showed for nytimes.com on Oct 18: it blocks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot and DuckAssistBot, but still allows Googlebot and Bingbot. That's a deliberate choice by a big publisher: stay in Google and Bing, opt out of AI answer engines' search bots too, not only training. If you copied a list like that for a small SaaS, you'd probably be opting out of ChatGPT search without meaning to.

It reads the robots.txt rules. A firewall rule that blocks a bot won't show up in robots.txt, so steps 2 to 4 above still matter.

Allowing the bots is just step one. One of my launch sites scored 0/40 on AI recall while all its SEO checks were green, so access alone doesn't get you mentioned. If getting cited is the actual goal, there's a longer playbook on getting a startup cited in AI search answers on the Aura++ blog.

Quick checklist

• Training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) and search tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are decided separately.
• Never block Googlebot or Applebot to opt out of AI. Use the -Extended tokens.
• Every named group needs your private Disallow lines repeated inside it.
• robots.txt must return 200 or 404, never 5xx. One file per host.
• Diff the live file against your repo if a CDN is in front.
• Your logs, not your robots.txt, tell you what bots actually got.

Question for anyone behind Cloudflare or another CDN: when you curl your live robots.txt, is it the same file you have in your repo? And did you find a search bot blocked (OAI-SearchBot, PerplexityBot, Claude-SearchBot) that you never meant to block?