robots.txt for AI Crawlers: Why Blocking GPTBot Doesn’t Remove You From ChatGPT

Originally published on the Merlonix blog.

Someone on your team read that AI companies are training on the open web, found a robots.txt snippet, and pasted it in:

User-agent: GPTBot
Disallow: /

It feels decisive. It is also almost certainly not doing what whoever pasted it thinks it does. GPTBot is OpenAI's training crawler — the one that gathers corpus data to train future models. It is not the crawler that fetches your pages when someone asks ChatGPT a question and ChatGPT goes and reads the live web to answer. That's a different user-agent (OAI-SearchBot), and this snippet doesn't touch it. So the site that "blocked ChatGPT" is still perfectly visible to ChatGPT's answer path — and a site that actually wanted to be cited in answers, but pasted a broader blocklist, may have quietly removed itself from the one crawler that mattered.

This is the whole problem with AI crawler control in 2026: the user-agents fall into categories with opposite consequences, they are named confusingly, and the copy-pasted blocklists floating around treat them all as one thing. Here's the actual map.

---

Two Kinds of AI Crawler (Three, Really)

Every AI crawler that hits your site is doing one of three jobs, and which one it is determines whether blocking it helps you or hurts you:

• Training crawlers gather text to train or fine-tune a model. Blocking one is a legitimate, common choice — you're opting your content out of the training corpus. It does not affect whether you show up in answers today, because the answer didn't come from your page; it came from the model's weights or from a separate live fetch.
• Answer-engine crawlers fetch your pages right now to answer a user's question — the live retrieval behind ChatGPT Search, Perplexity, and AI Overviews. If you block these, you disappear from the answer. This is the category most site owners actually care about and most blocklists accidentally break.
• User-action fetchers retrieve a specific URL because a human explicitly asked the assistant to read it ("summarize this page: …"). Blocking these means a user who hands your link to an assistant gets "I can't access that."

The trap is that a single vendor often runs one of each, under different names, and a blanket rule catches all of them or the wrong one.

The Per-Vendor User-Agent Map

These are the tokens that actually appear in the User-agent: line, grouped by what each one does. The names are the important part — a rule keyed to the wrong token silently does nothing.

Vendor	Training (corpus)	Answer engine / live fetch
OpenAI	GPTBot	OAI-SearchBot (ChatGPT Search)
Anthropic	ClaudeBot	Claude-User (fetch on a user's behalf)
Perplexity	—	PerplexityBot
Google	Google-Extended	Googlebot (classic index + AI Overviews)
Amazon	—	Amazonbot
Common Crawl	CCBot	—
ByteDance	Bytespider	—
Meta	meta-externalagent	—

Two rows deserve a callout because they are the ones people get backwards:

• OpenAI runs two. GPTBot trains; OAI-SearchBot answers. Blocking GPTBot opts you out of training and leaves you fully citable in ChatGPT. If your goal is "train on someone else's data, not mine, but keep me in the answers," that is exactly the right, precise move — block GPTBot, allow OAI-SearchBot. If your goal was "make ChatGPT not use my site at all," blocking GPTBot alone did not do it.
• Anthropic runs two, and one isn't really a crawler. ClaudeBot trains. Claude-User fires when a Claude user pastes your URL and asks Claude to read it. Blocking Claude-User doesn't protect you from training — ClaudeBot does that — it just breaks the experience for a human who deliberately handed your link to an assistant.

Google is the awkward one: Google-Extended is a training/AI opt-out token only — it does not control crawling or classic search indexing at all. Classic Googlebot governs both the normal index and AI Overviews, so there is no clean "keep me in search, out of AI Overviews" robots.txt lever the way there is for OpenAI. That's a limitation of the tokens Google publishes, not something a snippet can fix.

The Four Ways robots.txt Gets This Wrong

Once you see the categories, the common failures are obvious — and they're all silent, because a robots.txt mistake never returns an error. The file is valid; it just means something other than intended.

1. Blocking the trainer, thinking you blocked the answer engine

Disallow: GPTBot and expecting to vanish from ChatGPT. Covered above: you opted out of training and stayed in answers. Whether that's a bug depends entirely on which one you meant.

2. User-agent: * Disallow: / — the sledgehammer

A blanket disallow blocks every crawler, including OAI-SearchBot, PerplexityBot, Amazonbot — and Googlebot, so you've also just removed yourself from ordinary search. People reach for this thinking "AI"; it means "everything." If you want to block training but stay in answers and search, a global disallow is the opposite of that.

3. Copy-pasting a stale AI blocklist

The "block all AI crawlers" gists that circulate are frozen in time. They list GPTBot, CCBot, anthropic-ai (a token Anthropic has since split into ClaudeBot/Claude-User), and miss OAI-SearchBot and PerplexityBot entirely — so they block yesterday's trainers and let today's answer engines straight through, achieving neither the privacy goal nor the visibility goal. Token lists drift; a blocklist is only as current as the day it was written.

4. Getting the matching rules wrong

robots.txt user-agent matching (RFC 9309) is token-based and case-insensitive, and the most specific matching group wins — a bot obeys its own named group and ignores User-agent: * if a group names it. So this does not do what it looks like:

User-agent: *
Disallow: /private/

User-agent: OAI-SearchBot
Allow: /

OAI-SearchBot now follows only its own group — which has no Disallow — so it can crawl /private/ too. The * group stopped applying to it the moment you named it. Per-bot groups replace the wildcard for that bot; they don't layer on top of it.

And a subtle one that isn't a rule mistake at all: a missing robots.txt is not a block. A 404 or 410 on /robots.txt means "there are no rules," which under RFC 9309 §2.3.1.3 is allow everything. If your plan was "we don't have a robots.txt so the AI crawlers can't get in," you have in fact invited all of them.

robots.txt Is Advice. The Edge Is Enforcement.

Here's the failure that survives a perfect robots.txt: you Allow: OAI-SearchBot, you double-check the syntax, and the crawler still can't read your pages — because a Cloudflare Bot Fight Mode rule, a WAF, or a "block AI bots" toggle is returning 403 to that user-agent at the edge, before the request ever reaches the logic that reads robots.txt.

robots.txt is a cooperative protocol — a polite request a well-behaved crawler chooses to honor. Your WAF is enforcement — it drops the connection regardless of what robots.txt says. The two are configured in different places by different people, and they routinely disagree. The marketing team allows the answer engines in robots.txt; the infra team enabled a managed "block AI scrapers" rule six months ago; and the site is invisible to ChatGPT Search with a robots.txt that swears it isn't.

You cannot see this by reading robots.txt. The only way to catch it is to actually fetch a page as each crawler's user-agent and check the status code — which is a different question from "what does robots.txt permit," and the one that reflects reality.

Getting Allowed In Isn't the Same as Being Understood

Suppose an answer engine can now reach your pages. Two more things determine whether it can actually use them:

• /llms.txt — an emerging convention: a plain-Markdown file at your root that points an AI agent at your most important content (docs, pricing, key pages) instead of making it reverse-engineer your nav. It's a guide file, not a gate; it doesn't grant or deny access, it just makes a site legible to an agent that already has access. Cheap to add, and nothing else does the job.
• schema.org JSON-LD structured data — the <script type="application/ld+json"> blocks that state, in machine terms, what a page is (an Article, a Product, an Organization, a FAQ). An answer engine that has to infer meaning from rendered HTML does worse than one handed explicit structured data. This is the same structured data that's mattered for classic rich results for a decade; it now does double duty for AI answers.

Neither is access control — they're comprehension. But there's no point tuning them if a WAF rule is 403-ing the crawler two layers up, which is why the access question comes first.

How to Check What You're Actually Doing

Reason about it in the order the crawler experiences it:

1. Read your own robots.txt as a specific bot, not as a human. For each answer engine you care about (OAI-SearchBot, PerplexityBot, Claude-User, Amazonbot), find the most specific matching group and confirm it isn't disallowed — remembering that naming the bot makes it ignore User-agent: *.
2. Fetch a real page as that user-agent and check the status code, to catch a WAF/Bot-Fight rule overriding robots.txt at the edge:

   curl -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" -I https://example.com/

A 403 here with an Allow in robots.txt is the edge-override failure, invisible from robots.txt alone.

1. Confirm the comprehension layer. Is there a /llms.txt? Does the homepage carry any application/ld+json structured data? Both are curl-and-read checks.
2. Decide per category, not per "AI." Training and answers are separate levers. Write down which you want — "out of training, in answers" is the common one — then map it to the exact tokens above rather than a blanket rule.

If you'd rather not run all four by hand, the free AI Agent-Readiness checker does exactly this from outside your stack: it reads your /llms.txt, parses your /robots.txt per-bot (separating answer-engine access from training-crawler access, so blocking GPTBot or CCBot never counts against you), checks your homepage for JSON-LD, and — the part robots.txt can't tell you — actually fetches as each crawler to catch an edge rule that 403s a bot your robots.txt allows. No signup, one domain at a time.

---

The one-line version: "block the AI crawlers" is not a thing you can do, because "the AI crawlers" are three different jobs under a dozen different names, and the rule that opts you out of training is a different line from the one that keeps you in answers. Decide which you actually want — most sites want out of training and in the live answers — then key the rule to the exact user-agent, and verify at the edge that a WAF isn't quietly overruling the file.

Merlonix watches this the way it watches SSL, DNS, and domain expiry: continuously, and from outside your infrastructure, so a robots.txt edit, a new "block AI bots" WAF rule, or a redeploy that drops your structured data doesn't quietly lock the answer engines out weeks before anyone notices the referral traffic fall off. Run the free agent-readiness scan to see where a domain stands today, check its live SSL and DNS while you're at it, and browse the rest of the free tools. Getting into the answer is a configuration problem you can fix this afternoon — but only once you know which of the three crawlers you were talking to.