Deciding who may crawl you is now two separate decisions wearing one name, and confusing them is the most common self-inflicted visibility problem on the web.
Published by Suparank · 2026 · suparank.io
A few years ago, crawler policy was a settled subject: let search engines in, keep bad bots out, move on. AI agents broke that because they are not one kind of visitor. Some collect text to train models. Some fetch pages to build the index behind an AI search product. Some fetch a single page because a user asked an assistant to read it. These have genuinely different implications, and a robots.txt written as though they were the same thing will produce an outcome you did not intend. This reference sets out what is publicly documented, how to express it, and where the silent failures live. It is published by Suparank, which audits this surface as part of its GEO work.
The distinction that decides everything
Three purposes, three different consequences when you block them:
Training collection. Text is gathered to be used in model training. Blocking this is a content-licensing decision. It has no immediate effect on whether you appear in answers today, and its long-run effect on how well a model "knows" your brand is real but slow and unmeasurable.
Live retrieval / index building. Pages are fetched to populate the search index an AI product consults while answering. Blocking this removes you from the cited-source answers you are presumably trying to win. The effect is direct, and it arrives quickly.
User-initiated fetch. A specific page is fetched because a user pointed an assistant at it. Blocking this means that when a prospect asks an assistant to read your own page, the assistant cannot — a strange thing to prevent.
Almost every damaging configuration comes from wanting the first and accidentally getting all three: a blanket rule, a security vendor's "block AI bots" toggle, or a copied snippet from an article that predates the split.
Documented agents and what blocking each costs
The table below lists only agents whose purpose is publicly documented by their operator. Crawler documentation changes, so treat this as a starting point and confirm each string against the operator's own current documentation before you deploy a rule. If you encounter a user-agent string in your logs that you cannot find documented anywhere, do not assume its purpose from its name.
| User-agent | Operator | Purpose | Blocking it costs you |
|---|---|---|---|
GPTBot |
OpenAI | Collects web content for model training | Long-run, unmeasurable influence on what the model knows about you; no immediate loss of answer presence |
OAI-SearchBot |
OpenAI | Builds the index behind ChatGPT's search surface | Direct: your pages become unavailable as sources for ChatGPT's search-backed answers |
ChatGPT-User |
OpenAI | Fetches a page in response to a user's explicit request | A user who asks ChatGPT to read your page gets nothing |
ClaudeBot |
Anthropic | Web crawling for Anthropic's models | Exclusion from Anthropic's crawled corpus |
Claude-User |
Anthropic | Fetches a page on a user's direct request | Claude users cannot have your pages read to them |
PerplexityBot |
Perplexity | Indexing for Perplexity's search and answers | Direct: removal from Perplexity's citable index |
Google-Extended |
Not a crawler — a robots.txt token controlling use of already-crawled content in certain Google AI products | Content-usage restriction only; Googlebot crawling and ordinary Search are unaffected | |
Googlebot |
Core Search crawling, which also feeds AI Overviews | Everything: ordinary Search and AI Overviews alike. Never block this as an AI-policy measure |
Google-Extended is the one that confuses people
It is worth restating on its own, because it is regularly implemented wrongly. Google-Extended does not fetch anything and has no separate crawl. It is a usage control expressed in robots.txt syntax. Disallowing it does not remove you from Google Search, and it does not make you invisible to Googlebot. Conversely, blocking Googlebot in an attempt to opt out of AI features destroys your ordinary search presence and does not achieve the intended goal. These are different levers with different blast radii.
Syntax
Rules are per-agent groups, matched by the most specific applicable group — an agent that has its own group ignores the wildcard group entirely. That behaviour is the source of most surprises. A configuration that permits retrieval and user fetches while declining training collection looks like this:
User-agent: GPTBot Disallow: /
User-agent: OAI-SearchBot Allow: /
User-agent: ChatGPT-User Allow: /
User-agent: * Disallow: /admin/
Because GPTBot has its own group, it never reads the wildcard group. Every agent you name needs its complete instructions inside its own block.
Two further points. First, robots.txt is a request, not an enforcement mechanism; it is honoured by operators who choose to honour it and ignored by everyone else, so it is a policy statement, not a security control. Second, the file must be served at the root of each host and scheme you care about — subdomains have their own robots.txt, and a rule on the apex domain does not cover them. If you would rather not hand-write these groups, the AI crawler robots.txt generator produces per-agent blocks with the training and retrieval distinction made explicit.
Server-side rules: the invisible failure
This is the part that costs organisations the most and is diagnosed the least, because nothing in robots.txt reveals it. Your edge can be refusing these agents while your published policy welcomes them.
WAF bot-management rules. Managed rulesets often classify unfamiliar automated clients as bots and challenge or block them by default. An AI retrieval agent receiving a JavaScript challenge page fails, and records nothing useful about your site.
CDN and platform toggles. Several providers ship one-click "block AI crawlers" controls. They are legitimate, but they are usually blunt: they catch the retrieval agents alongside the training ones. If someone enabled one during a scraping scare, your visibility problem may date from that day.
Rate limiting. Aggressive per-IP limits can throttle a legitimate crawler into partial coverage. You get indexed inconsistently and it looks like a content problem.
Geographic and ASN blocking. Blanket country or cloud-provider blocks catch datacentre-hosted crawlers indiscriminately.
Consent walls and interstitials. If the substantive content only appears after an interaction, what the agent stores is your consent notice.
Verifying that you are actually reachable
Policy is what you wrote; reachability is what happens. Check the second.
Fetch your own pages with each agent's user-agent string and inspect the status code and the returned body. A 200 that contains a challenge page is a failure dressed as a success.
Compare served HTML with rendered output. If your main content only exists after client-side hydration, an agent that does not execute scripts sees an empty shell. Disable JavaScript and read the page.
Test every hostname and both schemes, including www and non-www, plus any subdomain carrying documentation or product content.
Read your access logs for the documented agents. Absence of any hits over a long period is evidence of blocking, not of disinterest.
Ask an assistant to read one of your URLs directly. A refusal or a summary that describes your cookie banner tells you immediately that something upstream is interfering.
The agent readiness checker runs several of these checks against a live URL, and where the cause turns out to be structural rather than a single rule, a full GEO and technical audit covers rendering, structure and access together.
Default position for most commercial sites: allow retrieval and user-initiated fetch agents everywhere, decide on training collection deliberately as a content-licensing question, never block Googlebot as an AI measure, and audit the edge before concluding that your robots.txt is the whole story.
Making the trade-off
There is a real tension for publishers whose product is the text itself: training collection offers them nothing and may substitute for their traffic. For most software and service businesses the calculation is different, because being described accurately in an AI answer is a distribution channel, not a leak. Either way, the decision should be made once, written down, and expressed as separate rules per purpose — not inherited from a snippet someone pasted in 2024. Review it when operators publish new agents, because they do, and a stale file quietly makes the decision for you. More on the measurement side of this work is at suparank.io.
Suparank is a chat-first AI visibility and GEO platform. It runs your buyer prompts through ChatGPT, Gemini, Claude and Google AI Overviews and reports mentions, citations, sentiment and share of voice against named competitors, alongside GEO and technical audits and connected GA4 and Search Console data. It does not do rank tracking. Free tier available, no credit card required. https://suparank.io/ · marketing@suparank.io · Chase Allen, SEO/GEO Specialist.
Originally published by Suparank.
Top comments (0)