Most robots.txt files were written years ago to keep Googlebot out of /wp-admin/ or a staging folder. Since then, a dozen AI crawlers have arrived. Each has its own user-agent token, and each one controls something different. If your file never mentions them, you still have a policy: whatever each crawler's default behaviour is this year.
This post covers three things:
- what each of the common AI tokens actually controls, from each vendor's own documentation;
- the mistakes I keep finding in real files;
- a stdlib-only Python script that reports what your robots.txt tells each crawler.
Why I started looking
In September I fetched robots.txt from 34 Nepali business websites: trekking companies, hotels, clinics, news sites and agencies. Only seven named any AI crawler at all. The other 27 said nothing.
One dental college blocked GPTBot and Google-Extended, probably expecting that to keep it out of "AI answers". It does neither of the things people usually assume, which is the subject of the next section. The full study, with its method, is on my site.
Thirty-four sites from one country is a small sample. But what made those files go wrong isn't specific to Nepal: the user-agent tokens are poorly understood, and blanket rules get copied from security plugins.
What each token controls
The single most useful fact: training crawlers and search crawlers are separate tokens. Blocking one does not block the other. The list below reflects each vendor's documentation as of November 2026. Check the linked pages before you rely on it, because these change.
OpenAI (docs)
-
GPTBot: training. Blocking it tells OpenAI not to use your content to train its foundation models. -
OAI-SearchBot: ChatGPT search. Block this and you won't appear in ChatGPT's search answers, apart from bare navigational links. -
ChatGPT-User: fetches made because a user asked for a page. OpenAI says robots.txt rules may not apply to these, and that this token doesn't decide search inclusion; useOAI-SearchBotfor that.
Anthropic (help centre)
-
ClaudeBot: training. -
Claude-SearchBot: indexing for Claude's search results. -
Claude-User: fetches for a user's question.
Google (common crawlers)
-
Google-Extendedis a robots.txt token, not a separate crawler. It controls whether content Google has already crawled may be used for Gemini model training and grounding. - Google states that it doesn't affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search, so Google-Extended doesn't remove you from them. Googlebot and the normal snippet controls (
nosnippet,max-snippet) are what govern those.
Others
-
PerplexityBot: Perplexity's index. -
Applebot-Extended: whether Apple may use content for training. -
CCBot: Common Crawl, an open dataset that many other systems are built from. Blocking it has the widest and least predictable downstream effect.
Three mistakes I keep seeing
1. "Block AI" applied as one switch.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This opts out of OpenAI training and Gemini training. It does not remove the site from ChatGPT search, because that's OAI-SearchBot, or from Google's AI Overviews, because those are Search. If the goal was "don't train on us, but do cite us", this file does the first half. If the goal was "stay out of AI answers entirely", it achieves very little of that.
2. A comment that explains the wrong model.
One file I read carried a careful comment saying that disallowing Google-Extended would remove the business from AI results. It was a thoughtful file, still working from a wrong understanding of what the token does. Comments in robots.txt are good practice. They just need to be right.
3. Silence.
The commonest case. No AI token is named, so each crawler falls back on the User-agent: * group. If that group is permissive, everything is allowed. That's fine if you chose it. But defaults move: a hosting provider or CDN can start blocking AI bots at the edge without touching your file. So check your server logs or CDN bot settings as well as robots.txt.
A coherent policy in a few lines
For most businesses, being cited in AI answers is worth more than the training opt-out is worth to them. A typical "cite me, don't train on me" file looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Everything not named, including OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot, falls through to your * group.
If your content is the product, such as paid research, licensed photography or subscription journalism, blocking more is defensible. Either way, write the decision down on purpose.
Audit it: 40 lines of standard-library Python
The script below does three things:
- It fetches
/robots.txtfor each domain. - It reports which AI tokens the file names.
- It uses Python's own
urllib.robotparserto report which of them are blocked from the homepage. That last part catches the case where the*group blocks everything.
#!/usr/bin/env python3
"""Report what robots.txt tells each AI crawler, for a list of domains.
python3 ai_robots_audit.py example.com example.org
"""
import sys, urllib.request, urllib.robotparser
AGENTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot",
"Claude-User", "PerplexityBot", "Google-Extended", "Applebot-Extended", "CCBot"]
def audit(domain):
url = f"https://{domain}/robots.txt"
try:
req = urllib.request.Request(url, headers={"User-Agent": "robots-audit/1.0"})
with urllib.request.urlopen(req, timeout=15) as r:
text = r.read().decode("utf-8", "replace")
except Exception as e:
return {"error": type(e).__name__}
named = [a for a in AGENTS if a.lower() in text.lower()]
rp = urllib.robotparser.RobotFileParser()
rp.parse(text.splitlines())
blocked = [a for a in AGENTS if not rp.can_fetch(a, f"https://{domain}/")]
return {"named": named, "blocked_homepage": blocked}
if __name__ == "__main__":
for d in sys.argv[1:]:
r = audit(d)
if "error" in r:
print(f"{d:30} no usable robots.txt ({r['error']})")
elif not r["named"]:
print(f"{d:30} no AI crawler named: no policy" +
(f"; blocked anyway: {', '.join(r['blocked_homepage'])}" if r["blocked_homepage"] else ""))
else:
print(f"{d:30} names {', '.join(r['named'])}; homepage blocked for: "
f"{', '.join(r['blocked_homepage']) or 'none'}")
Its limits are worth knowing:
-
urllib.robotparserfollows the original robots.txt rules, not every extension Google supports. - The script only tests the homepage.
- Substring matching for "named" is deliberately loose.
It's meant for a fast first pass across many domains, not as a validator.
A note on llms.txt
llms.txt is a proposed convention: a plain-text summary of your site for language models. It's cheap to add. But it's a proposal, not a standard, and none of the major assistants has committed to reading it. It doesn't replace being crawlable by the search tokens above.
Takeaway
Open your robots.txt and search for OAI-SearchBot, Claude-SearchBot and Google-Extended. If none appear, you haven't made a decision; your defaults have. Five minutes with the vendor docs linked above is enough to make one on purpose.
Disclosure, per DEV's guidelines for AI-assisted articles: this post was drafted with an AI writing assistant from my own published study. Every claim was checked against the vendor documentation linked above, and the script was run against live sites before publishing.
I'm Diwakar Bhatt, an SEO specialist in Kathmandu. I publish small, reproducible studies of how Nepali websites work in practice, from technical SEO to AI crawler policies. More at diwakarbhatt.com.np.
Top comments (1)
lol never thought my robots.txt could have an AI policy, does this actually stop chat