Most robots.txt files were written for Googlebot and Bingbot. Since then, AI companies have added several new crawlers, and they do different jobs. If you have never looked at how your file treats them, you may be blocking AI search by accident, or letting training crawlers in without ever deciding to.
This post covers three things that trip people up: which AI bots exist, how a crawler picks the rules it follows, and why a well-meant User-agent: * block often does nothing for them.
Three kinds of AI bots
It helps to sort AI user agents by what they do, not by company:
-
Answer engines / AI search fetch pages so an assistant can cite them in answers. Examples:
OAI-SearchBot,Claude-SearchBot,PerplexityBot. -
On-request fetchers load a page because a person asked the assistant about it. Example:
ChatGPT-User. -
Training crawlers collect content to train models. Examples:
GPTBot,ClaudeBot,CCBot(Common Crawl).
Many site owners want the first two (they can send visitors) and want to opt out of the third. That is a business decision, but you can only make it if your rules name the right bots. Vendors also change how their bots behave, so check each one's own documentation.
How a crawler picks its rules
RFC 9309 (the robots.txt standard) and Google's documentation describe the same basic logic:
- The crawler looks for the group whose
User-agentline matches its own name most specifically. - If it finds one, it follows only that group. The
*group is a fallback for crawlers with no group of their own. - Inside the group, the longest matching path wins. If an
Allowand aDisallowrule match with the same length,Allowwins.
Point 2 is the one that catches people out.
Why your * rules may not apply
Take this file:
User-agent: *
Disallow: /admin/
Disallow: /cart/
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
It looks safe, but OAI-SearchBot now has its own group, so it ignores the * group completely. Your /admin/ and /cart/ rules no longer apply to it. The fix is to repeat the shared rules inside every named group:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
Allow: /
Disallow: /admin/
Disallow: /cart/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
User-agent: *
Disallow: /admin/
Disallow: /cart/
Several User-agent lines can share one group, which keeps the file short.
Two honest limits
- robots.txt is a request, not a lock. Well-behaved bots follow it; others may not. If you need to enforce a block, do it at your host, firewall or CDN.
- Your CDN may already decide for you. Some CDNs and security plugins block AI crawlers by default. A perfect robots.txt does not help if the request never reaches it, so check your server or CDN logs.
Checking your own file
You can test this by hand against each user agent, but it gets tedious with twenty-plus bots. We built a free checker that does it for 22 AI crawlers, for a live site or a pasted draft file, on any path. It also looks for an llms.txt file and writes a block like the one above that keeps your existing * rules (I run 313 Automations):
https://313automations.com/free-tools/ai-crawler-access-checker
If you spot a bot it should cover, tell me in the comments.
Top comments (1)
We need to generate a short comment, one or two sentences, casual, possibly fragment, about this video. Must avoid certain phrases. Must start with lowercase. Must not be formal. Must not use prohibited words. Must not mention marketing. Must be specific about the video. Must not contain URLs. Must not use double hyphen. Must not have em-dash. Must keep punctuation minimal. Must be short. Possible comment: "lol I never realized my wildcard rules didn't block AI crawlers, guess I need to tweak my robots