DEV Community

AURA GEO
AURA GEO

Posted on

Allow AI Search, Opt Out of Training: A Granular robots.txt You Can Test in Python

#robotstxt #seo #ai #python

Most robots.txt advice about AI comes in two flavors: "block all AI bots" or "allow everything." Both skip the useful middle option. You can let AI search and user-requested fetches reach your pages while telling the training crawlers to stay out.

That only works if you know which token does which job, and if your file actually matches the way crawlers read groups. This post covers both, then tests the result with about 30 lines of Python and a couple of curl calls.

Everything below comes from the vendors' own documentation as of October 2026. Tokens and policies change, so check the linked pages before you deploy.

Three jobs, three kinds of bots

Each large AI operator separates crawling by purpose. Blocking one token doesn't block the others.

Search and indexing (they decide whether you can be shown and linked in answers):

  • OAI-SearchBot: OpenAI says it is used "to surface websites in search results in ChatGPT's search features," and sites opted out of it "will not be shown in ChatGPT search answers" (OpenAI crawlers).
  • Claude-SearchBot: per Anthropic, blocking it "may reduce your site's visibility and accuracy in user search results."
  • PerplexityBot: "designed to surface and link websites in search results on Perplexity," and not used to crawl content for AI foundation models (Perplexity crawlers).
  • Bingbot: Bing's standard crawler (Bing crawlers).
  • Googlebot: Google says robots.txt rules for Googlebot are the control for Search, including AI Overviews and AI Mode (AI features and your website).
  • Applebot: powers search in Spotlight, Siri and Safari (About Applebot).

User-triggered fetchers (they load a page because a person asked a question just now):

  • ChatGPT-User: OpenAI says that "because these actions are initiated by a user, robots.txt rules may not apply."
  • Perplexity-User: Perplexity says that "since a user requested the fetch, this fetcher generally ignores robots.txt rules."
  • Claude-User: Anthropic describes it as a way for site owners to control user-initiated access, and says its bots honor robots.txt.

Training and dataset controls:

  • GPTBot: disallowing it "indicates a site's content should not be used in training" OpenAI's foundation models.
  • ClaudeBot: restricting it signals that future content "should be excluded from our AI model training datasets."
  • Google-Extended: not a crawler. It's a control token with no user agent of its own. It governs use of Google-crawled content for training future Gemini models and for grounding in Gemini Apps and Vertex AI, and Google says it "does not impact a site's inclusion in Google Search" (Google's common crawlers).
  • Applebot-Extended: also doesn't crawl. Disallowing it opts you out of training Apple's foundation models, and those pages "can still be included in search results."
  • CCBot: the crawler of Common Crawl, a non-profit that publishes an open crawl archive anyone can use. Blocking it keeps future pages out of that archive.

Two operators don't fit this pattern. Microsoft has no separate training token in robots.txt. Its controls for Copilot and training are the noarchive and nocache robots meta values (Bing robots meta tags). Apple has a fallback rule: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." A Googlebot-only group therefore quietly governs Applebot too.

How crawlers pick a group (RFC 9309)

RFC 9309 is the robots.txt standard. Four rules from it cause most of the surprises:

  1. A crawler obeys the group that names its product token. Matching is case-insensitive. The * group applies only "if no matching group exists."
  2. A named group doesn't inherit anything from *. If OAI-SearchBot has its own group, every line under User-agent: * is irrelevant to it.
  3. Groups naming the same token are merged. Two User-agent: GPTBot blocks combine into one rule set (the RFC has an example). Many homemade parsers get this wrong, so don't depend on it. Keep one group per token.
  4. Within a group, the most specific path wins, meaning the longest match. If an Allow and a Disallow are equivalent, Allow should win.

Here's the bug rule 2 produces, and it's common:

User-agent: *
Disallow: /admin/

User-agent: OAI-SearchBot
Allow: /
Enter fullscreen mode Exit fullscreen mode

The author wanted "everyone stays out of /admin/, and OpenAI search is explicitly welcome." What they got: OAI-SearchBot obeys only its own group, so /admin/ is allowed for it. Every named group has to repeat the private paths.

A granular file

Here is a file that allows search and user fetches, keeps private paths private for everyone, and opts out of training:

# 1) Training and dataset opt-outs
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# 2) Search and user-triggered fetchers: allowed, same private paths
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bingbot
User-agent: Googlebot
User-agent: Applebot
Disallow: /admin/
Disallow: /cart/

# 3) Everyone else
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

A few notes:

  • Several User-agent lines in a row form one group. That's valid under the RFC and saves repetition.
  • The training group comes first on purpose. That's not because of the RFC, where order doesn't matter, but because of a Python quirk covered below.
  • Opting out of training is a business decision, not a GEO requirement. If you're happy to be used for training, delete group 1 and the * group covers those bots.
  • Remember what robots.txt can't do. ChatGPT-User and Perplexity-User say they may not follow it. To actually stop a client, you need server or WAF rules.

Test it with Python

Python's standard library includes urllib.robotparser. This script prints an allow/block matrix for every token against a few paths. It works on a local file or a live URL:

import sys
import urllib.robotparser

# python3 check_robots.py robots.txt
# python3 check_robots.py https://example.com/robots.txt
src = sys.argv[1]
rp = urllib.robotparser.RobotFileParser()
if src.startswith("http"):
    rp.set_url(src)
    rp.read()
else:
    with open(src, encoding="utf-8") as f:
        rp.parse(f.read().splitlines())

AGENTS = [
    "OAI-SearchBot", "ChatGPT-User", "GPTBot",
    "Claude-SearchBot", "Claude-User", "ClaudeBot",
    "PerplexityBot", "Perplexity-User",
    "Bingbot", "Googlebot", "Google-Extended",
    "Applebot", "Applebot-Extended", "CCBot",
    "SomeOtherBot",
]
PATHS = ["/", "/pricing", "/blog/post", "/admin/", "/cart/checkout"]

width = max(len(a) for a in AGENTS)
print(" " * width, *[p.ljust(14) for p in PATHS])
for agent in AGENTS:
    row = ["allow" if rp.can_fetch(agent, p) else "BLOCK" for p in PATHS]
    print(agent.ljust(width), *[r.ljust(14) for r in row])
Enter fullscreen mode Exit fullscreen mode

Running it against the file above (Python 3.13) prints:

                  /              /pricing       /blog/post     /admin/        /cart/checkout
OAI-SearchBot     allow          allow          allow          BLOCK          BLOCK
ChatGPT-User      allow          allow          allow          BLOCK          BLOCK
GPTBot            BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
Claude-SearchBot  allow          allow          allow          BLOCK          BLOCK
Claude-User       allow          allow          allow          BLOCK          BLOCK
ClaudeBot         BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
PerplexityBot     allow          allow          allow          BLOCK          BLOCK
Perplexity-User   allow          allow          allow          BLOCK          BLOCK
Bingbot           allow          allow          allow          BLOCK          BLOCK
Googlebot         allow          allow          allow          BLOCK          BLOCK
Google-Extended   BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
Applebot          allow          allow          allow          BLOCK          BLOCK
Applebot-Extended BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
CCBot             BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
SomeOtherBot      allow          allow          allow          BLOCK          BLOCK
Enter fullscreen mode Exit fullscreen mode

Run it against the buggy two-group example and you'll see OAI-SearchBot allowed into /admin/. That's the inheritance bug made visible.

Know the parser's limits

urllib.robotparser is handy, but it's not an RFC 9309 implementation. Reading its source shows three differences:

  • It uses the first matching group and matches substrings. It checks whether a group's token appears inside the name you pass in. When I first put the search group (which contains Applebot) above the training group, Applebot-Extended matched Applebot and came out as "allow." Putting the training group first fixed the test. Real crawlers match product tokens, but the lesson stands: test exactly the tokens you care about.
  • It uses the first matching rule, not the longest one. Overlapping Allow/Disallow pairs can give a different answer than the RFC.
  • It doesn't support * and $ path wildcards. Google documents those and many sites use them.

So use it as a fast smoke test for files with one group per token and no overlapping or wildcard rules. For anything more complex, also test with a parser that implements the standard, such as Google's open-source robots.txt library.

Test the live file with curl

The parser tells you what the file means. curl tells you what crawlers actually receive:

# 1. Status, content type, redirects
curl -sSI https://example.com/robots.txt

# 2. The exact body, as a crawler gets it
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/robots.txt

# 3. Compare with a browser user agent
diff <(curl -s https://example.com/robots.txt) \
     <(curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://example.com/robots.txt)
Enter fullscreen mode Exit fullscreen mode

What to look for, according to RFC 9309:

  • A 5xx response is the worst case. The file is "unreachable," and crawlers must assume complete disallow. A flaky origin or an edge rule that errors on bots can silently block everything.
  • A 4xx response means "unavailable," and crawlers may access anything. Your carefully written opt-outs simply don't exist.
  • Redirects: crawlers should follow at least five. A redirect to an HTML page (a login, a 404 template) gives them something that isn't a robots file.
  • Size: parsers must handle at least 500 KiB. Keep the file far below that.
  • Different body per user agent: a CDN or bot-protection layer may be rewriting or challenging the request.

Finally, allow for delay. The RFC says crawlers shouldn't use a cached copy for more than 24 hours. OpenAI says search changes can take "~24 hours." Perplexity says "up to 24 hours." Check again the next day.

Checklist

  • One group per token. No duplicated User-agent: X blocks.
  • Private paths repeated in every named group.
  • A deliberate decision on each training token. Don't rely on copied defaults.
  • Googlebot rules reviewed with Applebot in mind (Apple's fallback).
  • The live file returns 200, text/plain, and the same body for bots and browsers.
  • The matrix script run against the live URL after every deploy.

If you manage AI visibility with an agent, AURA GEO can generate a robots.txt patch from your site's real state. The tool description is explicit that it must be merged into your existing policy, never blindly swapped in. Run the script above on the merged result before you ship it. For background on the tokens themselves, see AURA's knowledge-base note on robots.txt for AI search.

Top comments (0)