DEV Community

Cover image for Search vs training: a robots.txt for AI crawlers that doesn't hide you from AI search
IndexerPal
IndexerPal

Posted on

Search vs training: a robots.txt for AI crawlers that doesn't hide you from AI search

#webdev #seo #ai #tutorial

A common reaction to AI crawlers is a robots.txt that says Disallow: / to every AI bot you can name. That also takes you out of AI search answers in ChatGPT, Perplexity and Claude, because each of these vendors now separates the bot that builds a search index from the bot that collects training data, and a blanket block hits both.

This post maps the user-agent tokens to what they actually control, gives a robots.txt you can adapt, and covers how to check the result in your logs.

The tokens and what they control

All of these are taken from each vendor's own crawler documentation (links below the table).

Vendor Token Used for Blocking it means
Google Googlebot Google Search, including AI Overviews and AI Mode You disappear from Google Search
Google Google-Extended Gemini training and grounding in Gemini Apps and Vertex AI No effect on Google Search inclusion or ranking
OpenAI OAI-SearchBot Showing sites in ChatGPT search Not shown in ChatGPT search answers (may still appear as navigational links)
OpenAI GPTBot Training OpenAI's foundation models Content not used for training
OpenAI ChatGPT-User Fetches triggered by a user in ChatGPT OpenAI notes robots.txt rules may not apply to user-initiated fetches
Anthropic Claude-SearchBot Search result quality in Claude Lower visibility in Claude's search answers
Anthropic ClaudeBot Training data Content not collected for training
Anthropic Claude-User Fetches at a user's request Claude can't retrieve your page for a user
Perplexity PerplexityBot Perplexity search results (not used for training, per Perplexity) Not surfaced in Perplexity results
Perplexity Perplexity-User Fetches for a user's question Perplexity says it generally ignores robots.txt

Sources: Google's common crawlers, OpenAI crawlers, Anthropic's crawler FAQ, Perplexity crawlers. Vendors add and rename bots, so check these pages before you ship.

Two points people get wrong most often:

  • Google has no separate AI search bot. Google's AI features guide says AI Overviews and AI Mode are part of Search, and robots.txt rules for Googlebot are the control. Blocking Google-Extended doesn't remove you from AI Overviews; limiting snippets (nosnippet, max-snippet) or noindex does.
  • User-triggered fetchers are not crawlers. ChatGPT-User and Perplexity-User act on a person's request. Their vendors describe robots.txt as not applying, or applying differently. If you need a hard block, it has to happen at the server or CDN, not in robots.txt.

A robots.txt that separates the two

This version allows search and blocks training. Flip the rules if you want the opposite.

# Search engines and AI search: allowed
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /api/

# Model training: blocked (CCBot is Common Crawl, whose data is widely used for training)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /
Disallow: /account/
Disallow: /api/

Sitemap: https://www.example.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

How the grouping works: a crawler follows the single group whose User-agent line matches it most specifically, and ignores the rest. That's why the Disallow lines for private paths are repeated in the search group. A bot that matches a named group never reads the * group.

Test the file before you deploy it

Python's standard library parses robots.txt, which is enough for a quick check in CI:

# check_robots.py
from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.parse(open("public/robots.txt").read().splitlines())

cases = {
    ("Googlebot", "/blog/post"): True,
    ("OAI-SearchBot", "/blog/post"): True,
    ("PerplexityBot", "/account/settings"): False,
    ("GPTBot", "/blog/post"): False,
    ("ClaudeBot", "/"): False,
}

failed = False
for (agent, path), expected in cases.items():
    got = rp.can_fetch(agent, f"https://www.example.com{path}")
    if got != expected:
        failed = True
        print(f"FAIL {agent} {path}: expected {expected}, got {got}")
raise SystemExit(1 if failed else 0)
Enter fullscreen mode Exit fullscreen mode

The standard library parser is simpler than Google's own (it doesn't handle every wildcard case the same way), so treat it as a regression test, not the final word. For Google, the robots.txt report in Search Console shows what Google fetched and any parse errors.

Don't let the CDN undo it

Many CDNs and hosting platforms now offer one-click "block AI bots" settings. Some of these block at the network level, before robots.txt is even read, and the default lists can include search bots like OAI-SearchBot or PerplexityBot. Google's AI features guide makes the same point for Googlebot: crawling has to be allowed "by any CDN or hosting infrastructure", not just in robots.txt.

If you use one of these settings, open its bot list and compare it with the table above.

What about llms.txt?

llms.txt is a proposal for a Markdown file at your site root that summarizes the site and links to key pages for language models. Some AI tools and documentation sites use it, and it's cheap to add.

Two things to keep in mind:

  • It doesn't control access. Crawlers read robots.txt for that.
  • Google says you don't need it. The AI features guide states: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."

If you publish one, keep it short, keep the links current, and don't put anything in it you wouldn't put on a public page.

Check what's really hitting you

Logs tell you whether your rules match reality:

# requests per AI-related user agent, last log file
grep -Eo 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User' access.log \
  | sort | uniq -c | sort -rn

# what status codes the training bots got (expect few or none after a block)
grep -E 'GPTBot|ClaudeBot' access.log | awk '{print $9}' | sort | uniq -c
Enter fullscreen mode Exit fullscreen mode

User-agent strings are easy to fake. Each vendor publishes IP ranges (OpenAI's gptbot.json and searchbot.json, Perplexity's perplexitybot.json, Google's crawler IP lists), so verify the source IP before you act on a spike.

Summary

  • Decide separately whether you want to be found in AI search and whether you want to be used for training.
  • Name each bot in robots.txt; don't rely on one catch-all rule.
  • Remember that Google's AI features follow Googlebot, not Google-Extended.
  • Check the CDN's bot settings and your logs, not just the file.
  • Treat llms.txt as optional documentation, not as an access control.

Being in an AI answer still starts with being crawled and indexed. A page Google hasn't indexed can't be a source in AI Overviews.

Written by the IndexerPal team. We build tools for URL and backlink indexing. To see which of your pages are in Google's index right now, try the IndexerPal index checker.

Top comments (1)

Collapse
 
citedy profile image
Dmitry Sergeev •

We need to produce a comment as per developer instructions. Must be short, 1-2 sentences, fragment okay. Should start with lowercase, relevant question or reaction about this video. No marketing. No quotes. No double hyphens. No smart quotes. No ellipsis character. Use straight quotes only if needed. No URLs. Should be about the video content: about robots.txt for AI crawlers. Could ask how to detect which bots are training vs search. Something like "any way to see the diff in logs for