Here is a robots.txt pattern worth checking for. A long block of AI bot names, copied from a forum thread, all set to Disallow: /. The goal is usually to stop content being used to train models. But the same founder often also wants ChatGPT and Perplexity to recommend their product.
Those two goals can live in the same file. A copied block like that often doesn't do it, because it blocks both kinds of bot.
Google's John Mueller touched on this in a recent interview reported by Search Engine Journal. He said AI systems "crawl slightly differently" because "they want to do it for training versus just like getting the freshest content." Then he added: "that's just is like another flavor of technical SEO."
So here is that flavor, written down. I went through the crawler docs from Google, OpenAI and Perplexity and sorted each bot by what it is for.
The two kinds of AI bot
Most AI companies now separate their bots by job:
- Search or answer bots fetch pages so the product can show and link them in answers. Block these and you can drop out of those answers.
- Training bots or tokens decide whether your content may be used to train models. Block these and, according to the docs below, your search visibility in that product is not the thing you are switching off.
There is also a third group: fetchers that run when a user asks a question. Their docs say robots.txt may not apply to them, so they are not a lever you control in the same way.
Google's AI features guide says robots.txt rules for Googlebot are the control for how your site is crawled for Search, and AI Overviews and AI Mode are part of Search. To limit what is shown from a page, it points to nosnippet, data-nosnippet, max-snippet and noindex.
Training and Gemini grounding are a separate control. Google's list of common crawlers describes Google-Extended as a robots.txt token that manages whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. The same page says it does not affect inclusion in Google Search and is not a ranking signal.
| Token | What it controls |
|---|---|
Googlebot |
Google Search, including AI Overviews and AI Mode |
Google-Extended |
Gemini training and grounding, not Search inclusion |
OpenAI (ChatGPT)
OpenAI's crawler docs say each setting is independent:
| User agent | What it is for |
|---|---|
OAI-SearchBot |
Surfacing sites in ChatGPT search. Opted out sites will not be shown in ChatGPT search answers, though they can still appear as navigational links. |
GPTBot |
Crawling content that may be used to train OpenAI's foundation models |
ChatGPT-User |
Visits triggered by a user's request. OpenAI says robots.txt rules may not apply, and it is not used to decide what appears in search. |
OpenAI also notes it can take about 24 hours for its search systems to pick up a robots.txt change.
Perplexity
Perplexity's bot docs list two:
| User agent | What it is for |
|---|---|
PerplexityBot |
Surfacing and linking sites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models. |
Perplexity-User |
Fetches when a user asks a question. Perplexity says it generally ignores robots.txt because a user asked for it. |
A robots.txt that matches both goals
If your goal is "show up in AI answers, but don't train on my content", a file built from those docs could look like this:
# Search and answer bots: allowed
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training controls: opted out
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Be aware of one tradeoff. Per Google's docs, blocking Google-Extended also opts you out of grounding in the Gemini app, not only training. If showing up in Gemini answers matters to you, you might choose to leave it allowed. That is the point of this whole post: make each line a decision, not a copy and paste.
robots.txt is not the whole story
Your firewall or CDN can block a bot that robots.txt allows. Google's AI features guide lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" as an SEO basic. OpenAI and Perplexity both publish IP ranges and recommend allowing requests from them. Perplexity's docs even include Cloudflare and AWS WAF steps, and warn that Cloudflare's Bot Fight Mode can still challenge its bot.
Three quick checks:
# 1. What does your live robots.txt actually say?
curl -s https://yourdomain.com/robots.txt
# 2. Are the search bots in your server or CDN logs at all?
grep -E "OAI-SearchBot|PerplexityBot|Googlebot" access.log | tail -20
# 3. Are they getting 200s, or 403s from a bot rule?
grep -E "OAI-SearchBot|PerplexityBot" access.log | awk '{print $9}' | sort | uniq -c
One caution from Google's docs: user agent strings can be spoofed. If you are writing allow rules, match the published IP ranges too, not only the name.
Then watch what happens
Changing robots.txt is easy. Knowing whether it helped is harder, because AI answers vary from one run to the next and changes take time to show up. I'd write down the date of the change, keep a fixed list of buyer questions, and check them again over the following weeks in ChatGPT, Perplexity and Google.
I work at Serpair, which tracks Google rankings and AI mentions over time so you can line them up against what you changed. If you want a starting point, our free SEO and AI visibility check shows where your domain stands today, no sign-up needed.
What does your robots.txt say about AI bots right now? I'm curious how many people have an AI block in there they don't remember adding.
Top comments (0)