AI assistants answer from two places: what the model learned in training and, when they search the web, the pages the search returns, which they show as citations. Vendors document the crawlers and controls (robots.txt tokens, location, search settings) but not how they rank sources, so treat any "ranking factor" list as a guess unless it cites the vendor.
This page sticks to what OpenAI, Perplexity, Anthropic and Google say in their own documentation, marks the places where the docs are silent, and gives you a script to check whether these assistants' crawlers may read your site.
Two ways an answer gets built
| Answer from model knowledge | Answer with web search | |
|---|---|---|
| Where brands come from | Patterns the model learned in training | Pages the search step retrieved for this question, plus what the model already knows |
| Sources shown | Usually none | Citations or a sources list |
| How current | As of the model's training data | As current as the pages retrieved |
| What you can influence | Slowly: being widely and consistently described across the web | Being on the pages that get retrieved, and letting the search crawler read your site |
Whether a given answer searched is often the model's own choice:
- ChatGPT can search the web on its own when a question benefits from current information, and you can also turn search on yourself; answers that used search show their sources (OpenAI help: searching the web with ChatGPT).
- OpenAI's API says the same about its web search tool: "the model can choose to search the web or not based on the content of the input prompt" (OpenAI web search guide).
- Perplexity says it uses AI "to search the internet in real-time" and that "each answer includes numbered citations linking to the original sources" (Perplexity help center).
That's why measuring visibility means recording, per answer, whether search was used.
Which pages the search step can use
Each vendor runs separate crawlers for search and for training, with separate robots.txt tokens. From their own documentation:
| Vendor | Token | What the vendor says it's for |
|---|---|---|
| OpenAI | OAI-SearchBot |
"used to surface websites in search results in ChatGPT's search features." "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." |
| OpenAI | ChatGPT-User |
Certain user actions in ChatGPT and Custom GPTs; "not used for crawling the web in an automatic fashion", and "robots.txt rules may not apply." |
| OpenAI | GPTBot |
Crawling content that "may be used in training our generative AI foundation models." |
| Perplexity | PerplexityBot |
"designed to surface and link websites in search results on Perplexity"; "not used to crawl content for AI foundation models." |
| Perplexity | Perplexity-User |
Fetches pages for user questions; "generally ignores robots.txt rules." |
| Anthropic | Claude-SearchBot |
"navigates the web to improve search result quality for users"; blocking it "prevents our system from indexing your content for search optimization." |
| Anthropic | Claude-User |
Retrieves content in response to a user query; blocking it prevents that. |
| Anthropic | ClaudeBot |
Training: blocking it signals that future materials "should be excluded from our AI model training datasets." |
Google-Extended |
Controls use for training future Gemini models and for grounding in Gemini Apps; it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." |
OpenAI adds that "it can take ~24 hours from a site's robots.txt update for our systems to adjust" for search results.
The practical point: blocking a training crawler doesn't remove you from AI search, and blocking a search crawler can. If you want to be cited in ChatGPT search answers, OpenAI's documentation says OAI-SearchBot must be allowed.
Check your own site
import sys
import urllib.robotparser
import requests
# User-agent tokens from each vendor's own crawler documentation.
AGENTS = {
"OAI-SearchBot": "ChatGPT search results",
"ChatGPT-User": "ChatGPT fetching a page for a user",
"GPTBot": "OpenAI model training",
"PerplexityBot": "Perplexity search results",
"Claude-SearchBot": "Claude search indexing",
"Claude-User": "Claude fetching a page for a user",
"ClaudeBot": "Anthropic model training",
"Google-Extended": "Gemini training and grounding (not Google Search)",
}
def check(site, path="/"):
url = f"https://{site}/robots.txt"
r = requests.get(url, timeout=20, headers={"User-Agent": "robots-check/1.0"})
rp = urllib.robotparser.RobotFileParser()
rp.parse(r.text.splitlines() if r.status_code == 200 else [])
print(f"{site}{path} (robots.txt HTTP {r.status_code})")
for agent, purpose in AGENTS.items():
verdict = "allowed" if rp.can_fetch(agent, f"https://{site}{path}") else "BLOCKED"
print(f" {agent:<17} {verdict:<8} {purpose}")
for site in sys.argv[1:]:
check(site)
python ai_robots.py datatooly.xyz on 2026-10-14:
datatooly.xyz/ (robots.txt HTTP 200)
OAI-SearchBot allowed ChatGPT search results
ChatGPT-User allowed ChatGPT fetching a page for a user
GPTBot allowed OpenAI model training
PerplexityBot allowed Perplexity search results
Claude-SearchBot allowed Claude search indexing
Claude-User allowed Claude fetching a page for a user
ClaudeBot allowed Anthropic model training
Google-Extended allowed Gemini training and grounding (not Google Search)
Pass your own domain. Python's urllib.robotparser follows the basic robots.txt rules but not every extension, so treat a surprising result as a reason to read the file yourself. A missing robots.txt (HTTP 404) means everything is allowed. Your CDN or firewall can also block these crawlers regardless of robots.txt; check its bot settings too.
Other inputs the vendors document
-
Location. Answers to local questions depend on where the asker is. The OpenAI API's web search tool accepts an approximate
user_location(country, city, region, timezone). The same "best dentist" question has different right answers in different cities. - Personalisation. Memory, chat history and custom instructions can shape answers in ChatGPT. OpenAI's shopping help article lists memory and custom instructions among the context it uses to pick products.
- Product results aren't ads. The same shopping article says product results are chosen independently and that ads are shown separately.
-
How much the search step reads. In OpenAI's API,
search_context_sizechanges how much retrieved material the model sees, and the docs say it "does not set an exact token count or guarantee a specific number of sources or citations."
What the vendors don't say
None of these vendors publishes how it ranks retrieved pages, how it picks which brands to name from them, or how training data shapes recommendations. Advice that lists precise "ranking factors" for ChatGPT or Perplexity is inference from experiments, not documentation. Some experiments are useful, but read them as observations that can change with the next model update.
What you can do with confidence follows from the documented parts:
- Let the search crawlers in (the table above), and check your CDN isn't blocking them.
- Find out which pages get cited for your buyer questions. Log the cited domains for every answer, as in How to check if ChatGPT mentions your brand. Those pages are what the search step currently reads for your category, whether they are yours, a competitor's, a review site or a forum.
- Make the facts easy to retrieve. Searched answers quote prices, plans and features from pages. Out-of-date pricing on your own site or a well-cited review page is a likely source of wrong answers about you.
- Measure, change one thing, and measure again with the same prompt set. Without a fixed baseline you can't tell a real gain from run-to-run noise; see AI share of voice for sample sizes.
Related guides
- How to check if ChatGPT mentions your brand (free).
- AI share of voice: measure it in ChatGPT and Perplexity.
- Track AI brand mentions over time.
- Measure AI share of voice in Python.
Do it without code
The free LLM visibility query builder turns your brand, category and competitors into a ready-to-run audit input, with a fixed example of the output. It doesn't call any AI model from your browser.
At scale
LLM Visibility Monitor on Apify records the sources behind each answer. It classifies every cited domain as owned, competitor, review site, Reddit or other user content, news, or other, ranks them, and lists the third-party sources cited in answers where a competitor appeared and you didn't. It also quotes the prices, plans and facts the answers state about your brand so you can check them. It asks the API models behind ChatGPT, Claude, Gemini, Perplexity and Grok and labels whether each answer used live web search; API answers can differ from the consumer apps. It's free to start, then pay-as-you-go.
Disclosure: I build the datatooly query builder and the Apify actor. Everything above is from the vendors' public documentation.
Sources: OpenAI: overview of OpenAI crawlers, OpenAI: web search tool guide, OpenAI Help Center: Searching the web with ChatGPT, Perplexity: How does Perplexity work?, Perplexity crawlers, Anthropic: crawlers and how to block them, Google's common crawlers (Google-Extended).
Top comments (0)