Your robots.txt might be hiding you from AI answers - a 5-minute check

#aiseomarketiwebdevng

Every audit I run starts the same way, and it is not with content. It is with robots.txt.

The pattern: someone has spent six months publishing comparison pages and wondering why ChatGPT never mentions them, and somewhere in the site config a Disallow: / rule was copied from a staging template, or a security plugin added User-agent: * with a blocklist, or a developer blocked "AI scrapers" in 2023 and never revisited it. Two years of content, invisible to the engines.

Here is the check, and the nuance people get wrong.

1. Which bots matter, and what each one actually does

These are not interchangeable. Blocking one does not block the others:

Bot	Used for	Blocking it costs you
`GPTBot`	OpenAI's crawling for model training	Training signal, but not necessarily ChatGPT search visibility
`OAI-SearchBot`	Building the ChatGPT search index	Appearing in ChatGPT search answers
`ChatGPT-User`	User-initiated fetch ("open this URL")	ChatGPT being able to read your page on demand
`ClaudeBot`	Anthropic's crawler	Claude understanding/citing your pages
`PerplexityBot`	Perplexity's index	Perplexity citations
`Google-Extended`	Gemini / Vertex training controls	Gemini training signal; does not affect Search ranking
`Bingbot`	Bing index — which Copilot runs on	Copilot visibility entirely

The one people get wrong most often: they block GPTBot to "opt out of training", feel principled, and don't realise OAI-SearchBot is a separate switch that governs whether they show up in ChatGPT search. Opting out of training is a legitimate choice. Opting out of being cited is usually an accident.

2. Read your own file

curl -s https://yourdomain.com/robots.txt

Then look for four things:

A blanket Disallow: / under User-agent: * — kills everything below it unless a specific agent overrides it.
Named agent groups with Disallow: / (User-agent: GPTBot followed by Disallow: / is the classic "we blocked AI in 2023" leftover).
Whole-site blocks hidden in a plugin's "security" tab, which you won't see in the file's first ten lines.
A Sitemap: line that points to a sitemap returning 404 — common after a domain or framework migration.

3. The minimum viable allowlist

If you want to be readable by assistants and search engines, you need explicit allows (specific agent groups are matched before *, so being explicit is safer than relying on the wildcard):

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Keep the training-crawler decision (GPTBot, CCBot) as a separate, deliberate choice. Just make it deliberately, not by inheritance.

4. Check the engine that powers the answer

Copilot runs on the Bing index, and a surprising number of sites are simply not in it. Verify with:

https://www.bing.com/search?q=site:yourdomain.com

Zero results means no amount of content work will help that slice of users. Submit the sitemap in Bing Webmaster Tools and wait; it is usually days, not weeks.

5. Then, and only then, work on content

Order matters. Crawler access → indexing → entity clarity → content → earned mentions. If you skip to step five, you are optimising pages nobody is allowed to read.

This is the boring part of AI visibility, and it is also the part that produces the biggest jumps, because it is binary: either the crawler gets a 200 and reads the page, or nothing else you do matters.

If you want to measure the result afterwards, I wrote about auditing whether AI mentions you at all and about the six kinds of pages models actually cite. The check above is what I run before either of them, and it takes five minutes.

(Disclosure: I build bidvix, which measures mention rates across five models — but crawler access is free to fix and I'd do that first either way.)

DEV Community