Your robots.txt decides whether ChatGPT can cite you. Here's the 5-minute check.

siccscha — Fri, 31 Jul 2026 22:23:09 +0000

More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue link. Whether those answers can ever cite you is decided by one file: robots.txt. And most robots.txt files decide it by accident.

Two kinds of AI crawlers — and only one of them matters for citations

Kind	User agents	Blocking it means
Training	`GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `CCBot`	your content isn't used to train models
Retrieval	`OAI-SearchBot`, `ChatGPT-User`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot`	you can never be cited in an AI answer — no link, no attribution, ever

Blocking training crawlers is a legitimate choice. Blocking retrieval agents is almost never what a site intended — it just silently removes you from AI answers.

The blanket-block mistake

The "block the AI bots" wave of 2024/25 produced a lot of this:

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
Disallow: /

That single group blocks training and retrieval — worst of both worlds if you still hold the training position but wanted visibility. I checked major news sites this week: nytimes.com allows 1 of 7 retrieval agents, reuters.com, theverge.com and wired.com 2 of 7 each. None of them is fully citable — and these are sites whose entire business is being the cited source.

A deliberate policy — citable, but not training material — looks like this:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

The CDN plot twist: rules you never wrote

Here's how I fell into this myself. My own site's served robots.txt contained a "managed" block list my CDN (Cloudflare) had injected above my hand-written rules — blocking the same crawlers my own section explicitly allowed, plus a Content-Signal: line that is an express reservation of rights under Article 4 of the EU DSM Directive. A legal statement, published in my name, that I had never written.

Two lessons:

Read the served file, not your source file. curl https://yoursite/robots.txt and look for sections you didn't write.
Contradictory groups for the same crawler are effectively undefined behavior. The spec's tie-breaking (longest match wins, Allow beats Disallow on equal length, a group naming the agent beats *) resolves conflicts within a group — but when two groups target the same bot with opposite rules, what happens depends on each vendor's implementation. You genuinely don't know what you're telling that crawler.

Beyond robots.txt: two more silent killers

Pages that are empty without JavaScript. Retrieval agents don't render. A page Google ranks fine (Google renders) can be blank to every AI system. Test: curl your page and check whether the text is in the HTML.
Missing attribution signals. AI answers lift and attribute passages more readily from pages with an author, a date, question-shaped headings and FAQPage markup.

The 5-minute check

curl https://yoursite/robots.txt
Search it for the five retrieval agents above. Any Disallow that hits them = you're out of AI answers.
Look for sections and Content-Signal: lines you didn't write.
curl a key page and check the text is present without JavaScript.