DEV Community

siccscha
siccscha

Posted on

Your robots.txt decides whether ChatGPT can cite you. Here's the 5-minute check.

#ai #seo #webdev #devops

More and more answers reach your readers through ChatGPT, Claude and Perplexity instead of a blue link. Whether those answers can ever cite you is decided by one file: robots.txt. And most robots.txt files decide it by accident.

Two kinds of AI crawlers — and only one of them matters for citations

Kind User agents Blocking it means
Training GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot your content isn't used to train models
Retrieval OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot you can never be cited in an AI answer — no link, no attribution, ever

Blocking training crawlers is a legitimate choice. Blocking retrieval agents is almost never what a site intended — it just silently removes you from AI answers.

The blanket-block mistake

The "block the AI bots" wave of 2024/25 produced a lot of this:

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
Disallow: /
Enter fullscreen mode Exit fullscreen mode

That single group blocks training and retrieval — worst of both worlds if you still hold the training position but wanted visibility. I checked major news sites this week: nytimes.com allows 1 of 7 retrieval agents, reuters.com, theverge.com and wired.com 2 of 7 each. None of them is fully citable — and these are sites whose entire business is being the cited source.

A deliberate policy — citable, but not training material — looks like this:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
Enter fullscreen mode Exit fullscreen mode

The CDN plot twist: rules you never wrote

Here's how I fell into this myself. My own site's served robots.txt contained a "managed" block list my CDN (Cloudflare) had injected above my hand-written rules — blocking the same crawlers my own section explicitly allowed, plus a Content-Signal: line that is an express reservation of rights under Article 4 of the EU DSM Directive. A legal statement, published in my name, that I had never written.

Two lessons:

  1. Read the served file, not your source file. curl https://yoursite/robots.txt and look for sections you didn't write.
  2. Contradictory groups for the same crawler are effectively undefined behavior. The spec's tie-breaking (longest match wins, Allow beats Disallow on equal length, a group naming the agent beats *) resolves conflicts within a group — but when two groups target the same bot with opposite rules, what happens depends on each vendor's implementation. You genuinely don't know what you're telling that crawler.

Beyond robots.txt: two more silent killers

  • Pages that are empty without JavaScript. Retrieval agents don't render. A page Google ranks fine (Google renders) can be blank to every AI system. Test: curl your page and check whether the text is in the HTML.
  • Missing attribution signals. AI answers lift and attribute passages more readily from pages with an author, a date, question-shaped headings and FAQPage markup.

The 5-minute check

  1. curl https://yoursite/robots.txt
  2. Search it for the five retrieval agents above. Any Disallow that hits them = you're out of AI answers.
  3. Look for sections and Content-Signal: lines you didn't write.
  4. curl a key page and check the text is present without JavaScript.

For one site, that's honestly enough. I also built the bulk/scheduled version — a technical SEO audit that includes exactly this citability check per vendor, plus a diff against the previous run: SEO Audit + AI Visibility on Apify. Disclosure: that's my tool; the manual check above needs nothing but curl.

Top comments (0)