Which AI crawlers actually read your llms.txt? (GPTBot, ClaudeBot & friends)

#ai #llm #seo #webscraping

You added an llms.txt. Reasonable next question: who's actually reading it? The honest answer surprises people — so let's clear it up, and more importantly sort out which bots you should care about and how you actually control them (spoiler: not with llms.txt).

First, split the bots into two piles

Lumping all "AI crawlers" together is where people get confused. There are two kinds, and you treat them very differently:

Answer / search crawlers — they fetch a page to answer a user's question right now, and they're the ones that get you cited in AI answers:

OpenAI: OAI-SearchBot, ChatGPT-User
Anthropic: Claude-SearchBot, Claude-User
Perplexity: PerplexityBot, Perplexity-User

You generally want these.

Training crawlers — they collect data to train future models:

GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent

Whether to allow these is a separate, opt-out decision — and blocking them does not hurt your AI-search visibility.

So which of them read llms.txt today?

Straight answer: adoption is still early. llms.txt is a young, voluntary convention. It's picking up support in AI developer tools and agent frameworks, but most of the big crawlers above do not yet fetch it automatically. No bot is required to read it.

That sounds like a reason to skip it. It isn't — for two reasons:

The clean pages it points to help models right now. Even a crawler that ignores the file itself benefits from the well-structured, Markdown-friendly pages you curated for it.
It's cheap readiness. A correct llms.txt costs ten minutes and positions you for the moment adoption tips — which, given how fast this space moves, won't be long.

The part people get wrong: llms.txt doesn't control access

This is the big misconception. llms.txt never blocks or grants anything. It only curates — a suggested reading list. If you want to control which AI crawlers may fetch your pages, that's robots.txt's job.

A visibility-friendly setup allows the answer crawlers and makes a deliberate call on the training ones:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

That example says "let AI search engines cite me, but keep my content out of model training." Your call on that last line — just know it's a training opt-out, not a visibility switch.

The short checklist

Allow the answer/search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, …) so AI tools can cite you.
Decide training opt-outs (GPTBot, Google-Extended) deliberately — blocking them doesn't touch AI-search visibility.
Publish a curated llms.txt and then actually check it — the most common failure is a valid-looking file pointing at dead links. Run it through an llms.txt validator that fetches every URL.