Which AI crawlers actually read your llms.txt? (GPTBot, ClaudeBot & friends)

You added an llms.txt. Reasonable next question: who's actually reading it? The honest answer surprises people — so let's clear it up, and more importantly sort out which bots you should care about and how you actually control them (spoiler: not with llms.txt).

First, split the bots into two piles

Lumping all "AI crawlers" together is where people get confused. There are two kinds, and you treat them very differently:

Answer / search crawlers — they fetch a page to answer a user's question right now, and they're the ones that get you cited in AI answers:

• OpenAI: OAI-SearchBot, ChatGPT-User
• Anthropic: Claude-SearchBot, Claude-User
• Perplexity: PerplexityBot, Perplexity-User

You generally want these.

Training crawlers — they collect data to train future models:

• GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent

Whether to allow these is a separate, opt-out decision — and blocking them does not hurt your AI-search visibility.

So which of them read llms.txt today?

Straight answer: adoption is still early. llms.txt is a young, voluntary convention. It's picking up support in AI developer tools and agent frameworks, but most of the big crawlers above do not yet fetch it automatically. No bot is required to read it.

That sounds like a reason to skip it. It isn't — for two reasons:

1. The clean pages it points to help models right now. Even a crawler that ignores the file itself benefits from the well-structured, Markdown-friendly pages you curated for it.
2. It's cheap readiness. A correct llms.txt costs ten minutes and positions you for the moment adoption tips — which, given how fast this space moves, won't be long.

The part people get wrong: llms.txt doesn't control access

This is the big misconception. llms.txt never blocks or grants anything. It only curates — a suggested reading list. If you want to control which AI crawlers may fetch your pages, that's robots.txt's job.

A visibility-friendly setup allows the answer crawlers and makes a deliberate call on the training ones:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

That example says "let AI search engines cite me, but keep my content out of model training." Your call on that last line — just know it's a training opt-out, not a visibility switch.

The short checklist

• Allow the answer/search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, …) so AI tools can cite you.
• Decide training opt-outs (GPTBot, Google-Extended) deliberately — blocking them doesn't touch AI-search visibility.
• Publish a curated llms.txt and then actually check it — the most common failure is a valid-looking file pointing at dead links. Run it through an llms.txt validator that fetches every URL.

llms.txt is the map; robots.txt is the gate. Keep the map honest and the gate deliberate, and you're ahead of nearly everyone.

Comments

[Bulti]

Good breakdown of the two crawler categories. Worth flagging on adoption: Ahrefs looked at 137k sites with an llms.txt and found 97% got zero bot traffic to that file, and Semrush's server-log analysis found effectively zero AI-crawler visits to it either. Matches your point that llms.txt itself isn't doing much work yet - the clean, well-structured pages underneath it are what actually gets used when a crawler shows up. Curious if you've seen adoption numbers moving at all in the last month or two.

[Andriy Pyvovarchuk]

thanks — and that's the split i'd make too. publishing adoption and crawler adoption are two different curves, and only one of them is moving.

the most direct evidence i've seen is the spec author's own note on the v2 revision in august: thousands of sites now publish a file, documentation platforms generate one automatically, and coding agents use them reliably. "coding agents" is the operative bit. the consumption that exists today is agents and dev tooling fetching a file on purpose, not search crawlers discovering it. so a server-log study counting big-crawler hits to /llms.txt and finding roughly nothing is consistent with the file being used — just not by the bots those logs are counting.

what has actually changed since august is the spec, not the traffic. v2 added link relations so a page can point at the llms.txt that covers it (rel="describedby") and at its own markdown version (rel="alternate" type="text/markdown"), allowed both .md url forms, and defined subpath files — a file covers the pages under its path, most specific wins. it also dropped the context-expansion tooling, and with it the mechanical meaning of the Optional section. those are all discoverability changes, so if adoption does move, that's the mechanism i'd expect it to move through.

i pulled my own logs after reading this. they point the same way from a much smaller sample, but the shape turned out to be more interesting than "no traffic".

the ai crawlers are here constantly. 11,170 requests across the ~90 days my analytics retains (2026-07-22 to 2026-10-20), from thirteen distinct bots, nine of them seen in the past week — claudebot 2614, amazonbot 2063, chatgpt-user 1742, perplexitybot 1567, oai-searchbot 1470, gptbot 621, meta-externalagent 568.

of those 11,170, exactly 33 were for /llms.txt. 65 counting llms-full.txt and llms.json too. the same crawlers fetched /robots.txt 1,344 times and /sitemap.xml 687 — so here an ai crawler is about 40x more likely to ask for robots.txt than llms.txt, and 21x more likely to want the sitemap. they navigate by the old conventions and walk straight past the new one.

so it isn't discovery. they crawl the site fine and the guide articles are among their top paths — they just don't want the file. that matches your numbers and narrows what they mean: the gap isn't that crawlers can't find llms.txt, it's that they aren't asking for it.

one thing cuts the other way, and it surprised me: the markdown versions of pages got 159 fetches over the same window, against 33 for llms.txt. tiny numbers either way, but that's the v2 link-relations direction — point at a page's .md and they'll take it — rather than the curated file. the a2a agent card got 5.

caveat so none of this is worth more than it is: one site, a small one, and a 90-day window rather than forever.