If you're adding llms.txt hoping to get cited more often by ChatGPT or Google's AI answers, the evidence says it won't do that. But the file isn't useless either — it just does a different job than most people think, and a few major AI companies genuinely use it for that job.
Here's what I found when I went looking for actual measurements rather than opinions.
What the file is
A markdown file at your domain root containing a curated map of your site — the pages that matter, with short descriptions, organised so a machine can see the structure without crawling everything.
The proposal dates from September 2024. The reasoning is sound: a large documentation site has thousands of pages, and a system reading it without guidance may index outdated material and misread the structure. A curated index gives it a better starting point.
Note what that describes. It's a navigation aid — not access control, not a ranking signal. It can't block any crawler.
The citation claim doesn't hold up
Google's Search Central documentation states it directly: Google Search does not use llms.txt. Not for rankings, not for AI Overviews, not at all. That position has been consistent since Gary Illyes confirmed it in July 2025.
Independent measurement points the same way:
- Across hundreds of thousands of domains, 97% of llms.txt files were never fetched by anything
- One monitoring exercise covering 500M+ AI bot visits over 90 days recorded 408 requests targeting llms.txt directly
- Statistical models looking for a citation effect found none
- A study of 300,000 domains put implementation at just over 10%, eighteen months into the conversation
John Mueller compared llms.txt to the keywords meta tag, and the comparison is worth taking seriously rather than as a dismissal.
The keywords tag failed for a specific reason: it was a self-declaration nobody could verify. A site said what it was about, and nothing stopped it from saying whatever was convenient.
llms.txt has the same shape. It's a file in which a site describes its own importance, with no verification anywhere in the loop.
But several AI companies do use it
This is where "useless" overshoots.
- Anthropic recommends it in its guidance on writing for agents
- OpenAI maintains llms.txt files for its Agents SDK
- Perplexity has said it retrieves the file to help prioritise which pages to read
- Chrome's Lighthouse moved its llms.txt audit out of experimental into a default agentic browsing category in May 2026
So the Chrome team and the Search team are on visibly opposite sides of this.
The reconciliation: the file serves agentic retrieval, not conversational answering.
When a developer asks a coding assistant to work against someone's documentation, the assistant can fetch that site's llms.txt as a table of contents and traverse only the pages it needs, rather than scraping the whole site. That workflow is real and constant — and it's invisible to studies measuring crawler behaviour or citation rates, because it's neither crawling nor citation.
Which explains the adoption pattern precisely. The companies shipping llms.txt are documentation-heavy: developer platforms, infrastructure providers, API companies. Marketing sites and blogs mostly haven't, and that isn't neglect — their audience isn't coding agents.
So should you ship one?
Three questions settle it:
Do agents work against your content? If you publish documentation, an API reference or technical material developers point tools at — yes. The benefit is real for those users whatever the citation studies say.
Are you adding it to improve AI visibility? Then no, and the effort belongs somewhere with evidence behind it: facts stated so a machine can repeat them, structured data matching your copy, corroboration outside your own site.
Do you have spare capacity? Then it's defensible. It takes an hour and breaks nothing. Just don't count it as visibility work.
The implementation mistake worth avoiding
A popular approach generates a markdown copy of every page alongside the index. If those copies are indexable, you've created duplicate content at scale — every page now exists twice, competing with itself.
The file is supposed to be a curated index: your important pages, described briefly, linked. Not a second copy of the site. A small site can list twenty entries; a large one should list the twenty that matter rather than all nine hundred.
The file that actually controls AI access
Worth separating, because these two get conflated constantly.
robots.txt has genuine, deliberate backing from every major AI crawler operator. OpenAI, Anthropic, Google, Perplexity and the rest publish user-agent strings and honour the directives.
llms.txt cannot block anything. It's a suggestion about what to read once reading is already permitted.
Practical consequence: if AI visibility matters to you, the thing worth checking isn't whether you have an llms.txt file. It's whether your robots.txt is accidentally blocking the crawlers you want. That takes two minutes and it's the one of the two that can actually cost you.
Has anyone here measured a difference after shipping llms.txt? I've only found studies showing no effect, but those measure crawling and citation — not the agentic retrieval case, which is much harder to observe from outside. Curious whether anyone has data on that side.
I write about search, AI visibility and multilingual sites at bandziuk.com — this post first appeared there.
Top comments (1)
Your breakdown of the llms.txt file's actual utility versus common misconceptions is insightful, especially the comparison to the keywords meta tag. It’s crucial for developers to understand that while llms.txt can aid agentic retrieval for documentation-heavy sites, its impact on visibility in search engines is minimal. I appreciate how you've highlighted the specific use cases where it shines, which can help teams make more informed decisions about its implementation. If you're considering enhancements to the llms.txt integration for AI tools, I’d be interested in discussing how I might contribute to that effort.