TL;DR: llms.txt is currently placebo. Ahrefs looked at 137,000 domains and 97% of the files received zero requests. Retrieval bots were about one percent of the small remainder. Google has publicly said it will not use it. Add one in ten minutes if you want the box ticked. Do not pay a retainer for it.
llms.txt is a proposed convention: a markdown file at the root of your site that tells language models which pages matter and summarises what your site is about. The idea is reasonable and the analogy to robots.txt is obvious, which is exactly why it spread so fast through agency pitch decks.
What the measurement showed
Ahrefs studied roughly 137,000 domains that had implemented the file and looked at whether anything actually requested it. The overwhelming majority - about 97 percent - received zero requests. Among the small remainder that saw any traffic at all, the retrieval bots that would need to read the file to make it useful accounted for about one percent.
Separately, Google has said publicly that it does not use llms.txt. That matters because a convention only works if the consumers adopt it. robots.txt works because every crawler honours it. A file nobody reads is not a standard, it is a suggestion.
Why it still shows up on invoices
It is cheap to produce, it looks technical, and it is impossible for a client to disprove in the short term. A deliverable that takes ten minutes, sounds sophisticated and cannot be falsified within the engagement window is close to an ideal line item for an agency that is not being careful. We do not think most people selling it are lying - the convention is genuinely plausible and the measurement is recent.
The one honest reason to add it
Cost. It takes ten minutes and it does no harm, and there is a non-zero chance adoption improves later. Treat it the way you treat a well-formed humans.txt: hygiene, not strategy. What is not acceptable is a slide, a line item, or a monthly fee attached to it.
A related trap: make sure yours does not 404 or 500
We have found live sites where the llms.txt route threw a server error because of a framework routing default. That is worse than not having the file: an error page at a well-known path is a small negative signal about site health. If you add one, check that it actually returns the file, then forget about it.
How to check the thing that actually matters
Before any content decision, find out whether retrieval crawlers can reach you at all. This is five minutes and it outranks everything else on the list.
Request your own pages with the user agents used by AI retrieval, and look at the status code. A 403 here makes every other optimisation irrelevant.
Check your CDN or firewall rules directly rather than trusting the site to load in your browser: you are not being filtered, bots are.
Fetch a page with JavaScript disabled and read what comes back. A client-rendered app frequently serves an empty shell, and that shell is what gets indexed.
Check that the pages you care about are actually in your sitemap and return 200, not a redirect chain.
Cloudflare has blocked AI crawlers by default on new zones since July 2025. A large share of sites we look at are invisible for that single reason and have no idea, because the site works perfectly for humans.
About that correlation, honestly
The mentions figure is the strongest signal in the published work, and it should be read carefully. A correlation of 0.664 between third-party brand mentions and AI visibility is not proof that buying mentions produces visibility. Mentions are also a proxy for being a bigger, better-known brand, and bigger brands are more visible for many reasons at once.
What it does justify: spending on being genuinely present in the sources an engine already cites for your category, rather than on files nobody requests. That is a defensible reading of the data. Treating 0.664 as a lever you pull is not.
What to do with the budget instead
The published research points at third-party mentions as the strongest correlate of AI visibility - Ahrefs found r = 0.664 for brand mentions against 0.218 for backlinks across roughly 75,000 brands. Before that, check the boring thing: whether AI crawlers can reach your site at all. Cloudflare has blocked them by default on new zones since July 2025, and that single setting outweighs every content decision underneath it.
Check whether OAI-SearchBot and the other retrieval agents are blocked at your firewall. This is the highest return per hour in the discipline.
Check what a crawler actually receives from your site. Client-rendered apps often serve an empty shell.
Fix entity contradictions - two brands under one legal entity, prices that disagree between pages.
Then spend on third-party placements, mapped from the sources an engine already cites in your category.
If an agency has put llms.txt on a proposal in front of you, ask them for the request logs. It is a fair question and the answer tells you a lot about how the rest of the engagement will go. Our own take on what does work is at /geo. Ivan / 2pizza.team
Originally published at 2pizza.team. We build AI and automation systems for small teams - fixed price, two to six weeks. See the work.
Top comments (0)