Ask a room of SEOs whether you need an llms.txt file and you will get an argument. Ask what actually fetches the file and the room goes quiet.
There is data now, and it is not kind. Ahrefs pulled server logs from 137,210 domains. 28% of them publish an llms.txt. In May 2026, 97% of those files received zero requests. Around 1,100 domains saw any traffic to the file at all.
That sounds like the end of the story. It is not, because the small slice of requests that did land tells you exactly who this file is for, and it is not who anybody was optimizing for.
What the file actually is
llms.txt is a Markdown file at the root of your domain, at /llms.txt, listing your most important pages with a one-line description of each. Jeremy Howard published the proposal on September 3, 2024.
The reasoning is good. A docs page rendered as HTML is mostly not documentation. It is nav bars, sidebars, cookie banners, and framework wrapper divs. Make a model chew through all of that and it spends most of a fetch on nothing. A Markdown index costs a fraction of the tokens and points straight at the pages that matter.
The format is simple and only the H1 is required. Cloudflare publishes a clean example, trimmed here:
# Cloudflare Developer Documentation
> Each product below links to its own llms.txt, which contains a full index
> of that product's documentation pages.
## Application performance
- [Argo Smart Routing](https://developers.cloudflare.com/argo-smart-routing/llms.txt): Route web traffic across the fastest, most reliable network paths
- [Cache / CDN](https://developers.cloudflare.com/cache/llms.txt): Make websites faster by caching content across our global network
Two things worth stealing from that. Cloudflare nests the format, so every product gets its own llms.txt and the root file becomes a directory of directories. And every link carries a description, which is the part most people skip and the part that does the actual work. A bare list of URLs tells a model nothing about which one to open.
There is a sibling file, llms-full.txt, which is the whole docs set flattened into one large Markdown file so an agent can load everything in a single request. It is not in the original proposal. It grew out of documentation platforms, Mintlify in particular, and it spread because it is what an agent actually wants when you point it at a docs site.
The SEO question is settled, and the answer is no
Google's AI features documentation says you do not need to create new machine readable files or AI text files to appear in AI Overviews or AI Mode. John Mueller has separately called llms.txt speculative, compared it to the keywords meta tag, and pointed out the awkward fact that no AI system uses it.
robots.txt works because crawlers agreed to obey it. sitemap.xml works because search engines agreed to consume it. llms.txt is a good idea that shipped without a counterparty. If you added one hoping to appear in more AI answers, you did not get what you paid for.
Now the interesting half
Ahrefs did not just count the zeros. They named the requests that landed.
| Fetcher | Share of requests | What it is doing |
|---|---|---|
| SEO audit tools | 21.7% | Checking whether the file exists, for a report |
| General web crawlers | 13.1% | Indiscriminate crawling |
| GPTBot | 4.51% | OpenAI's crawler |
| Claude-Code indexer | 3.52% | A developer's coding agent reading docs |
| ClaudeBot | 0.8% | Anthropic's web crawler |
| OAI-SearchBot | 0.74% | OpenAI's search-facing bot |
Read that table twice.
The single largest consumer of llms.txt is SEO tooling checking whether you have an llms.txt. A snake eating its own tail.
And the second largest named AI fetcher is Claude Code. That is not a search engine discovering your brand. That is a developer sitting in a terminal who already knows your product exists, pointing an agent at your documentation and telling it to go read the auth flow.
Every IDE agent has converged on the same behavior. Point Cursor, Windsurf, Claude Code, or Copilot at a docs site and the first thing it reaches for is /llms.txt and /llms-full.txt, because that is the cheapest possible way to load a doc set.
llms.txt is a documentation delivery format that got marketed as an SEO tactic. That mismatch is why every study of it reads like a failure.
Writing one, in about ten minutes
1. List the pages you would send a new hire: Not every page. The ones that say what the thing is, how to install it, and how to do the three most common jobs. Fifteen to fifty links is a normal range. A file with 400 links is a sitemap with extra steps.
2. Write the H1 and the blockquote: The H1 is your project name. The blockquote is one or two sentences that let a model understand what you are before it reads anything else. Be concrete. "MCP-native tools that connect Claude and Cursor to the data they are missing" beats "a platform for modern teams".
3. Group the links under H2 headings: By product, by job, or by section of your docs. The heading is context, so name it something a model can reason about. "Getting started" and "API reference" are useful. "Resources" is not.
4. Give every link a description: The format is [Page title](https://example.com/page/): what this page covers. If your pages already have meta descriptions, reuse them. This is the highest-value part of the file and the part most published examples get lazy about.
5. Generate it, do not hand-maintain it: A hand-written llms.txt goes stale in a month. Generate it from your docs navigation and each page's frontmatter at build time and it cannot drift from the real doc set. On a static site that is twenty lines of build code.
Serve it as text/plain at https://yourdomain.com/llms.txt and use absolute URLs. Relative links are the most common mistake and they break the file for any agent that fetched it from somewhere else.
So should you ship one
Ship it if you have documentation: If developers point AI tools at your docs, this is the difference between an agent reading a clean index and an agent scraping four rendered HTML pages to find your auth flow. That is a real improvement for the exact person most likely to become a customer, and it costs a build script.
Skip it if you run a marketing site and want AI visibility: A ten-page site about your agency does not need a machine-readable index of ten pages. Nothing will fetch it, and the hour costs you an hour of the work that does move citations.
Never treat it as a substitute for the fundamentals: It cannot make thin content citable, it cannot fix a page that renders as an empty shell without JavaScript, and it grants no crawl permission you did not already give in robots.txt.
The uncomfortable version, since somebody should say it. A lot of llms.txt advice exists because "add this file" is an easy thing to sell and an easy thing to check off. It has the shape of a deliverable, it takes minutes, and nobody can prove it did nothing. That combination is exactly how the keywords meta tag survived for a decade after it stopped mattering. Ship the file for the coding agents. Do not put it in a strategy deck.
How to tell whether yours is doing anything
Grep your access logs for llms.txt and look at the user agents. That is the only measurement that answers the real question. Coding-agent traffic means the file is doing its job. Three months of nothing but audit tools tells you exactly what it is worth to you.
If your goal was AI visibility, measure the outcome instead of the tactic. Ask your target questions in ChatGPT, Claude, and Perplexity and record whether you get named at all.
Full disclosure, I build tooling in this space. ContextBolt SEO is a hosted SEO MCP server. You paste one URL into Claude Code, Cursor, or Codex, then ask in plain language whether AI tools are mentioning you and how your share of voice is moving against named competitors, plus the keyword, rank, and backlink work you would otherwise open a $129 tool for. It is $35 a month, month to month.
The longer version of this, with the full spec table and a copyable example, is my guide on how to write an llms.txt file.
llms.txt is a good file with a bad reputation, earned by the people who oversold it. It is not a ranking signal and it never was. It is how you hand a coding agent your documentation without making it read your nav bar. Judged as that, it works.
Top comments (1)
wild that only one tool is actually using these right now. guess it's just a bet on claudes ecosystem for now lol