We publish an llms.txt on our main domain and on every language portal. It is exactly what the proposal asks for: a curated markdown index of the site, at the root, written for AI systems to read. I pulled four months of nginx logs to answer one question: who actually reads it?
From April 7 to August 5, 2026, our front-ends served 1,321 requests to /llms.txt and /llms-full.txt across eleven hostnames (plus a few stray subdomains). About eleven a day, 92 percent answered with a 200. That volume already tells you this is not a busy interface. But the volume is not the story. The story is who those requests came from, because I went through every one of them, checked the interesting ones against published IP ranges, and the answer says something uncomfortable about the file, and something genuinely funny about the ecosystem around it. Ahrefs and Evil Martians have already counted this traffic at scale, and I will point at both. What I wanted was per-request forensics: when a hit wears an AI badge, is it real, and who sent it?
One line of context so you know where I stand: I run engineering at GoodBarber, an app platform. We run a production MCP server, we publish Claude Skills, we want the machine-readable web to work. This is not a takedown by someone who resents agents. It is a log review by someone who ships for them.
What llms.txt promised
Quick recap, then the data. llms.txt was proposed by Jeremy Howard in September 2024: a markdown file at your site root "to provide information to help LLMs use a website at inference time". Same placement logic as robots.txt and sitemap.xml. Mintlify made it a default for every docs site it hosts in November 2024, Shopify quietly shipped it to storefronts in spring 2026, and Originality.ai's tracker counted an 8.8x adoption jump in a year. On the publish side, the format is a success.
The open question was always the other side: does anything read it? Not "does anything fetch any file on the internet", because scanners fetch everything. Does the intended audience, the assistants and their crawlers, actually consume it?
Who actually reads our llms.txt
Here is the reader board for our 1,321 requests, by user-agent:
| Reader | Requests | What it is |
|---|---|---|
| Generic browser user-agents, HeadlessChrome, WebPageTest agents, assorted scripts | ~600 | tools that did not bother introducing themselves |
satoric-indexer/0.1 |
116 | unknown indexer; even Ahrefs could not identify it |
| Googlebot | 80 | URL crawling, and Google says it ignores the file |
closing-web-clock/1.0 (+policy-surface observation) |
54 | anonymous project observing "policy surfaces"; zero public footprint |
| SemrushBot SiteAudit | 46 | SEO audit tool |
| BuiltWith | 44 | tech profiler |
LLMS-Txt-Scanner/1.0 (Research) |
31 | anonymous scanner on Azure IPs, self-labeled research |
AgentRadar-Research/1.0 (BCG Henderson Institute; agentic-web-research) |
29 | identifies itself as a BCG crawler studying the agentic web; Evil Martians logged the same operator |
statespace-indexer/0.1 |
27 | agentic-web indexer; its likely parent repo was archived on August 5, 2026 |
prompt-injection-survey/1.0 (research) |
21 | self-described security research scanning llms.txt as an attack surface |
SEO auditors, tech profilers, headless browsers, and a small zoo of research scanners. So far this matches what Ahrefs found across 137,210 domains in June (97 percent of llms.txt files got zero requests; of the traffic that exists, SEO audit tools are the biggest reader at 21.7 percent, AI assistants 2.5 percent), and what Evil Martians found in their own logs in July. Both are worth your time. Neither answered the question I actually had, which is why this post exists: when a request does carry an AI assistant's name, is it real, and did the assistant come on its own?
The 26 requests that wore an AI badge
Out of 1,321 requests in four months, exactly 26 carried the user-agent of an AI assistant or its crawlers. I checked each one against the vendors' published IP ranges (gptbot.json, chatgpt-user.json, searchbot.json, perplexitybot.json, and Anthropic's known blocks). They decompose like this:
Three were impostors. A single DigitalOcean IP requested our llms.txt three times in the same second, presenting as GPTBot/1.0, then PerplexityBot/1.0, then ClaudeBot/1.0. Bare version strings, none of the vendors' real UA formats, an IP in nobody's published ranges. Somebody's compliance checker cycling through bot costumes to see if we cloak (serve different content to bots than to humans). If you count assistants by user-agent alone, your llms.txt numbers include this.
Four were GPTBot, genuinely. All four IPs validate against OpenAI's published gptbot.json ranges. This is the interesting part, because all four requests carried a Referer header. Three referers were https://agentready.md, an AI-readiness checker that validates llms.txt files and runs a public leaderboard of them. The fourth was https://sitestatsdb.com, which I could not even inspect, because it now sits behind a bot-verification wall. GPTBot did not wake up wanting our llms.txt. It crawled third-party pages that link to our llms.txt, and followed the links like it follows any link. In four months and 215,484 GPTBot requests to our sites, the number of times it requested llms.txt without a third party pointing at it: zero.
I think this detail matters beyond our logs. Ahrefs found GPTBot to be the single biggest AI fetcher of llms.txt files (4.51 percent of requests) and concluded, correctly, that "Zero requests came from AI bots for llms.txt files that don't exist. They never go looking." Our referer data shows the mechanism behind both facts: an ecosystem of llms.txt directories and audit tools now links out to these files, and a crawler that follows links will follow those too. The training crawler "reads" llms.txt the way it reads any URL it encounters: as content, not as protocol. The llms.txt promotion ecosystem is generating the llms.txt consumption statistics.
Ten were ChatGPT-User. Azure IPs, half of them in the current published rotation (the list rotates, and these hits go back to April). ChatGPT-User is OpenAI's on-demand fetcher: it fires when a human asks ChatGPT something that requires opening a specific page. Ten requests in four months, each one meaning a human request was behind the fetch. Given what people use ChatGPT for these days, my honest guess is "check my site's llms.txt" prompts and GEO (generative engine optimization) audit tools, and I cannot rule out that some of those humans were us.
Two were ClaudeBot. Anthropic's crawler, from its usual 216.73.216.x block. These are the only two requests in the entire window where a dedicated AI crawler fetched the file with no referer and no human in the loop. Two, out of 262,242 ClaudeBot requests to our sites in the same period. Credit where due, and note what it is: the training crawler stocking a corpus, not an assistant consulting an index on a user's behalf. No user-facing assistant ever came on its own.
Four were Claude-User and three were Claude Code. Claude-User is Anthropic's on-demand fetcher, same story as ChatGPT-User: a human asked. The Claude Code hits are the most telling: Claude-User (claude-code/2.1.x) requests coming from a residential ISP line in France and a French cloud box. That is a developer at a keyboard pointing Claude Code at the file, quite possibly someone on my own team. It is also the one documented use case that actually works: a coding agent reading a docs index because a human told it to.
Meanwhile, in the same logs
Now the control group, because "assistants do not read llms.txt" could just mean "assistants do not read us". They read us plenty. Same window, same domains, AI bots only:
| User-agent | Requests, all paths | Requests to /llms*, unprompted |
|---|---|---|
| ChatGPT-User | 472,985 | 0 (10 user-triggered) |
| Meta-ExternalAgent | 369,504 | 0 |
| ClaudeBot | 262,242 | 2 |
| GPTBot | 215,484 | 0 (4 via directory links) |
| OAI-SearchBot | 171,121 | 0 |
| PerplexityBot | 63,475 | 0 |
| Perplexity-User | 22,667 | 0 |
| Claude-User | 19,324 | 0 (4 user-triggered) |
| Claude-SearchBot | 17,512 | 0 |
| DuckAssistBot | 6,511 | 0 |
| Claude-User (claude-code) | 4,513 | 0 (3 user-triggered) |
That is 1.6 million requests from AI systems in four months. ChatGPT-User alone made 89,668 blog-article fetches, about 740 a day, triggered by real people asking real questions; our most-fetched pages are deep technical pieces on app store compliance that classic SEO metrics would call worthless. The assistants are not ignoring our content. They are devouring it, hundreds of article fetches a day. And in the same four months, the file that exists specifically for them was fetched unprompted exactly twice, both times by the training crawler. Every other AI-labeled visit had a human behind it or a link in front of it.
And the ratio that settles it: Claude Code user-agents made 4,513 requests to our pages in four months. Documentation pages mostly, 2,264 hits on /help/ alone, plus 1,182 on the blog. Number of those requests that went to llms.txt: three. Even the archetypal llms.txt consumer, the coding agent that the strongest case for llms.txt is built on, overwhelmingly goes straight to the actual pages.
This is consistent with everything the vendors themselves publish, by the way. OpenAI's bot documentation explains robots.txt directives and never mentions reading llms.txt. Same for Anthropic's and Perplexity's. Google wrote it into its official AI documentation in June: "Google Search ignores them." Meanwhile all three AI vendors publish llms.txt files for their own docs sites, largely because their docs platforms generate one by default. Everyone is publishing the format. Nobody is committing to consume it.
John Mueller said it plainly back in April 2025: none of the AI services have said they use it, "and you can tell when you look at your server logs that they don't even check for it." Fifteen months later, on our logs, still true.
An object of research before an object of use
So who is left? Go back to the reader board. Once you remove the SEO auditors, the profilers, and the scripts, the most attentive readers of our llms.txt are research crawlers: an agent presenting itself as BCG Henderson Institute research on the agentic web, an anonymous LLMS-Txt-Scanner running from Azure, a closing-web-clock observing "policy surfaces", a statespace-indexer whose likely operator archived its repo the day before I wrote this, and my personal favorite, prompt-injection-survey/1.0 (research): a self-described survey of llms.txt as a prompt injection attack surface, because a plaintext file that sites voluntarily write for LLMs to ingest is, of course, a lovely place to plant instructions. All of this assuming their user-agents tell the truth, which, as the DigitalOcean impostor demonstrated, is optional. Ahrefs found the same pattern at scale: 12 percent of all llms.txt requests come from tools auditing, scoring, validating, or studying llms.txt itself.
The file has a thriving ecosystem of directories that list it, auditors that score it, scanners that catalog it, researchers that study it, and security people who probe it. Every layer of that stack is real and shipping. The one layer that does not exist is the consumer it was designed for.
llms.txt has become an object of research before ever becoming an object of use.
I want to be fair to the proposal here: nothing in the spec is wrong. It is a reasonable, well-designed answer to a real problem, context windows are finite and HTML is noisy. The assistants simply built around the problem instead: they fetch the real pages on demand, at scale, triggered by user questions. The protocol layer the file was supposed to occupy got filled by retrieval.
What I would actually do
My recommendation, which is what we do ourselves:
- Keep the file if it costs you an hour. Ours stays. It is one hour of maintenance, it is harmless, and if a platform someday announces real consumption, you are ready. Google's official position is literally "won't harm (nor help)".
- Keep it accurate, precisely because humans send agents to it. Every genuine assistant-family read in our logs, the two ClaudeBot fetches aside, had a human behind it or a link in front of it. Those are the readers you are writing for: a developer pointing Claude Code at your docs, a prospect asking ChatGPT to look at your file. Stale content there is worse than no file.
- If you have developer docs, that is the one real use case. Coding agents directed at an llms.txt index do consume it (tooling like mcpdoc is built on exactly this). If your product has no docs for developers, your llms.txt is decoration.
- Never report it as an AI-visibility deliverable. If someone bills you for "GEO optimization" and the deliverable is an llms.txt file, you now have four months of somebody else's server logs saying what that is worth. What actually got our content into assistants, per the same logs: deep, specific, technical articles that answer the questions people actually ask, plus a robots.txt that lets retrieval bots in.
- Treat the file as code, not copy. Version-control it, restrict who edits it, keep it to plain links and descriptions. The most motivated readers of the format right now include people studying it as an injection surface. Do not give them material.
Check your own logs
Methodology, so you can replicate: 121 days of nginx access logs (April 7 to August 5, 2026) across our production hostnames, every request matching GET /llms, user-agents tallied, and every AI-labeled hit checked against the vendors' published IP ranges where they exist (OpenAI, Perplexity), and, for Anthropic, which publishes no list, against the blocks that carry its 262,000 other requests in the same logs. One caveat cuts both ways: user-agent strings are claims, not proofs. Our ~600 generic-browser requests surely hide a few interesting readers, and any study that counts assistants by UA alone, including this one before the IP checks, will overcount them. Could an assistant fetch llms.txt in stealth, hiding in those 600? In principle. But a vendor that cheerfully declares itself on 1.6 million fetches to the same servers has no evident reason to go incognito for this one file.
The whole analysis is a couple of grep and awk commands. If you publish an llms.txt, run them. The base rate says you will find silence; if you find an AI assistant that came looking on its own, unprompted, no referer, IP in the vendor's published range, I genuinely want to see that log line. Comments are open.
Top comments (0)