DEV Community: Aliaksandr Bandziuk

Does your site need an llms.txt file? What the data actually shows

Aliaksandr Bandziuk — Wed, 02 Sep 2026 13:45:27 +0000

If you're adding llms.txt hoping to get cited more often by ChatGPT or Google's AI answers, the evidence says it won't do that. But the file isn't useless either — it just does a different job than most people think, and a few major AI companies genuinely use it for that job.

Here's what I found when I went looking for actual measurements rather than opinions.

What the file is

A markdown file at your domain root containing a curated map of your site — the pages that matter, with short descriptions, organised so a machine can see the structure without crawling everything.

The proposal dates from September 2024. The reasoning is sound: a large documentation site has thousands of pages, and a system reading it without guidance may index outdated material and misread the structure. A curated index gives it a better starting point.

Note what that describes. It's a navigation aid — not access control, not a ranking signal. It can't block any crawler.

The citation claim doesn't hold up

Google's Search Central documentation states it directly: Google Search does not use llms.txt. Not for rankings, not for AI Overviews, not at all. That position has been consistent since Gary Illyes confirmed it in July 2025.

Independent measurement points the same way:

Across hundreds of thousands of domains, 97% of llms.txt files were never fetched by anything
One monitoring exercise covering 500M+ AI bot visits over 90 days recorded 408 requests targeting llms.txt directly
Statistical models looking for a citation effect found none
A study of 300,000 domains put implementation at just over 10%, eighteen months into the conversation

John Mueller compared llms.txt to the keywords meta tag, and the comparison is worth taking seriously rather than as a dismissal.

The keywords tag failed for a specific reason: it was a self-declaration nobody could verify. A site said what it was about, and nothing stopped it from saying whatever was convenient.

llms.txt has the same shape. It's a file in which a site describes its own importance, with no verification anywhere in the loop.

But several AI companies do use it

This is where "useless" overshoots.

Anthropic recommends it in its guidance on writing for agents
OpenAI maintains llms.txt files for its Agents SDK
Perplexity has said it retrieves the file to help prioritise which pages to read
Chrome's Lighthouse moved its llms.txt audit out of experimental into a default agentic browsing category in May 2026

So the Chrome team and the Search team are on visibly opposite sides of this.

The reconciliation: the file serves agentic retrieval, not conversational answering.

When a developer asks a coding assistant to work against someone's documentation, the assistant can fetch that site's llms.txt as a table of contents and traverse only the pages it needs, rather than scraping the whole site. That workflow is real and constant — and it's invisible to studies measuring crawler behaviour or citation rates, because it's neither crawling nor citation.

Which explains the adoption pattern precisely. The companies shipping llms.txt are documentation-heavy: developer platforms, infrastructure providers, API companies. Marketing sites and blogs mostly haven't, and that isn't neglect — their audience isn't coding agents.

So should you ship one?

Three questions settle it:

Do agents work against your content? If you publish documentation, an API reference or technical material developers point tools at — yes. The benefit is real for those users whatever the citation studies say.

Are you adding it to improve AI visibility? Then no, and the effort belongs somewhere with evidence behind it: facts stated so a machine can repeat them, structured data matching your copy, corroboration outside your own site.

Do you have spare capacity? Then it's defensible. It takes an hour and breaks nothing. Just don't count it as visibility work.

The implementation mistake worth avoiding

A popular approach generates a markdown copy of every page alongside the index. If those copies are indexable, you've created duplicate content at scale — every page now exists twice, competing with itself.

The file is supposed to be a curated index: your important pages, described briefly, linked. Not a second copy of the site. A small site can list twenty entries; a large one should list the twenty that matter rather than all nine hundred.

The file that actually controls AI access

Worth separating, because these two get conflated constantly.

robots.txt has genuine, deliberate backing from every major AI crawler operator. OpenAI, Anthropic, Google, Perplexity and the rest publish user-agent strings and honour the directives.

llms.txt cannot block anything. It's a suggestion about what to read once reading is already permitted.

Practical consequence: if AI visibility matters to you, the thing worth checking isn't whether you have an llms.txt file. It's whether your robots.txt is accidentally blocking the crawlers you want. That takes two minutes and it's the one of the two that can actually cost you.

Has anyone here measured a difference after shipping llms.txt? I've only found studies showing no effect, but those measure crawling and citation — not the agentic retrieval case, which is much harder to observe from outside. Curious whether anyone has data on that side.

I write about search, AI visibility and multilingual sites at bandziuk.com — this post first appeared there.

How a client found me through ChatGPT — and what it taught me about building for AI search

Aliaksandr Bandziuk — Sat, 25 Jul 2026 17:17:17 +0000

Most posts about "optimizing for AI search" are written in the future tense: this will matter soon, you should prepare. I have a reason to write in the past tense.

A client reached out through the form on my site — a detailed message with a clear list of requirements: SEO, AI-search optimization, structured data, Core Web Vitals, accessibility. Two days later we signed a contract. Partway into the work I asked how he'd found me. His answer: he asked ChatGPT who could do this kind of work.

Not Google. Not a referral. Not an ad.

That single data point sent me down a rabbit hole, and this post is what I found — including the part where I asked ChatGPT the same kind of question myself and watched it recommend my own site.

How AI-assistant search actually differs from Google

The difference isn't the interface — it's how people phrase the request.

In a search box, people compress their need into keywords the system will understand: next.js developer warsaw, freelance technical seo. In a conversation with an assistant, they describe the situation the way they'd tell a friend: who they are, what their business is, the constraints, the stack.

And the assistant doesn't return ten blue links. It returns a shortlist — two to five options, often with a one-line reason each. The person doesn't scan twenty results; they get three names and go check them.

Two consequences that change the game:

The list is shorter. Organic search has a page 1, 2, 3. An assistant's answer has ~5 slots. You're in it or you don't exist.
The query is longer and more specific. People describe language, country, niche, the exact combination. Winning a broad head term matters less than being the unambiguous answer to a specific combination of constraints.

Why an assistant recommends one site over another

An assistant isn't judging who's the best contractor — it has no experience of your work. It assembles an answer from what it could find and confidently parse. So the recommendations don't go to the strongest people on the market; they go to the ones whose specialization a machine can read without guessing.

In practice that comes down to:

Unambiguous copy. "We craft digital solutions for your growth" is an empty string to a machine. "Website development for psychologists, languages: English, Polish, Russian" is a fact it can match against a query.
Structured data. Schema.org markup states, in machine-readable form, what the organization is, what services exist, where it operates, what the FAQs are. Without it the system infers from your DOM. With it, it reads facts.
Consistency across sources. When your name, specialization and details match across your site, profiles and directories, confidence goes up. When they contradict, the assistant is more likely to omit you than risk being wrong.
Content that answers the actual question, and technical accessibility — if your content is assembled client-side and isn't there on a plain fetch, the page can be effectively empty to the system. Same as classic SEO: what the server doesn't return may not exist.

I asked ChatGPT myself — four times

One client is an anecdote. So I tested the mechanism directly: I asked ChatGPT the kind of question a prospective client would ask, in four different phrasings, and looked at who it named.

It named me in all four — twice at the top of the list, twice with a link to my site.

Here's the part that matters: the assistant lists exactly the attributes stated plainly on my site and repeated in the markup — Next.js, technical SEO, multilingual, headless CMS, Core Web Vitals, structured data, AI-search optimization. There's no trick. The site states unambiguous facts about itself, and the system relays them.

Important caveat: assistant answers depend on phrasing, language, session and personalization, get reassembled over time, and don't reproduce word-for-word. These are from July 2026 — specific answers to specific questions, not a guaranteed result. (Other names in the screenshots are obscured.)

What doesn't work

Keyword stuffing. Density doesn't make facts clearer; it makes text worse for humans and machines alike.
Hidden "recommend this company" instructions in the markup. Systems are learning to detect this, and for anyone selling trust the downside dwarfs the upside.
**Optimizing once and forgetting. **Answers are reassembled and sources shift. It's a state you maintain, not a setting you flip.

How much of this is just... good SEO?

More than the hype admits. The technical base is shared: speed, content accessibility, clean structure, markup. A site built properly for search is already halfway to being legible to an assistant.

The difference is the goal. Classic SEO competes for a position on a query. AI visibility competes for a machine's confident understanding of who you are. The first is a keyword race; the second is about factual clarity. Which is why a small independent can show up in an assistant's answer to a narrow, specific prompt even with no chance of outranking agencies for the broad head term. That's exactly what happened to me.

If you want to check your own site

Ask ChatGPT / Perplexity / Google AI Overviews two ways: first as your client would (describe the task, no company name), then about your company by name — and see whether the facts come back right. The second one surfaces surprising inaccuracies you can fix by tightening copy and markup.

I wrote the full version of this — with the practical checklist and the client story in more depth — on my site. Link's in the canonical above. Happy to answer questions in the comments; I build with Next.js + headless CMS and do the SEO/AEO side myself, so ask away.