DEV Community

Cover image for Is llms.txt worth it if almost nobody reads it?
Apogee Watcher
Apogee Watcher

Posted on Originally published at devlog.apogeewatcher.com

Is llms.txt worth it if almost nobody reads it?

In 2026 the industry finally measured something practitioners suspected: most llms.txt files never get fetched. Ahrefs’ study of domains running its Web Analytics reported that about 97% of published files received zero traffic in the sample window, and that AI search bots barely show up among the few requests that do arrive. Otterly’s GEO experiment and other crawls of large domain sets point the same way, and Google Search has said it does not treat llms.txt as a robots-style control for ranking. So the honest agency question is no longer “how do we write the perfect file?” It is whether the file is worth the upkeep when almost nobody reads it.

We already separated three artefacts in robots.txt, llms.txt, and cats.txt are not the same job: crawl policy, optional agent map, and the joke that proves a 200 on a .txt is not a protocol. What follows is the decision layer on top of that taxonomy. For crawl speed and fetchability habits that still move risk, keep Why AI crawlers need fast, crawlable pages open (Apogee Watcher). For measurement without buying another GEO dashboard, use Monitor AI search optimisation without a subscription.

What “almost nobody reads llms.txt” actually means

“Unread” in these studies means no HTTP fetches of /llms.txt in the observation window, not “AI models refuse to cite you.” That distinction matters in client meetings. A zero-fetch file failed as a distribution channel; it did not prove your product pages are invisible in ChatGPT answers, and it did not prove Lighthouse’s experimental Agentic Browsing check is wrong to mention the file.

When fetches do happen, researchers keep finding a skewed audience: coding agents, researchers, and tools that study llms.txt itself, not a reliable pipeline of AI Overview crawlers hunting for your brand summary. Ahrefs also reported that bots do not go looking for a missing file, so publishing one does not create demand. At best it serves readers who were already pointed at your docs.

The base rate for a random marketing site is harsh. Most files sit untouched, and treating that silence as “we’re GEO-ready” is the expensive mistake.

What recent llms.txt studies measured (and what they did not)

Point at the artefacts, not the vibes:

  • Ahrefs (137k sites with analytics): high share of domains that publish a file still see zero fetches; among the tiny share that are hit, coding agents outrank AI search bots; Lighthouse’s audit accounts for a tiny fraction of fetches.
  • Otterly and similar GEO tests: /llms.txt receives a tiny slice of AI bot traffic relative to ordinary content URLs; presence does not reliably lift overall crawler interest.
  • Large-domain adoption crawls (for example Invisit on Tranco): a minority of sites serve a real file; major operators still do not document third-party llms.txt as a Search or assistant ranking lever; Google Search ignores it for ranking controls.

None of that means the proposal is useless for your own agents. Separate research on docs sites shows token savings when an agent is instructed to start at /llms.txt. The lever is the prompt you control, not speculative discovery by someone else’s crawler.

When llms.txt is still worth shipping

Use this as a portfolio rule, not a slogan.

Site type Verdict Why
API, SDK, or developer docs you own Ship and maintain Coding agents and internal assistants are the readers that show up; a curated Markdown map can cut token waste when the prompt says to open it first
CMS that auto-generates the file Keep, then QA Cheap to leave on; expensive if it lists retired URLs and wrong prices
Product marketing / SaaS homepage story Optional, one afternoon max Fine if you can point at pricing and docs mirrors; skip if nobody will update it
News, large blog, or huge catalogue Skip for now Curating thousands of changing URLs is real labour for near-zero fetch rates
“We need it for Google / AI Overviews” Skip Search does not use it as robots-for-AI; invest in crawlable HTML and field Core Web Vitals instead
You block the AI bots in robots.txt that would need the map Skip or fix the conflict A guide for agents you refuse at the gate has no compliant readers

Worth it, then, when someone you control will fetch it, or when generation is free and the file stays accurate. Not worth it as a retainer line item sold as AI-search visibility. The table is the whole argument. The rest is how you defend the skip without sounding lazy.

Minimum maintainable llms.txt (or a clean skip)

If you ship one:

  1. Keep it short: H1, a few Markdown links to canonical docs, pricing, and contact; no dump of every blog URL.
  2. Link formats agents and Lighthouse audits expect ([label](url)), not bare URLs alone, if you care about experimental Agentic Browsing checks.
  3. Put an owner and a refresh trigger (release notes, pricing change). A stale file is worse than no file because agents trust confident wrong answers.
  4. Log fetches for a month. If nothing hits /llms.txt except your own checkers, stop debating prose and either wire it into an agent prompt you own or delete the ticket.

If you skip, say so in the ticket with the study base rate and the Google Search stance. That is a stronger professional answer than a decorative file, and clients usually accept a sourced skip faster than a file nobody can prove is read.

What to measure instead of llms.txt for AI visibility

Agencies still need an AI-adjacent story clients will fund. Prefer work that shows up in logs and field data: crawlability and performance for the bots you allow (see the Watcher AI crawlers piece); scheduled lab and CrUX coverage on the URLs that earn revenue, not a homepage-only paste; and answer or citation checks you can repeat without a GEO SaaS tax (Watcher’s monitor-without-subscription angle). llms.txt can sit beside that stack as an optional docs affordance, but it should not be the centrepiece of the retainer narrative.

FAQ

Does Google Search use llms.txt?

Google has stated Search does not treat it as a ranking or robots-style control. Do not sell it as a Search Console lever.

Should we add llms.txt because Lighthouse Agentic Browsing mentions it?

Treat that as an experimental readiness check, not proof of traffic. Fix or ship the file if you already want agent docs; do not reorder the whole sprint for a fractional lab audit.

Is a zero-fetch file a security problem?

The bigger practical risk is a compromised or stale file that agents trust. If you publish, keep ownership and review. If you cannot, omit it.

Put the llms.txt ship-or-skip verdict in your runbook

Agency technical leads: before the next “add llms.txt for AI visibility” ticket, pick a row from the table above and write the verdict into the runbook. That is the same upkeep question from the opening, answered with a sourced ship or skip instead of another unchecked box. Use Why AI crawlers need fast, crawlable pages for fetchability work that still compounds, and Monitor AI search optimisation without a subscription when the client wants measurement without another dashboard. Leaving an unread file as the plan is how retainers fund silence. If you want scheduled PageSpeed and Core Web Vitals history across the portfolio while that debate settles, start a Watcher trial.

References

Top comments (0)