DEV Community

Merlonix
Merlonix

Posted on Originally published at merlonix.com

llms.txt Explained: What It Is, Whether AI Answer Engines Read It, and How to Write One

#ai #webdev #devops #seo

Originally published on the Merlonix blog.

/llms.txt is the file everyone adding "AI SEO" to their site keeps hearing about and nobody quite explains. It gets lumped in with robots.txt because it lives in the same place and has a similar name, so people assume it's another access-control file — a way to let the good AI crawlers in or keep them out. It is not. robots.txt says who may fetch what. llms.txt says here is what matters, and where it is. One is a gate; the other is a table of contents. Adding an llms.txt grants no access and blocks nothing — a crawler that a WAF or robots.txt rule is keeping out will never see it, and a crawler that's already allowed in doesn't need your permission to read it. What it does is make a site legible to an agent that has to otherwise reverse-engineer your navigation to find the three pages that actually answer the question.

Here's the whole thing: what the file is, the exact format, what to put in it, the honest answer to "does anything actually read it yet," and how to confirm yours is valid rather than just present.

What llms.txt Actually Is

llms.txt is a proposed convention — published at llmstxt.org in late 2024 — for a single Markdown file at your domain root (https://example.com/llms.txt) that curates your most important content for a large language model. The premise is simple: an LLM working with a limited context window can't crawl and hold your entire site, and rendered marketing HTML is noisy — nav, cookie banners, footers, scripts. A short, hand-curated Markdown file that names your key pages and links straight to them is far cheaper for an agent to consume than making it guess which of your 400 URLs are the canonical docs.

It is not a standard in the RFC sense — it's a community convention with growing adoption, mostly among documentation sites and developer tools. It is also not:

  • robots.txt — that's access control (who may crawl what), a cooperative protocol enforced, or overruled, at the edge. llms.txt is curation and grants nothing.
  • A sitemap — sitemap.xml is an exhaustive, machine-generated list of every URL for a search crawler to discover. llms.txt is the opposite: short, hand-picked, human-written, and opinionated about what's important. A sitemap says "here is everything"; llms.txt says "here are the six things that matter."
  • JSON-LD structured data — that annotates a single page's meaning inline (this is an Article, this is a Product). llms.txt operates at the site level, pointing across pages.

Think of it as the README you'd hand a new hire who has ten minutes: not the whole codebase, just "start here, then here, and here's where pricing lives."

The Format

The spec is deliberately minimal, and it's just Markdown, so it renders fine for a human too. The shape:

# Project or Site Name

> An optional one-line blockquote summary of what this
> site or project is, so an agent gets the gist immediately.

Some optional free-form Markdown here — a paragraph of
context, key facts, whatever an LLM should know first.

## Docs

- [Getting started](https://example.com/docs/start): the five-minute setup
- [API reference](https://example.com/docs/api): every endpoint, typed

## Guides

- [Deploying to production](https://example.com/guides/deploy)
- [Troubleshooting TLS](https://example.com/guides/tls)

## Optional

- [Changelog](https://example.com/changelog)
Enter fullscreen mode Exit fullscreen mode

The pieces that matter:

  • An H1 with the site or project name. This is the one genuinely required element.
  • An optional blockquote (>) immediately after, holding a short summary. Agents are told to treat this as the elevator pitch.
  • H2 sections grouping links by kind (Docs, Guides, API, About…). Each list item is a Markdown link, optionally followed by : a short description of what's behind it.
  • An ## Optional section has a special meaning in the spec: everything under it is skippable — an agent tight on context can drop it and lose nothing essential. It's your way of marking the "nice to have" links as lower priority.

There's also a companion file, llms-full.txt, for sites that want to inline the actual content of their key pages as one big Markdown document rather than just linking to them — useful for docs sites that want an agent to ingest everything in one fetch. llms.txt links; llms-full.txt includes. Most sites only need the former.

What to Actually Put in It

The failure mode isn't a syntax error — it's an llms.txt that's technically valid and practically useless because it lists your blog's forty most recent posts, or duplicates your nav. Curate it the way you'd brief a smart stranger:

  • The pages that answer real questions — pricing, docs, the "how it works" page, the API reference. Not the press kit.
  • A summary that states what you are in one sentence, in the blockquote. A single sentence that names what your product does tells an agent more than your whole homepage hero.
  • Descriptions on the links, not just URLs. [Pricing](…): four tiers from free to $699/mo, no per-seat billing is a citable fact; a bare link isn't.
  • Keep it short. The entire point is a small, high-signal file. If it's as long as your sitemap, you've rebuilt your sitemap.

Does Anything Actually Read It? The Honest Answer

This is the question that matters and the one most "add an llms.txt today!" posts skip, so here it is straight: adoption on the publishing side is real and growing; consumption by the major answer engines is not something anyone can currently guarantee. OpenAI, Anthropic, Google, and Perplexity have not committed to reading llms.txt at retrieval time the way they commit to honoring robots.txt. Some developer-facing tools and agent frameworks do look for it. The big consumer answer engines mostly still crawl and parse your rendered pages directly.

So why add one? Three reasons that hold regardless of whether ChatGPT reads it this quarter:

  1. It's a curation artifact you own. Even if no engine parses it today, it's a canonical, version-controlled statement of "these are our important pages," which is useful to you, to internal agents, and to any tool that does consume it.
  2. The cost is near zero and the downside is zero. It's a static Markdown file. It can't slow your site, break rendering, or hurt your search ranking. The risk/reward is lopsided.
  3. It's standards-track and the trend line is up. Betting a five-minute static file on a convention that documentation platforms are adopting is a cheap option on a likely future, not a speculative rewrite.

What you should not do is treat it as a substitute for the things engines demonstrably do use today — being crawlable (robots.txt + no WAF rule silently 403-ing the answer engines) and being parseable (JSON-LD structured data). llms.txt is the third layer, not the first. If an answer engine can't fetch your pages at all, a perfect llms.txt changes nothing.

Where It Sits in the Bigger Picture

Getting cited by an AI answer engine is three questions in order, and llms.txt is only the last one:

  1. Can the crawler reach you? That's robots.txt and your edge — and they routinely disagree, because a WAF or a "block AI bots" toggle can 403 a crawler your robots.txt explicitly allows. (Different failure entirely; we wrote up why blocking GPTBot doesn't remove you from ChatGPT — the answer-engine crawlers are different user-agents from the training ones.)
  2. Can it understand the page it fetched? That's JSON-LD structured data — telling an engine this is an Article / a Product / an FAQ instead of making it infer meaning from HTML.
  3. Can it find your important pages fast? That's llms.txt — the curated table of contents.

Skip step 1 and the other two are moot. Nail all three and you've done everything a site can currently do to be legible to an AI agent.

How to Check Yours Is Actually Valid

"Present" and "valid" are different. A file that returns 200 but is empty, or is a wall of prose with no headings or links, is present-but-useless — and an agent parsing it as Markdown gets nothing structured out of it. Verify it the way a consumer would:

  1. Fetch it and confirm it's really there: 
   curl -s -o /dev/null -w "%{http_code}\n" https://example.com/llms.txt
Enter fullscreen mode Exit fullscreen mode

A 404 means you don't have one. A 200 that serves your HTML error page (a SPA catch-all route) is worse — it "exists" but is garbage.

  1. Confirm it looks like Markdown, not a text dump. A real llms.txt has at least one heading (#) or one Markdown link ([text](url)). If it has neither, a parser can't extract structure from it.
  2. Read it as the brief it's meant to be. Does the blockquote state what you are? Do the links go to the pages that answer questions, with descriptions? Is it short?

If you'd rather not eyeball it, the free AI Agent-Readiness checker does exactly this from outside your stack: it fetches your /llms.txt, checks that it's present and shaped like real Markdown (a heading or a link — not an empty file or an error page served in its place), and folds that into a 0–100 score alongside the two things that matter more today — whether your robots.txt actually lets the answer-engine crawlers in, and whether your pages carry JSON-LD structured data. It tells you which of the three layers is your weak point, which is the only question worth acting on. No signup, one domain at a time.

The one-line version: llms.txt is a curated, Markdown table of contents you write for AI agents — cheap to add, standards-track, and legible, but it is not access control and it is not yet something the major answer engines promise to read. Add it because it's a near-free option on a likely future and a canonical statement of your key pages — but add it after you've confirmed the crawlers can actually reach you and parse you, because those are the layers engines use today.

Merlonix watches all three layers the way it watches SSL, DNS, and domain expiry: continuously, and from outside your infrastructure, so a redeploy that drops your llms.txt, a new WAF rule that 403s the answer engines, or a template change that strips your JSON-LD doesn't quietly erode your AI-answer visibility weeks before anyone notices the referral traffic fall off. Run the free agent-readiness scan to see where a domain stands today, check its live SSL and DNS while you're there, and browse the rest of the free tools. Being findable by an agent is a configuration problem you can mostly fix this afternoon — once you know which of the three layers is the one holding you back.

Top comments (0)