llms.txt: what it is and how to add one

Originally published on the Lyra blog.

An llms.txt file is a plain Markdown file you host at /llms.txt that hands AI crawlers a clean, curated map of your site's most important pages. It is not styled for people. It is a shortlist a language model can read in one pass: your site name, a one-paragraph summary, and a few sections of links with short descriptions. The idea, proposed by Jeremy Howard in 2024, is simple. Models work better with context than with raw HTML, so give them the context directly.

This guide covers what goes in an llms.txt file, whether it actually moves the needle, how it differs from robots.txt and a sitemap, and how to write and host one in a few minutes.

What is llms.txt and what goes in it?

llms.txt is a Markdown file at the root of your domain that points AI models at the pages that matter, with a sentence of context on each. A browser will render it as text. A model reading it gets a quick, structured sense of what your site is and where the good parts live, without crawling and parsing every page first.

The format is loose but conventional. A well-formed file has four parts:

1. An H1 with the name of your site or project. This is the only required line.
2. A blockquote summary: two or three sentences on what the site is and who it serves.
3. Optional free-text notes under the summary for anything a model should know up front.
4. H2 sections of links, each link followed by a short description after a colon.

Because it is just Markdown, it reads the same way structured content reads to a model: predictable headings, plain prose, no markup noise. That predictability is the point. The same instinct drives good answer engine optimization: answer the question plainly, structure it cleanly, and machine readers pick it up. An llms.txt file applies that thinking at the level of your whole site instead of a single page.

It pairs with the technical hygiene that already helps you rank. If you have worked through SEO for SaaS, an llms.txt file is a small, fast addition to that same checklist, not a replacement for it.

Does llms.txt actually help?

Honest answer: it is an emerging convention, not a ranking guarantee. No major AI provider has publicly confirmed that an llms.txt file changes how they rank, retrieve, or cite your pages. Adoption among the big crawlers is still uneven, and some teams treat it with healthy skepticism for exactly that reason.

So why add one? A few reasons that hold up even without a confirmed ranking benefit:

• It is cheap. Writing one takes minutes. Maintaining it means editing a text file when you ship or retire a major page. There is no infrastructure and no ongoing cost.
• It removes ambiguity. A curated shortlist with descriptions is easier for any model to parse than a 200-page sitemap or a wall of rendered HTML. You are doing the summarizing instead of hoping the crawler gets it right.
• It is a signal. Shipping one says you think about machine readers as an audience. That mindset, more than the file itself, is what helps you show up in AI answers.

Treat it the way you would treat clean structured data or a tidy sitemap: good practice with a low ceiling on downside, not a magic lever. The win is not the file alone. It is the habit of writing for both people and the models that increasingly sit between people and your site. The same logic is why we fact-check every claim and link in a post before it ships: if a model is going to cite you, the page it cites had better be accurate and easy to read.

If you want to see a live one, Lyra serves her own at trylyra.ai/llms.txt. It is a small, real example of the format described here.

How is llms.txt different from robots.txt and sitemap.xml?

They live in the same place and they all talk to crawlers, but they do different jobs. Mixing them up is the most common confusion, so here is the split.

File	What it does	Who it is for
robots.txt	Sets rules: which paths crawlers may or may not access.	Crawlers deciding what to fetch.
sitemap.xml	Lists every URL so crawlers can discover them all.	Search engines indexing the full site.
llms.txt	Curates your best pages with plain-language descriptions.	Language models that want context fast.

A sitemap is exhaustive. It wants to list everything. An llms.txt file is the opposite: it is opinionated and short, a highlight reel of the pages you would actually want an AI to read and quote. robots.txt is about permission, not content: it decides which crawlers may read you, and getting the AI crawler rules right is its own job, because blocking the wrong bot quietly deletes you from AI answers. None of the three replaces the others. You can and usually should have all three.

One nuance worth keeping straight. Some teams also publish full-text versions of pages at paths like page.md or an llms-full.txt that inlines whole documents. That is an extension, not a requirement. The core llms.txt file is just the map.

How do I write an llms.txt file?

Start with the pages you would hand a new hire on day one. Your homepage, your core product or pricing pages, your docs, and a handful of your strongest blog posts. Skip thin pages, duplicates, and anything expired. Quality over coverage. A model gets more from 15 well-described links than from 150 bare URLs. Your docs pages deserve the same scrutiny as the rest of the list: they carry the highest concentration of AI agent traffic on most sites, so getting them structured for docs SEO matters at least as much as which links make the file.

Then write the file. Here is a minimal example you can adapt:

# Acme Analytics

> Acme Analytics is a privacy-first product analytics tool for small SaaS
> teams. We help founders see which features drive retention without
> shipping user data to third parties.

Self-serve, no credit card to start. Docs are public and versioned.

## Product

- [Overview](https://acme.com/): What Acme does and who it is for.
- [Pricing](https://acme.com/pricing/): Plans, limits, and the free tier.
- [Integrations](https://acme.com/integrations/): Supported sources and SDKs.

## Docs

- [Quickstart](https://acme.com/docs/quickstart/): Install and send your first event.
- [API reference](https://acme.com/docs/api/): Endpoints, auth, and rate limits.

## Blog

- [Retention metrics that matter](https://acme.com/blog/retention/): The four numbers we track.
- [Self-hosting Acme](https://acme.com/blog/self-host/): Run it on your own infra.

A few rules that keep it useful:

• Use absolute URLs. A model may read the file out of context, so relative paths can break.
• Write descriptions, not labels. "Pricing: plans, limits, and the free tier" tells a model more than "Pricing" alone.
• Keep the summary specific. Say what you do and who you serve. Skip the marketing adjectives.
• Stay current. When you ship or kill a major page, update the file. A stale map is worse than none.

How do I host it at /llms.txt?

The file has to resolve at your domain root, at https://yourdomain.com/llms.txt, and return plain text or Markdown. How you get it there depends on your stack.

• Static sites: drop llms.txt in your public/ or root output directory. Most static hosts serve it as-is.
• Next.js, Astro, and similar frameworks: put it in the public/static folder, or add a route that returns the text with a text/plain or text/markdown content type.
• A CMS or platform you do not control: check whether it lets you add a root-level file or a custom route. If not, a redirect or edge function can serve it.

Once it is live, open the URL in a browser and confirm you see your raw Markdown, not a 404 and not an HTML page wrapping it. That is the whole job. There is no registry to submit to and no verification step.

Where this fits in your AI search strategy

An llms.txt file is one small piece. It makes your site easy to map. It does not write the pages that make the map worth reading. Those still come from clear, accurate, well-structured content that answers real questions, which is what gets cited whether or not a crawler reads your llms.txt first. The file is a pointer; the pages are the payload.

This is where the work scales or stalls. Maintaining the file is trivial. Producing a steady stream of pages worth pointing to is the hard part, and it is exactly the gap Lyra fills. Lyra is an autonomous writer who finds topics worth covering, drafts them in your blog's existing voice, fact-checks the claims and links, scores the draft, and opens a GitHub pull request for you to review. Nothing publishes on its own. You bring your own Anthropic key, and Lyra is in early access while we build in the open, so talk to the founder to see whether she's a fit. If you want the deeper version of how an autonomous AI blog writer works end to end, the pillar page walks through it.

Add the llms.txt file this afternoon. It is five minutes of work with no real downside. Then spend your energy on the content it points to, because that is what actually earns the citation.

---

I'm building Lyra, an autonomous blog writer that writes in your blog's voice, fact-checks every claim, and opens a pull request you review. This post comes from her blog, where we publish what we learn running the pipeline. Happy to answer questions in the comments.