DEV Community: Roger Remacle - Lab451.org

llms.txt vs llms-full.txt: What's the Difference? (2026)

Roger Remacle - Lab451.org — Wed, 17 Jun 2026 15:10:00 +0000

This is a cross-post from lab451.org/blog. I've been working on tooling for the llms.txt standard for a few months and the most common question I get is "which file do I actually need." Here's the answer at length.

They sound nearly identical. They live next to each other at the root of your site. They serve the same broad audience. But they do fundamentally different jobs — and knowing which one to ship (or whether to ship both) is the difference between AI models understanding your site and AI models actually answering questions about it.

The TL;DR

If you only read one paragraph: llms.txt is a small Markdown file that tells AI models what your site is and where the important pages live. llms-full.txt is a much larger Markdown file that contains the actual text of those pages already extracted, cleaned, and concatenated. The first is a table of contents; the second is the book.

Marketing sites and small blogs usually only need llms.txt. Documentation sites, API references, knowledge bases, and anything technical where you actually want models to be able to answer questions about your content should ship both.

Two files, two jobs

The reason there are two files instead of one comes down to a tradeoff every LLM has to make: context window space is expensive, but fetching pages is slow. Different situations call for different sides of that tradeoff.

When a user asks ChatGPT "what does Lab451 do," the model needs the smallest possible amount of context to answer accurately. It doesn't need your full pricing page, your terms of service, or every blog post — it needs a sentence or two. llms.txt is exactly that: a tiny file the model can fetch in milliseconds, parse in a few tokens, and use to give a quick, accurate one-paragraph answer.

When a user asks ChatGPT "what's the best way to set up llms-full.txt for a 500-page documentation site," the model needs much more. It needs to understand your full documentation, find the relevant sections, and synthesize specifics. Following links one by one from llms.txt would mean ten or twenty separate fetches and a lot of wasted context. llms-full.txt sidesteps that entire dance: download once, answer in detail.

The two files don't compete — they complement each other. The model can grab whichever is the right tool for the question being asked. Some models check both in sequence; some pick one based on the query depth. Either way, having both available means you've covered both ends of the tradeoff.

Side by side

	llms.txt	llms-full.txt
Purpose	Map of your site	Full text of your site
Format	Structured Markdown (H1 + blockquote + H2 link lists)	Free-form Markdown (concatenated page bodies)
Typical size	1–10 KB	100 KB – several MB
Token cost when read	~250 – 2,500 tokens	~25,000 – 1,000,000+ tokens
Update frequency	When site structure changes	Every meaningful content change
Best for	Any site (marketing, blog, SaaS landing)	Docs, API references, knowledge bases, technical content
Model behavior	Read once, follow links as needed	Read once, answer from memory
Hosted at	`/llms.txt`	`/llms-full.txt`
Required?	Recommended for everyone	Recommended for content-heavy sites

Format differences in detail

Both files are Markdown, both are plain text, both live at the root of your domain. The differences are structural.

llms.txt: prescriptive structure

llms.txt follows a tight, parseable shape. There's exactly one H1, an optional blockquote summary, optional free-form Markdown, and then H2 sections each containing nothing but link lists. A spec-compliant parser can extract these elements deterministically. Here's the canonical shape:

# Lab451

> Lab451 generates llms.txt, llms-full.txt, sitemap.xml, and robots.txt
> for any public website in about 30 seconds.

## Docs

- [Getting started](https://lab451.org/docs/quickstart): Generate your first set of files
- [API reference](https://lab451.org/docs/api): Endpoints, authentication, rate limits

## Optional

- [Terms of service](https://lab451.org/terms)
- [Privacy policy](https://lab451.org/privacy)

Notice what's not there: no page content. No paragraphs from the Getting Started doc. No code samples from the API reference. Just titles, URLs, and one-line descriptions. The file is a finger pointing at the pages, not the pages themselves.

llms-full.txt: prescriptive content, flexible structure

llms-full.txt takes a different approach. The format is much looser — there's no required shape — but the content requirements are stricter. It should contain:

The full text of every important page on your site
Cleaned of navigation, headers, footers, cookie banners, and chrome
Converted to clean Markdown (or at minimum, structured plain text)
Concatenated into a single file with clear section breaks between pages

Here's a shortened example of what one looks like in practice:

# Lab451 — Full Documentation

---

## Getting Started

Lab451 generates the four files AI models need to understand your site.
You give it a URL, choose a file type, and click Generate. The crawler
maps your site, extracts content, and produces spec-compliant output.

To get started, visit lab451.org and paste your domain. The free plan
handles sites up to 50 pages without an account...

---

## API Reference

### Authentication

All API requests require a Bearer token in the Authorization header.
Get your token from the Account page under "API Keys"...

### Endpoints

#### POST /api/generate

Generates a single file type for a given domain. Required parameters:

- `domain` — the target URL (must include https://)
- `fileType` — one of: llms, llms-full, sitemap, robots
- `maxPages` — page cap (defaults to plan limit)
...

The --- horizontal rule between sections is convention, not requirement. What matters is that each page is identifiable as its own chunk, the headings reflect the original page hierarchy, and the model can navigate to any subsection by scanning H2s and H3s.

Size, budget, and context windows

The size difference between the two files is enormous, and it has real consequences for how models consume them.

A typical llms.txt file for a small-to-medium site is between 1 KB and 10 KB. That's roughly 250 to 2,500 tokens — a tiny fraction of any modern model's context window. Reading llms.txt is essentially free, which is why models will cheerfully fetch it on almost any query that touches your domain.

llms-full.txt is a different beast. A documentation site with 200 pages of meaningful content might produce a 500 KB file — around 125,000 tokens. That fits comfortably in modern long-context models (GPT-4o, Claude 4, Gemini 2.5 Pro all handle this easily), but it's a real chunk of context the model has to weigh against everything else in the conversation.

The practical limits as of mid-2026:

File size	Token equivalent	Status
Under 50 KB	~12,500 tokens	Read fully by every major model
50 KB – 500 KB	~12,500 – 125,000 tokens	Read fully by long-context models; chunked or summarized by smaller ones
500 KB – 2 MB	~125,000 – 500,000 tokens	Read partially; models may retrieve only relevant sections
Over 2 MB	500,000+ tokens	Usually retrieval-only; rarely loaded whole

The honest takeaway: if your llms-full.txt is over a megabyte, you're approaching the practical ceiling. Beyond that, models increasingly fall back to retrieval-style consumption (grep for relevant chunks) rather than holistic reading. That's not necessarily bad — it still works — but it changes the equation. For the largest sites, the answer isn't "bigger llms-full.txt"; it's "smarter chunking and well-named sections."

💡 Rule of thumb: aim for an llms-full.txt under 500 KB if possible. If you're over that, scrutinize what's actually in there. Old blog posts, deprecated docs, terms of service, and changelogs rarely earn their place. The point is to give models the content that actually answers questions, not every word you've ever published.

Which AI models read which file

As of mid-2026, the picture is uneven but converging. Some crawlers explicitly fetch both files; some only fetch llms.txt; some only honor llms-full.txt when explicitly linked. The practical state of play:

Crawler	llms.txt	llms-full.txt
ChatGPT (GPTBot, OAI-SearchBot)	Yes	Yes (when discoverable)
Claude (ClaudeBot)	Yes	Yes
Perplexity (PerplexityBot)	Yes	Yes
Google (Googlebot, Google-Extended)	Indexed	Indexed
Bing / Copilot (Bingbot)	Indexed	Indexed
Grok (xAI-Bot)	Yes	Partial
Mistral, Meta	Partial	Partial

"Yes" means the crawler reliably fetches the file and there's reasonable evidence it's used. "Indexed" means the file gets indexed alongside other site content but its specific use is unclear. "Partial" means fetching happens but isn't consistent across all queries.

The pragmatic conclusion: any crawler that fetches llms.txt will follow a link from it to llms-full.txt if one is listed in the Optional section. So even if a model doesn't crawl llms-full.txt as a well-known URL, mentioning it from your llms.txt ensures it gets discovered.

When to ship just llms.txt

There are good reasons to skip llms-full.txt entirely. Ship only llms.txt if any of these apply:

You're a marketing site. Your homepage, About, Pricing, and Contact pages don't need to be read in detail by an AI model. Users asking AI about you want a one-paragraph summary; llms.txt delivers that perfectly.
You're a small blog. Individual posts are better read in their original context (where you have analytics, ads, related-posts widgets). Pointing models at the posts via llms.txt is enough; serving the text again in llms-full.txt just duplicates content without strategic benefit.
Your content is rapidly changing. A news site, stock-tracker, or live-event dashboard would have to regenerate llms-full.txt constantly. The maintenance overhead outweighs the benefit; better to let crawlers hit the live pages via llms.txt.
Your content is gated. If most of your pages require authentication or payment, you can't put their full text in a publicly-served llms-full.txt. Listing the public-facing summaries in llms.txt is the right level of disclosure.
Your content is primarily visual or interactive. Tools, calculators, configurators, and data visualizations don't flatten into text well. Pointing models at the tool URL via llms.txt is fine; trying to describe the UI in llms-full.txt is usually worse than nothing.

When to ship both

The case for shipping both is strongest when you have text-heavy content where the value is in the specifics — exact API parameters, exact installation steps, exact configuration syntax. Specifically, ship both when:

You have documentation. If a user could plausibly ask an AI "how do I do X with your product," and the answer requires pointing to specific function signatures, command flags, or exact configuration syntax, you want llms-full.txt. Models will quote your docs back at users; you want them quoting from a canonical, clean source.
You have an API. API references thrive in llms-full.txt. The whole point is that a model can pull up your endpoint table, parameter list, and response format in a single fetch and answer questions accurately.
You have a knowledge base or help center. Support articles often answer "how do I X" questions that AI assistants now field on your behalf. Putting them in llms-full.txt means AI gives the same answer your support team would.
You have evergreen technical content. Tutorials, guides, and walkthroughs benefit from being in llms-full.txt for the same reason — the value is in the specifics, and you want models quoting your version rather than a stale paraphrase.
You want to be the canonical source on a topic. Being in llms-full.txt raises the odds that a model quotes your phrasing when summarizing the topic. If you've written the definitive guide on something, having it in llms-full.txt is the difference between "according to Lab451" and "according to a guide I read somewhere."

How they work together at inference time

Watching a real model handle a query that triggers both files is instructive. Here's a simplified trace of how a request like "how do I add llms.txt to a Next.js site" might flow through a system that supports both files:

The model needs to answer a specific technical question. It identifies potentially relevant sites (Lab451 included).
It fetches lab451.org/llms.txt first — small, fast, cheap. From this, it learns Lab451 is an llms.txt generator and that there's a "Docs" section containing relevant pages.
It sees that llms.txt mentions a /llms-full.txt in the Optional section. The model decides — based on query complexity — to fetch it.
It pulls down lab451.org/llms-full.txt, finds the "Adding llms.txt to your site" section, finds the Next.js example, and quotes the relevant configuration directly.
The user gets a precise answer with a citation pointing back to lab451.org.

Without llms-full.txt, step 3 would instead trigger a chain of fetches — first the Docs index, then the "Adding llms.txt" page, then the Next.js page — each one a separate request with its own HTML parsing, navigation chrome stripping, and context cost. The model probably still gets there, but it takes longer, costs more context, and is more likely to grab the wrong content along the way.

Both files are tools. llms.txt is the cheap, fast tool that handles 80% of queries. llms-full.txt is the heavier tool that handles the 20% where specifics matter. Shipping both means the model can pick the right one.

Generating llms-full.txt without going insane

The reason a lot of sites only ship llms.txt isn't that they don't want both — it's that maintaining llms-full.txt by hand is miserable. Concatenating every page's content into a single file, stripping nav and chrome, keeping it in sync as the site evolves — that's a job for a script, not a human.

A few practical approaches:

Static-site generators with built-in support

Mintlify, Fern, Docusaurus (via plugin), and most modern docs platforms now ship llms-full.txt generation out of the box. If your docs already build to static HTML, check whether your generator can also emit llms-full.txt in the same build step. This is by far the lowest-effort path.

Build-time scripts

For sites without built-in support, a build script can extract Markdown from your content directory, strip frontmatter or normalize it, and concatenate everything into a single file at /llms-full.txt. This works especially well for Hugo, Eleventy, Astro, and Next.js sites where content already lives as Markdown files.

Crawl-based generation

For sites that don't have source Markdown — WordPress sites, headless CMS sites, or anything served dynamically — the right answer is to crawl your own site and convert each page's rendered HTML to clean Markdown. This is what Lab451 does, and what other tools in this space do too.

Hybrid: generate, then curate

The highest-quality llms-full.txt files combine automated generation with editorial review. Auto-generate the file from your docs source, then have a human pass over it once to remove anything that shouldn't be there (deprecated content, internal-only notes, accidental duplicates). This is overkill for most sites, but for teams whose AI presence matters strategically, it's worth the quarterly hour.

Common mistakes

A few specific failure modes to avoid:

Putting full page content in llms.txt

The biggest mistake by far. llms.txt is a map, not the territory. If your llms.txt contains paragraphs of body content, you've confused it with llms-full.txt. Move the body content into llms-full.txt, and put only links and descriptions in llms.txt.

Linking to llms-full.txt only from llms-full.txt

llms-full.txt isn't typically discovered by crawlers as a well-known URL the way llms.txt is. The way crawlers find it is by reading llms.txt and following a link. So if you ship both, make sure llms.txt mentions llms-full.txt in its Optional section.

Leaving navigation chrome in llms-full.txt

If your llms-full.txt generator pulls page HTML and converts to Markdown without first stripping the header, footer, sidebar, and cookie banner, you end up with a file where 30% of the content is "Home | About | Contact" repeated 200 times. Models will still parse it, but they waste context on noise. Clean extraction is table stakes.

Including pages that shouldn't be there

Search result pages, tag archives, pagination, login pages, and user-account pages don't belong in llms-full.txt. Filter them out at generation time. The rule of thumb: if a page has unique, canonical, evergreen content that someone would want to read, it belongs. Otherwise it doesn't.

Letting it go stale

An llms-full.txt that's six months out of date is worse than no llms-full.txt at all — models will confidently quote outdated pricing, deprecated API endpoints, and old product names. Tie regeneration to your deploy pipeline if you can, or set a monthly cron job. Make staleness impossible.

FAQ

Can I have llms-full.txt without llms.txt?

Technically yes; practically no. llms-full.txt is discovered via llms.txt in most crawler implementations. Without an llms.txt linking to it, your llms-full.txt sits unread. Always ship them together.

Is llms-full.txt just a single big Markdown file?

Yes. The whole point is that a model can fetch one URL and get everything. Splitting it across multiple files defeats the purpose. If your content is genuinely too big for one file, the answer is smarter content curation, not file splitting.

What if my site is mostly visual or interactive?

Skip llms-full.txt. There's no useful way to flatten a Figma template gallery or an interactive calculator into Markdown. llms.txt alone, pointing at descriptive pages, is the right choice.

Does llms-full.txt hurt my SEO?

No. llms-full.txt isn't indexed as a web page in the traditional sense — it's a resource file, like robots.txt or sitemap.xml. Google and Bing index its existence but don't treat it as a duplicate of your pages. The content within isn't competing with your real pages for rankings.

Should I gate llms-full.txt behind authentication?

No. If you want models to read it, it has to be publicly fetchable. If you have content you don't want models to see, leave it out of llms-full.txt rather than trying to gate the file itself. A gated llms-full.txt is the same as no llms-full.txt.

What about pricing or rapidly-changing content?

Two options. The simpler one: leave volatile content out of llms-full.txt entirely, and rely on the model to fetch the live page via the llms.txt link when asked. The more sophisticated one: regenerate llms-full.txt on a short cadence (hourly cron for pricing, daily for changelogs) so it stays close-enough to current.

How big is too big?

Practical ceiling: around 2 MB. Past that, most models stop reading the file whole and switch to retrieval-style access on chunks. That still works, but you lose the "everything in one shot" property that's the file's whole point. Aim under 500 KB if you can.

If you found this useful, the original post on lab451.org has a few extra resources, and there's a longer guide on the llms.txt spec itself if you want the deeper background. Happy to answer questions in the comments.

llms.txt — what it is, what AI models use it, how to write one, and the mistakes that quietly tank the whole exercise.

Roger Remacle - Lab451.org — Tue, 26 May 2026 08:12:46 +0000

What is llms.txt?
llms.txt is a plain-text file you put at the root of your domain — at https://yoursite.com/llms.txt — that tells large language models what your site is about, which pages matter, and how to navigate it. It's written in Markdown. It's small. It exists for one reason: AI models have very limited context windows, and they can't read your whole site, so you give them a curated map instead.

The standard was proposed by Jeremy Howard of Answer.AI in September 2024, and over the following eighteen months it became the de facto convention for what's now called Generative Engine Optimization, or GEO — the practice of getting your site cited by ChatGPT, Claude, Perplexity, Gemini, and the other large language models that have started to replace the traditional search box.

If you've ever written a robots.txt or a sitemap.xml, you already understand the shape of llms.txt. It's the same idea — a small file at a well-known URL that gives automated systems structured hints about your site — except the audience is language models rather than search-engine crawlers, and the format is Markdown rather than text directives or XML.

Why it exists (the actual problem)
When you ask ChatGPT "what is FastHTML and how do I get started," ChatGPT might do three things behind the scenes: search the web, pick a handful of pages, and try to read enough of them to give you a sensible answer. The second and third steps are where modern sites fall over.

A typical modern web page is 200KB to several megabytes of HTML, CSS, JavaScript, third-party scripts, cookie banners, navigation chrome, and ads. An LLM's context window — even a generous one — is small relative to that. The model has to either render the page through some kind of browser-like pipeline (slow, expensive, fragile) or strip the markup down to text (loses structure, mangles tables, drops semantics). Either way, by the time the actual content reaches the model, it's degraded.

Worse, the model has no idea which pages on your site are important. Is your homepage the canonical statement of what you do? Is the About page the one to read for context? Is your documentation under /docs, /help, /wiki, or somewhere else? Without explicit guidance, the model guesses. It often guesses wrong.

llms.txt solves both problems at once. It's a small, pre-cleaned, Markdown-formatted document that says: "This is what this site is. Here are the pages that matter. Here's where to go for deeper information." The model spends a fraction of its context budget and walks away with an accurate picture of your site.

Who actually reads llms.txt
This is the question every skeptical post about llms.txt opens with, and it deserves an honest answer. As of mid-2026, support is uneven but growing. Here's the realistic state of play:

None of these companies have published a formal commitment that says "we use llms.txt and weight it at X." What you'll see in practice is that crawlers fetch the file, log it, and use it as one signal among many. The honest framing is closer to schema.org markup in 2014 than to robots.txt in 2024: not strictly required, not universally honored, but adopted quickly enough that not having one is starting to look like a tell.

The defensible position in 2026: ship an llms.txt because it costs you almost nothing, it makes your site more legible to anything that does read it, and the downside if no model ever reads it is zero. It's a hedge with no premium.

The spec, line by line
The formal specification at llmstxt.org is short — short enough to walk through end to end. Here's the structure, with every section explained.

# Site or project name

> A one-paragraph blockquote summary.
> This is the only section besides the H1 that is parsed structurally.

Optional free-form Markdown details about the project. Paragraphs,
lists, anything except headings.

## Docs

- [Page name](https://example.com/page): Optional one-line description
- [Another page](https://example.com/another): What it covers

## Examples

- [Example](https://example.com/example): One-line context

## Optional

- [Less critical link](https://example.com/extra): Skippable if context is tight

Exactly one H1. This is the only required element. It's the name of the site or project, not a tagline.
An optional blockquote. This is your "elevator pitch" summary. Models often quote it verbatim when asked "what is this site about." Make it good. One or two sentences, plain English.
Free-form Markdown after the blockquote can contain paragraphs, lists, and emphasis — anything except additional headings, until you hit the H2 sections.
H2 sections containing link lists. Each H2 is a category (Docs, Tutorials, API Reference, Blog, Examples). Each list item is a Markdown link, optionally followed by a colon and a one-line description.
The "Optional" H2 is special. Links here can be skipped by parsers that need a shorter context. Use it for secondary material — appendices, deeper references, anything not essential to understanding what your site is.

What's not allowed: images, HTML, tables, code blocks, additional H1s, or nested headings inside the H2 link sections. Parsers expect Markdown text and Markdown lists only. The simpler you keep it, the more reliably it's read.

llms.txt vs llms-full.txt
The base llms.txt file is a map. It tells a model where to go, but to follow the links the model still has to fetch each page. For documentation sites and other content where you want the model to have everything in one shot, there's a companion file: llms-full.txt.

llms-full.txt contains the actual content of every page on your site, concatenated into a single Markdown document. No navigation, no boilerplate, no chrome — just the words. A model can download one file and have a complete picture of your product without making fifty follow-up requests.

Here's a simplified mental model of the two:

For a marketing site or a small blog, llms.txt alone is plenty. For a documentation site, an API reference, a product manual, or anything you actually want models to be able to answer questions about, ship both. The pairing was popularized by Mintlify and Anthropic in early 2025 and has stuck.

How it relates to robots.txt and sitemap.xml
These three files cover three different audiences with three different purposes. They complement each other; they don't replace each other.

A common confusion: llms.txt doesn't block anything. It's not a permissions file. If you want to keep crawlers off certain paths, that goes in robots.txt. If you want them to know where your pages live for indexing, that's sitemap.xml. llms.txt is purely a here's what we are document.

One subtle interaction: your llms.txt should not link to pages you've blocked in robots.txt. Some crawlers will silently drop the inconsistent links; others might log an error and deprioritize the whole file. Keep the three in agreement.

Writing a good llms.txt
The mechanical rules above are necessary but not sufficient. A good llms.txt — one that actually changes how models talk about your site — follows a few principles that aren't in the spec.

Lead with the summary that you want quoted back
The blockquote at the top is the single most important sentence in the file. When a model is asked "what is yoursite.com," there's a strong chance it will paraphrase your blockquote. Write it the way you'd want it to appear in a Perplexity answer or a ChatGPT citation. No marketing fluff, no superlatives, no "leading platform for." Plain language, what the site does, who it's for.

# Lab451

> Lab451 generates llms.txt, llms-full.txt, sitemap.xml, and robots.txt
> for any public website in about 30 seconds. Free for sites under 50
> pages. No signup required for the basic flow.

Compare that to what a marketing draft would have produced ("Lab451 is the leading AI-discoverability platform empowering websites to maximize their generative engine presence"). The first version is what a model will repeat. The second is what a model will quietly rewrite into the first.

Group links by intent, not by site architecture
The H2 sections shouldn't mirror your top nav. They should mirror the questions a user might ask a model. If someone asks "how do I get started with X," the model should find a section called Getting Started or Quickstart. If they ask "what does X cost," there should be a Pricing section. Think of the H2s as the answers to the questions you'd expect, not as a sitemap.

One-line descriptions earn their keep
The Title: description pattern is optional, but the description is often what tips a model toward citing one page over another. Keep descriptions to a single line. State what the page covers, not how good it is

## Docs

- [Getting started](https://lab451.org/docs/quickstart):
  Generate your first set of files in under a minute.
- [API reference](https://lab451.org/docs/api):
  Endpoints, authentication, rate limits, response formats.
- [WordPress integration](https://lab451.org/docs/wordpress):
  Drop-in instructions for self-hosted and managed WordPress sites.

Use the Optional section for ballast
Old blog posts, deep references, legal pages, anything you'd be happy for a model to read if it had spare context but wouldn't lose sleep over. The Optional section is permission to include them without crowding the must-reads.

Keep the file under 10 KB if you can
The spec doesn't set a size limit, but in practice the entire llms.txt for most sites should fit comfortably under 10 KB. If yours is larger, you're probably listing pages that belong in the sitemap, not the llms.txt. The map is curated; the sitemap is comprehensive.

Real examples worth copying
Three llms.txt files in the wild that get the format right and are worth studying:

FastHTML — the original reference implementation by Jeremy Howard. Clean H1, tight blockquote, sensible H2 grouping, well-used Optional section. The canonical example.

Anthropic Docs — large documentation site, organized by product surface. Notice how the H2s map to how a developer would ask for help, not how the docs are filed.

Lab451 — short and direct. Useful as a template for marketing sites that don't need the full content treatment.

If you study these in a text editor (right-click, View Source) you'll notice they all share something: there's no clever formatting. No bullet points masquerading as paragraphs, no headings used for emphasis, no nested structures. The spec rewards restraint.

Common mistakes
The errors that quietly tank an llms.txt are mostly format violations the model parses silently and then ignores. Here's the list, in rough order of how often we see them.

Treating the H1 as a tagline
The H1 is supposed to be the name of the site or project, full stop. "# The world's best widget for marketers" is not a name. "# WidgetCo" is. Save the positioning for the blockquote.

Skipping the blockquote
Without the blockquote, models lose the structural cue that tells them "this is the summary." They fall back to inferring it from the link descriptions or the page contents, which is the slow, lossy path you were trying to avoid in the first place.

Linking to pages that 404 or redirect
Sounds obvious; happens constantly. A model that follows a link in your llms.txt and hits a 404 will deprioritize the whole file. Treat llms.txt as production output and check it the same way you check your sitemap.

Adding headings inside the H2 sections
The structure is: H1, optional blockquote, optional free-form Markdown, then H2s with link lists. You can't have an H3 inside an H2 link list. If you need to group further, make more H2s.

Stuffing keywords into descriptions
The traditional SEO instinct is to pack the description with target keywords. In an llms.txt context, this backfires — models are trained to be suspicious of unnatural language and may discount the file. Write descriptions the way you'd explain the page to a colleague.

Forgetting to update it
An llms.txt that's six months out of date and links to a pricing page that no longer exists is worse than no llms.txt at all. Models will cheerfully repeat your outdated information. Build regeneration into the same workflow you use for the sitemap.

Hosting, caching, and deployment
The file lives at /llms.txt at the root of your domain. Not /static/llms.txt, not /.well-known/llms.txt, not /seo/llms.txt. Crawlers look at the root. If you can serve /robots.txt, you can serve /llms.txt — the deployment story is identical.

A few practical notes:

Content-Type: serve as text/plain; charset=utf-8 or text/markdown; charset=utf-8. Either works. Don't serve as application/octet-stream — some crawlers will refuse it.
Caching: a Cache-Control: public, max-age=3600 header is fine. The file changes infrequently; an hour of cache saves you nothing in bandwidth but prevents stale-serving headaches after updates.
HTTPS: serve over HTTPS. Most crawlers will follow from HTTP, but some increasingly treat insecure responses as a quality signal.
Subdomains: each subdomain gets its own llms.txt. blog.example.com/llms.txt and docs.example.com/llms.txt are independent files. Models do not look "up" to the apex.
Multilingual sites: the convention is still settling. The pragmatic answer for now is one llms.txt per language subdirectory (/en/llms.txt, /fr/llms.txt) plus a default at the root. Don't try to mix languages in one file.

Measuring whether it works
The honest answer here is that GEO measurement is in roughly the same place that SEO measurement was in 2002. There's no Google Search Console for "how often does ChatGPT cite you." There are a few proxies, and that's about it.

Things you can measure:

Crawler hits on llms.txt in your server logs. Filter by user-agent (GPTBot, ClaudeBot, PerplexityBot, etc.). Frequency is your strongest signal that the file is being consumed.
Referrer traffic from AI chat interfaces. chat.openai.com, perplexity.ai, claude.ai, gemini.google.com. Numbers will be small but rising. Tag them in your analytics.
Direct citation checks. Periodically ask each major LLM "what is yoursite.com" and see what they say. Save the answers. Track changes after you update llms.txt. This is manual, annoying, and the most reliable signal you'll get in 2026.
Brand mention monitoring. Tools like Mention, Brandwatch, and newer GEO-focused services (Profound, Goodie, Otterly) scrape LLM responses at scale. The category is young; the tools are improving fast.

What you can't reliably measure: ranking, share-of-voice against competitors, or click-through rate from AI answers. The industry hasn't built that infrastructure yet. Anyone who claims they have is selling something.

Where this is heading in 2026 and beyond
A few trends worth watching, beyond the basic adoption curve:

Markdown shadow pages
The same proposal that introduced llms.txt also suggests that important pages should be available at their URL with .md appended — so https://example.com/docs/intro.html would also be reachable at https://example.com/docs/intro.html.md as a clean Markdown version. The FastHTML and nbdev ecosystems already do this. Expect more documentation generators to follow.

Pay-per-crawl economics
Cloudflare's mid-2025 launch of bot-payment infrastructure introduced the idea that AI crawlers might pay micropayments for the content they fetch. By late 2026, expect llms.txt files to start including pricing metadata for their referenced URLs — a 402 Payment Required handshake for the high-value pages. This is speculative but plausible.

Verification and signed manifests
As llms.txt adoption grows, so does the incentive for sites to lie in them. A pricing page that claims to be free, an outdated API spec presented as current — the trust problem is obvious. Expect 2027-era extensions that allow signed llms.txt files (probably via DNS TXT records or HTTPS certificate metadata) for sites that want to make verifiable claims about themselves.

Convergence with schema.org
There's an obvious overlap between llms.txt's "tell models what this site is" purpose and schema.org's "tell search engines what this entity is" purpose. The two haven't merged; they're solving related problems with different tools. Watch for proposals that reference schema entities from inside llms.txt, or schema.org WebSite objects that point to llms.txt locations.

Frequently asked questions
Do I need an llms.txt if I already have a sitemap.xml?
Yes. They serve different audiences and answer different questions. The sitemap tells indexers what exists. The llms.txt tells models what matters. Most sites should have both.

Will llms.txt help me rank in Google search?
Almost certainly not directly. Google has been clear that Googlebot doesn't use llms.txt as a ranking signal. Where it might help is in Google's AI Overviews, where Gemini does seem to weight clean, structured content sources. The honest answer: small, indirect, and not the main reason to ship one.

Should I block AI crawlers in robots.txt instead of writing an llms.txt?
That's a separate decision. If you don't want AI models reading your content, block them in robots.txt and skip llms.txt entirely. If you do want them reading your content, write an llms.txt that makes them efficient at it. The two strategies aren't on a spectrum; they're opposite ends.

Can I dynamically generate llms.txt server-side?
Yes, and many large sites do. The file is just text — there's no requirement that it be a static file. A common pattern is to generate it at build time, cache it for an hour, and regenerate on content changes. Just make sure the response is fast (under 200ms) and stable (returns the same content for the same URL within a reasonable window).

What's the difference between llms.txt and AI.txt?
ai.txt was a 2023 proposal from Spawning that focused on consent — telling AI training pipelines whether they could use your content. llms.txt, proposed a year later by Answer.AI, focuses on structure — telling AI models how to use your site at inference time. They're orthogonal; some sites have both. The industry's center of gravity is firmly on llms.txt.

How often should I regenerate llms.txt?
Whenever your site's structure changes meaningfully — new product pages, restructured documentation, retired sections. For active sites, monthly is a reasonable cadence. For mostly-static sites, regenerate when you ship a notable update. Tying it to your existing sitemap-regeneration workflow is the cleanest pattern.

Do I need to submit llms.txt anywhere?
No. Unlike sitemap.xml (which you can submit to Google Search Console), there's no "submit to AI" button. Crawlers find /llms.txt by convention, same as they find /robots.txt. Put it at the right URL and the right crawlers will fetch it on their next visit.

Can I dynamically generate llms.txt server-side?
Yes, and many large sites do. The file is just text — there's no requirement that it be a static file. A common pattern is to generate it at build time, cache it for an hour, and regenerate on content changes. Just make sure the response is fast (under 200ms) and stable (returns the same content for the same URL within a reasonable window).

What's the difference between llms.txt and AI.txt?
ai.txt was a 2023 proposal from Spawning that focused on consent — telling AI training pipelines whether they could use your content. llms.txt, proposed a year later by Answer.AI, focuses on structure — telling AI models how to use your site at inference time. They're orthogonal; some sites have both. The industry's center of gravity is firmly on llms.txt.

How often should I regenerate llms.txt?
Whenever your site's structure changes meaningfully — new product pages, restructured documentation, retired sections. For active sites, monthly is a reasonable cadence. For mostly-static sites, regenerate when you ship a notable update. Tying it to your existing sitemap-regeneration workflow is the cleanest pattern.

Do I need to submit llms.txt anywhere?
No. Unlike sitemap.xml (which you can submit to Google Search Console), there's no "submit to AI" button. Crawlers find /llms.txt by convention, same as they find /robots.txt. Put it at the right URL and the right crawlers will fetch it on their next visit.

Further reading and generating your llms.txt files
Lab451 produces a spec-compliant llms.txt — plus llms-full.txt, sitemap.xml, and robots.txt — for any public website. Free for sites under 50 pages (llms.txt and llms-plus.txt) files and up 100 sitemap.xml files. No signup, no plugin, no OAuth dance.

Lab451 - Generate your files →