Mayank Sharma

Posted on Oct 12

llms.txt: what it is, how to write one, and whether it actually does anything

#seo #nextjs #webdev #ai

Every few months a new file shows up that promises to make your site "AI-ready". The current one is llms.txt. We publish one on our own site, and I've read the arguments on both sides, so here is what I'd tell a developer who just got a ticket that says "add llms.txt".

Short version: it's cheap to add and harmless, it's useful for coding assistants and docs, and there's no public evidence yet that it changes whether ChatGPT or Perplexity cite you. Ship it if it takes ten minutes; don't sell it to your boss as an SEO win.

What llms.txt is

llms.txt is a proposal published at llmstxt.org in September 2024 by Jeremy Howard of Answer.AI. The idea: put a Markdown file at the root of your site (https://example.com/llms.txt) that gives a language model a short, curated map of your most useful pages.

Why not just use sitemap.xml? A sitemap lists every URL for a crawler. llms.txt is meant to be read by a model at inference time, with limited context, so it's short, human-written and annotated. Think "README for your website", not "index of everything".

The format

The spec is small. In order:

An H1 with the name of the site or project. This is the only required part.
A blockquote with a one-paragraph summary.
Optional free text with anything a model should know.
H2 sections, each with a Markdown list of links: - [Title](url): short note.
An optional section literally called ## Optional for links that can be skipped when context is tight.

Here's a realistic example for a small SaaS:

# Acme Invoices

> Acme Invoices is invoicing software for freelancers in the EU. It creates
> VAT-compliant invoices, tracks payments and exports to accounting tools.

Prices are in EUR. There is a free plan for up to 5 invoices a month.

## Product

- [Features](https://acme.example/features): What the app does, with screenshots
- [Pricing](https://acme.example/pricing): Plans, limits and what's in the free tier

## Docs

- [Getting started](https://acme.example/docs/start): Create your first invoice
- [API reference](https://acme.example/docs/api): REST endpoints and auth

## Optional

- [Changelog](https://acme.example/changelog): Release notes

A few things that make the difference between a useful file and noise:

Write the notes yourself. "Plans, limits and what's in the free tier" helps a model choose a link. A copied meta description usually doesn't.
Leave things out. Thank-you pages, login, tag archives and thin pages don't belong there. Short is the point.
Keep it current. A link to a page that 404s is worse than no link.

The proposal also suggests offering clean Markdown versions of pages (for example /docs/start.md) and an llms-full.txt with the full text inlined. Those are optional, and most sites skip them.

Shipping it

As a static file (any stack)

Put llms.txt wherever your framework serves static files from the root: public/ in Next.js, Vite and Create React App, static/ in SvelteKit and Hugo, the web root on a plain server. Check it returns 200 with a text content type:

curl -sI https://your-site.com/llms.txt | head -n 5

Generated in Next.js (App Router)

If your pages come from a CMS, generate the file so it never drifts out of date:

// app/llms.txt/route.ts
import { getPublishedPages } from "@/lib/cms";

export const revalidate = 86400; // rebuild daily

export async function GET() {
  const pages = await getPublishedPages(); // [{ title, url, summary, section }]
  const sections = new Map<string, string[]>();

  for (const p of pages) {
    const line = `- [${p.title}](${p.url}): ${p.summary}`;
    sections.set(p.section, [...(sections.get(p.section) ?? []), line]);
  }

  const body = [
    "# Acme Invoices",
    "",
    "> Invoicing software for EU freelancers: VAT-compliant invoices, payment tracking, accounting exports.",
    "",
    ...[...sections].flatMap(([name, lines]) => [`## ${name}`, "", ...lines, ""]),
  ].join("\n");

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

If you'd rather not write one by hand, I work on a free llms.txt generator at Presence Scout. It drafts the file from your sitemap and existing page titles, with no AI-written text, so you get a starting point to edit rather than a finished file. The notes will only be as good as your titles and descriptions, which is honestly true of any generator.

Does it actually do anything?

This is where I'd be careful.

Where it clearly helps: developer tools. Coding assistants and agents work better when you hand them a compact map of a library's docs, and a lot of documentation sites now publish llms.txt for exactly that reason. If you maintain an API or an SDK, this alone justifies the file.

Where the evidence is thin: AI search. I haven't seen any of the major assistants document that their crawlers fetch llms.txt and use it to decide what to cite. So treat claims like "llms.txt gets you into ChatGPT answers" as unproven.

What matters more for AI search: whether the crawlers can read your pages at all. If robots.txt, your CDN's bot rules or a JavaScript-only render keep them out, no index file will fix it. Check that first; a five-line curl with the crawler's user agent will tell you more than any new file.

My checklist

[ ] Crawlers can fetch your key pages (status 200, real HTML, not a JS shell)
[ ] llms.txt at the root, under ~50 links, with hand-written notes
[ ] No links to private, thin or redirecting pages
[ ] Generated from your CMS, or on a reminder to review quarterly

It's a small file. Treat it like one: quick to ship, easy to maintain, and not a strategy on its own.

I'm Mayank, Digital Marketing Manager at Presence Scout. We build tools that show where a brand appears in AI answers and Google results.

DEV Community