Generating llms.txt from a CMS in Next.js — and whether it's worth shipping

#seo #nextjs #ai #webdev

A static llms.txt goes stale the first time someone publishes a page. If the site runs on a CMS, the file should come from the same content as the pages — otherwise you are shipping a map that disagrees with the territory.

Here is the whole thing in the Next.js App Router. One route handler, no dependencies.

Generating llms.txt in a Next.js route handler

// app/llms.txt/route.ts
import groq from "groq";
import { client } from "@/sanity/sanity.client";

const SITE = "https://example.com";

export const revalidate = 86400;

export async function GET() {
  const pages = await client.fetch(groq`
    *[_type == "singlepage" && language == "en" && defined(slug.en.current)]{
      "slug": slug.en.current,
      "title": coalesce(seo.metaTitle, title),
      "description": coalesce(seo.metaDescription, excerpt)
    }`);

  const body = [
    "# Example site",
    "",
    "> One sentence about what this site is.",
    "",
    "## Services",
    "",
    ...pages.map(
      (p: { slug: string; title: string; description?: string }) =>
        `- [${p.title}](${SITE}/${p.slug})` +
        (p.description ? `: ${p.description}` : ""),
    ),
  ].join("\n");

  return new Response(body, {
    headers: { "content-type": "text/plain; charset=utf-8" },
  });
}

That is Sanity in the example, but nothing above is Sanity-specific: swap the fetch for Contentful, Payload, Strapi or a filesystem read and the shape holds.

Three things that bite when you generate the file

A path with a dot skips middleware. In a localised app, /llms.txt does not get a locale prefix and never reaches your [lang] segment — it hits the route directly. Convenient here, and the same rule is why /llms.txt needs no special matcher exclusion in most i18n setups. Worth knowing before you debug a 404 that is actually a redirect.

Use the same cache lifetime as the pages. If the site revalidates daily and the file revalidates hourly, you will serve a map that lists pages the cache does not have yet, or misses ones it does. Mine shares one constant with the page segments.

Do not generate a markdown copy of every page. A popular variant ships page.md next to every route and links them all from the index. If those copies are indexable you have just duplicated the entire site, competing with itself, for the benefit of nobody. The file is a curated index, not a mirror.

The llms.txt format, with an example

An H1 with the site name, an optional blockquote summary, then H2 sections of links. Each entry may carry a description after a colon:

# Acme Analytics

> Retail analytics consulting: assortment, pricing and reporting.

## Services

- [Assortment planning](https://example.com/services/assortment): how we build and review a range

## Optional

- [Press kit](https://example.com/press)

## Optional has a defined meaning in the spec — material that can be skipped when the reader is short of context. Everything else counts as core.

The domains above are placeholders. A real file, generated by the route at the top of this post, is at bandziuk.com/llms.txt — worth a look next to the skeleton, because the gap between a valid file and a useful one is entirely in the descriptions.

Does llms.txt actually do anything? What the measurements say

I would rather you shipped this knowing what it does, because most of what is written about llms.txt is wrong in a specific direction.

Google's Search Central documentation says Google Search does not use llms.txt — not for ranking, not for AI Overviews. Independent measurement agrees: across hundreds of thousands of domains, 97% of llms.txt files were never fetched by anything, and one monitoring exercise covering more than 500 million AI bot visits over ninety days recorded 408 requests targeting the file. Adoption sits just above 10% of surveyed domains, eighteen months in.

So if you are adding it to get cited more often by assistants, the evidence does not support that.

What it does support: agentic retrieval. Anthropic recommends the file in its guidance for writing for agents, OpenAI maintains one for its Agents SDK, Perplexity says it uses the file to prioritise what to read, and Chrome's Lighthouse moved its llms.txt audit into a default category in May 2026. When a coding assistant works against someone's documentation, fetching a table of contents beats scraping the site.

Which explains the adoption pattern exactly: the sites shipping it are documentation-heavy — developer platforms, API companies, infrastructure. If that is your content, ship it. If you run a marketing site and hoped it would move AI visibility, spend the hour elsewhere.

An llms.txt generator, if you would rather not write the route

I built a free llms.txt generator that reads a sitemap, groups the pages into sections and writes a valid file you can edit and download. It reads the sitemap only — it does not crawl your pages — so the titles it produces come from the URLs and need a pass by hand before you publish.

The long version, with the measurements, the robots.txt comparison and the mistakes worth avoiding, is here.

I'm Aliaksandr Bandziuk — I build multilingual sites on Next.js and headless CMS and work on search and AI visibility: bandziuk.com.

DEV Community