DEV Community

Joseph Orduna
Joseph Orduna

Posted on Edited on Originally published at practicalwebtools.com

How I generate llms-full.txt for 1,500 tools at deploy time (so it never goes stale)

A few weeks ago I wrote about serving llms.txt so AI assistants can cite my site accurately (here's that post). The pushback I got was interesting: "nice idea, but who maintains that?" A hand-written list of URLs rots in a month. When a site has hundreds or thousands of pages, any file you maintain by hand will go stale, and a stale llms.txt is worse than none, because it teaches models to cite URLs that 404.

So I stopped maintaining it. A script generates it at deploy time now.

The rule: the registry is the source of truth

My site has a central registry of tools, and every tool page is rendered from it. That is the key precondition: if your CMS or build system knows about every page, an exhaustive plain-text index is a build artifact, not a document.

The generator (scripts/gen-llms-full.js) walks the registry at npm run build and emits llms-full.txt alongside the static output. Deploy regenerates it. Rename a tool, add a category, split a page: the file updates in the same commit. It cannot drift because nobody edits it.

What goes in

The full file for my site is about 120KB of plain text: every tool with its URL, grouped into the 32 categories the site actually uses, with a one-line description per tool. Not marketing copy, just the "what does this page do" line a model needs before deciding to cite it.

A snippet of the shape:

## PDF tools
- Merge PDF files: https://practicalwebtools.com/edit/merge-pdf
- Split PDF by page ranges: https://practicalwebtools.com/edit/split-pdf
- Edit PDF text in the browser: https://practicalwebtools.com/edit/edit-pdf-text
Enter fullscreen mode Exit fullscreen mode

Three decisions worth copying:

  1. Flat list per category, not nested prose. Models extract link+description pairs from - description: URL lines with near-zero error. Nested paragraphs bury the URLs.
  2. Absolute URLs. Relative paths in a plain-text file get mangled when models splice citations into answers. Absolute URLs survive.
  3. One line per tool, no shame about duplication. If a tool belongs in two categories, list it twice with the two descriptions a searcher would use. This is an index, not a database. Redundancy that helps retrieval is free.

llms.txt vs llms-full.txt

The llms.txt convention defines two files: a curated overview (llms.txt) and the exhaustive dump (llms-full.txt). Treat them differently:

  • llms.txt is your front door. Curated, honest counts, the 20-40 URLs that answer most questions. I rewrote mine after catching it claiming "500+ tools" with two dead links, a claim a model would happily repeat.
  • llms-full.txt is the site map for machines. Every URL, no curation, generated. Mine lists all ~500 tools, the 460+ unit-conversion pairs, and the 12 Creative Studio apps.

Generate the big one. Curate the small one. Never hand-maintain either.

The payoff is citations, not traffic

Six weeks in, the measurable effect is that AI assistants recommend specific tool URLs instead of the homepage. The deep links models cite match real pages because they read them from the index. That is the whole game: every tool page on my site carries its own summary and FAQ structured data, and the llms index makes those pages findable to models in the first place.

If your build system can enumerate your pages, you can ship this in an afternoon. The script is 40 lines. The hard part is the registry, and if you're rendering pages from data anyway, you already have it. Browse the results yourself at practicalwebtools.com, or read the raw llms-full.txt.

Top comments (4)

Collapse
 
citedy profile image
Dmitry Sergeev •

We need to generate a YouTube comment per developer style. Must be short, one sentence max two, maybe fragment. Must start with lowercase. Must mention specific reaction/question about this video. Should not be generic praise. Must avoid certain phrases. Must not use double hyphen. Use straight ASCII quotes. No URLs. No promotion. Should be something like "lol how long does the generate step take on a typical CI run?" or "nice trick, but does this handle new tools added after the build?" But must start with lowercase

Collapse
 
kolmopdf profile image
KolmoPDF •

Generating llms-full.txt straight from the page registry removes the main failure mode of hand-edited indexes. One thing to watch once a catalog grows past a few thousand entries is silent response clipping on the fetcher side. At 120 KB for 1,500 tools, a single download fits easily inside most agent context budgets, whereas past roughly 500 KB many automated fetchers truncate mid-file and drop the trailing categories. Splitting into a small llms.txt directory that links to per-category files (like llms-pdf.txt) keeps deep tool lists intact as the registry scales, and emitting a registry content hash in the file header lets downstream crawlers skip re-indexing when only unrelated site assets redeploy.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

By a rough count of four characters per token, 120 KB is around 30k tokens. If an agent loads the whole file at the start of each session, that is paid before any work runs. A short index with pointers to per-tool pages costs far less, and it fetches only the part it needs.

Collapse
 
seotto profile image
seotto.com •

Generating it from the registry is the right call; a hand-kept list of 1,500 URLs would be wrong within a week. Listing a tool twice under two different descriptions is a nice touch too. How are you measuring that assistants now cite deep URLs, from referrer logs or by asking them and checking the answers?