Python script vs Firecrawl for llms.txt in 2026: which I use

If your docs live in a repo, write a small script and run it in CI. Mine took 47 minutes to get right and now costs nothing per deploy. Use Firecrawl's llms.txt generator when you don't control the source, or when you want an llms-full.txt built from a live site. Plan to rewrite a chunk of its auto-written descriptions. For a tiny site, filling in a form by hand is faster than either.

The llms.txt Generator on aidevhub that I mention near the end is one I built. Before that I tried four other generators. Each one either crawled my site through someone else's servers or produced bare links with no descriptions, which throws away half the value of the file. Mine is free and runs entirely in your browser. It needs no signup and uploads nothing. If you have a better one, tell me and I'll point people to it.

The task: one llms.txt for a 38-page docs site

On Tuesday, October 20, 2026, a user of my side project emailed me a screenshot. Their IDE agent had confidently called an endpoint that doesn't exist, /v2/jobs/retry, with a payload shape I've never shipped. The agent had clearly guessed from the URL structure. It hadn't read the docs. Fair enough. My docs don't give an agent any map to read.

That's the whole point of llms.txt. The spec at llmstxt.org is short, and you should read it yourself, but here's the shape:

• The file lives at /llms.txt and is plain Markdown.
• The first thing in it is an H1 with the project name. That's the only required part.
• Next comes an optional blockquote with a short summary, plus optional free-form notes.
• After that come H2 sections, each holding a list of links in the form - [Title](url): optional notes.
• A section titled Optional marks links an agent can skip when it's short on context.

The spec also suggests serving a Markdown version of each page at the same URL with .md appended, so agents can fetch clean text instead of parsing your HTML.

My input: 38 Markdown pages in five folders (getting-started, guides, api, recipes, changelog). The docs build as a static site from the repo.

Expected output, written down before I started so I couldn't move the goalposts:

1. A file that follows the spec structure exactly.
2. Every one of the 38 pages linked, with a one-line description I'd actually sign off on.
3. Changelog pages under ## Optional.
4. Under 4 KB, because agents shouldn't burn context on a table of contents.
5. Regenerated automatically on every deploy, since stale maps are worse than no map.

I ran two approaches against that list: a script I wrote, and Firecrawl's hosted generator.

The Python script: what happened

I tried the lazy version first and it broke in two ways. I used folder names directly as section headings, which gave me ## api in lowercase. The changelog also sorted alphabetically into the middle of the file, right where an agent would waste tokens on it. Second pass fixed both:

#!/usr/bin/env python3
"""Build llms.txt from a folder of Markdown docs with YAML-ish front matter."""
import re
import sys
from pathlib import Path

BASE_URL = "https://docs.example.com"
FRONT_MATTER = re.compile(r"^---\n(.*?)\n---\n", re.S)


def read_meta(path):
    text = path.read_text(encoding="utf-8")
    fields = {}
    match = FRONT_MATTER.match(text)
    if match:
        for line in match.group(1).splitlines():
            if ":" in line:
                key, value = line.split(":", 1)
                fields[key.strip()] = value.strip().strip('"')
    title = fields.get("title") or path.stem.replace("-", " ").capitalize()
    return title, fields.get("description", "")


def build(root, name, summary):
    root = Path(root)
    sections = {}
    missing = []
    for page in sorted(root.rglob("*.md")):
        rel = page.relative_to(root)
        section = rel.parts[0] if len(rel.parts) > 1 else "docs"
        if section == "changelog":
            section = "optional"
        title, desc = read_meta(page)
        if not desc:
            missing.append(str(rel))
        url = f"{BASE_URL}/{rel.with_suffix('').as_posix()}.md"
        line = f"- [{title}]({url})" + (f": {desc}" if desc else "")
        sections.setdefault(section, []).append(line)

    out = [f"# {name}", "", f"> {summary}", ""]
    # "Optional" always goes last, per the spec's intent
    for section in sorted(sections, key=lambda s: (s == "optional", s)):
        out.append(f"## {section.replace('-', ' ').capitalize()}")
        out.extend(sections[section])
        out.append("")

    for rel in missing:
        print(f"warning: no description in {rel}", file=sys.stderr)
    return "\n".join(out).rstrip() + "\n"


if __name__ == "__main__":
    if len(sys.argv) != 4:
        sys.exit("usage: build_llms.py DOCS_DIR PROJECT_NAME SUMMARY")
    sys.stdout.write(build(*sys.argv[1:]))

Run it with python3 build_llms.py docs "Jobrunner" "Hosted job queue with retries and cron." > public/llms.txt.

It finished in 0.31 seconds. The output was 3,412 bytes with all 38 links. The stderr warnings turned out to be the most useful part. Seven pages had no description in their front matter, so those links came out bare. I'd never noticed because my site generator falls back to the first paragraph for meta tags. Writing those seven descriptions by hand took longer than writing the script, and honestly that's the correct place to spend the time.

Two things annoyed me. First, the .md URLs. My static site generator didn't publish raw Markdown, so every link in my shiny new file 404'd. I added a build step that copies docs/**/*.md into the output directory. That's six lines of shell, but I lost about 20 minutes wondering why curl kept returning HTML error pages. Second, the script only knows about files. A docs page that's generated at build time (my API reference comes from an OpenAPI file) doesn't exist in docs/, so I had to add it to the folder as a stub. Small, but it's the kind of thing that rots.

Total time, start to committed CI step: 47 minutes. Every deploy since then has regenerated the file for free.

Firecrawl: what happened

Firecrawl's generator (llmstxt.firecrawl.dev) takes the opposite approach. You give it a public URL. It crawls the live site and uses an LLM to write titles and descriptions. Then it hands you an llms.txt plus an llms-full.txt containing the full text of the pages it found.

I pasted in my docs URL and waited. It took 4 minutes 12 seconds on my run.

The crawl picked up 61 URLs. My docs header links to the blog, the pricing page and the status page, and the crawler followed those. So 23 of the 61 entries weren't docs at all. I deleted them by hand. A crawler can't know which of your pages are documentation and which are marketing, and I don't really hold that against it.

The descriptions were the bigger issue. They were grammatical and mostly accurate, and about half of them were generic enough to be useless. "This page provides information about authentication" tells an agent nothing it didn't already get from the word "Authentication" in the link title. I rewrote 14 of them. A few were better than mine, to be fair. Its line for the retries guide mentioned the exponential backoff cap, which my own front matter description didn't.

My run also came back with sections that didn't match how the docs are organized, so I regrouped them by hand into my five folders. After cleanup the file was 3,890 bytes and roughly equivalent to the script's output. Cleanup took about 35 minutes.

The llms-full.txt was a genuine win, though. It came to 212 KB of clean page text. That's too big for most context windows, but it's handy to drop into a chat when you want an assistant to answer questions about the whole product. My script doesn't produce one, and I'd need to add Markdown stripping and concatenation to match it.

On cost: without an API key the hosted generator limited how much it would process. With a key, runs draw on your Firecrawl credits. Their limits and endpoints have shifted more than once, so check the current pricing page instead of trusting my numbers. The other thing to think about is that your site content goes through their service. For public docs I didn't care. For an internal docs portal I would.

The scoring table

These are the criteria I actually cared about, measured on this one site. Your numbers will differ.

Criterion	Python script	Firecrawl generator
Time to first usable file	47 min once, then 0.31 s per run	4 min 12 s, plus about 35 min cleanup
Needs access to the source repo	Yes	No, only a public URL
Content leaves your machine	No	Yes, crawled by their service
Stray non-docs pages included	0	23 of 61
Description quality	As good as your front matter	Readable, often generic
Produces llms-full.txt	Not without more code	Yes
Regenerate on every deploy	One CI line	Via their API, uses credits
Error visibility	Warns on missing descriptions	Silent, you spot gaps by reading
Cost	Free	Free tier, then paid credits

The error visibility row is the one I didn't expect to matter. The script complained about exactly the pages that needed attention. Firecrawl filled the gaps with plausible text, which looks finished and isn't.

Which one I actually recommend

The script. I don't think it's close for anyone whose docs sit in the same repo as their build.

My reasoning is that llms.txt is a curated map. The curation (which pages matter, which go under Optional, what one sentence an agent needs before clicking) belongs where the docs are written and reviewed. A crawler run after the fact gives you a draft of that curation, and then you're editing a file that drifts the moment someone adds a page. The script ties the map to the source, and CI keeps it from going stale. If you're on a hosted docs platform, check whether it already generates llms.txt for you before writing anything. Some do, and then the question is moot.

Firecrawl wins in one case I hadn't considered until this test: docs you don't own. I've since used it to generate an llms.txt for a library I depend on whose maintainers haven't published one. I keep that file in my project so my agent stops inventing function signatures. No repo access needed, and the llms-full.txt is perfect for that job.

There's a third case where both are overkill. A six-page marketing site, a landing page with a short API section, or a project whose "docs" are a README and two guides. You don't need a crawler or a script there. You need to type a name, a summary and a dozen links into something that formats them correctly. That's what I built the llms.txt Generator for. It assembles a spec-shaped file in the browser from what you enter, and nothing is sent anywhere.

Whatever you pick, open the result and read it like an agent would. Check that the H1 is the first line. Check that every link resolves (curl a few of the .md URLs, because that's where mine broke). Make sure the Optional section is actually last.

One admitted uncertainty. I don't know how much this file gets used yet. In the first week my logs showed 112 requests to /llms.txt, and I couldn't identify the user agent on most of them. The user who sent the screenshot says their agent has stopped inventing endpoints since they pointed it at the file. That's one data point, and I'll take it.

Written with AI assistance and human review. Try the tool at aidevhub.io/llms-txt-generator.