Make Any Website AI-Readable: Generating llms.txt Files with

#ai #llm #python #seo

AI assistants like ChatGPT, Claude, and Perplexity are increasingly crawling the web for context. But most websites aren't optimised for AI readability — they're built for human browsers with complex HTML, JavaScript navigation, and boilerplate-heavy layouts.

The llms.txt standard is changing this. It's a simple convention: place a llms.txt file at your site root that gives AI systems clean, structured content they can actually understand.

I built a tool that generates these files automatically for any website.

What is llms.txt?

Think of it as robots.txt but for LLMs. Three files form the standard:

llms.txt — A curated summary of your site with key links
llms-full.txt — Complete site content in clean markdown
Per-page data — Structured JSON with extracted content per URL

The Generator

The llms.txt Generator crawls any website using BFS (Breadth-First Search) and:

Respects configurable crawl depth and URL filters
Extracts clean content via trafilatura (not regex — actual text extraction)
Outputs markdown or plaintext
Handles JavaScript-rendered pages
Produces both summary and full-content files

Why This Matters for SEO

Traditional SEO targets Google's crawler. But a new category is emerging: SEO for AI.

When a user asks ChatGPT "what is [your product]?, the AI searches its training data and web results. If your site has a clean llms.txt, the AI gets structured, accurate content instead of parsing your homepage HTML.

Input Parameters

Parameter	Default	Description
`startUrls`	required	Website URLs to crawl
`maxPages`	50	Maximum pages to process
`outputFormat`	markdown	Output format (markdown/plaintext)
`includePatterns`	[]	URL patterns to include
`excludePatterns`	[]	URL patterns to exclude

Example: Documenting a Python Library

I tested it on Pydantic's documentation (docs.pydantic.dev). The crawler:

Started at the root docs page
Followed internal links via BFS
Extracted clean content from each page
Produced a structured dataset with per-page markdown

Result: 2 pages processed, full content extracted with zero boilerplate.

Try It

Live on the Apify Store: llms.txt Generator

Pricing is $0.01 per page processed. Free tier covers ~50 pages.

The llms.txt standard is still emerging, but early adopters will have an advantage as AI-driven search grows. Is your website AI-readable?

DEV Community