Google reads robots.txt. AI assistants increasingly read llms.txt - a simple markdown file at the root of your site that tells language models what is worth reading, in the format they prefer.
If you are running a docs site, a blog, or a product, here is the short version of why it matters: AI tools and agents increasingly crawl the web looking for content to quote and cite. A good llms.txt gives them a curated, structured summary instead of forcing them to guess from navigation menus and sitemap XML. The spec lives at llmstxt.org, and early adopters report their content appearing more accurately in AI-generated answers.
What a good llms.txt contains
- One-line description of the site
- Links to your key pages with short markdown descriptions
- Optional: the full content of important pages inline
Generating one in 30 seconds (free, no signup)
- Go to https://rag-scrape-api.owerryking.workers.dev/llms-txt-generator
- Enter your site URL
- Copy or download the generated llms.txt and host it at yoursite.com/llms.txt
The generator crawls your site (up to 20 pages), pulls the titles and descriptions, and formats everything per the llmstxt.org spec. No account needed.
Prefer automation?
There is an API route for that:
POST https://rag-scrape-api.owerryking.workers.dev/llms-txt
{ "url": "https://yoursite.com", "maxPages": 20 }
It returns ready-to-serve llms.txt content. Wire it to a cron job and your file stays current as your site changes.
Three tips
- Run the generator after your sitemap exists - crawlers follow sitemaps and internal links
- Heavily JavaScript-rendered sites may need the crawl to start from a content-rich entry page
- Keep it under ~100 links; it is a menu for a model, not a second sitemap
There is also a full free URL-to-Markdown converter at /convert if you want to see exactly what an AI sees when it reads your pages.
Free tier (50 requests/month, no card): https://rag-scrape-api.owerryking.workers.dev
Top comments (0)