The robots.txt file was the hero of the 2000s. It told Google and Bing where to go. But in 2025, we have a new challenge: LLMs and AI agents.
AI tools like ChatGPT, Claude, and specialized coding agents are constantly trying to understand our websites. If they have to scrape every single HTML page to find information, they waste your bandwidth, "hallucinate" structure, and get lost in the noise of your footer and navigation tags.
That’s why the llms.txt standard (proposed by the folks at Answer.ai) is becoming a must-have.
I decided to build the fastest way to generate one: an Apify Actor that turns your sitemap into a clean, LLM-ready roadmap.
What I built
I created an automated llms.txt Generator. Instead of manually writing your site map for AI, this tool does the heavy lifting:
Sitemap Deep-Dive: It doesn't just read one file; it recursively follows sitemap indexes.
Smart Metadata Extraction: It pulls
tags and to give the AI context for every link.Glob Filtering: You can easily exclude /tags/, /admin/, or legal pages that just clutter the AI's context window.
Polite & Ethical: It respects robots.txt by default and allows you to set concurrency limits so you don't stress your server.
Why you need this ?
If you have a documentation site, a blog, or a product landing page, an llms.txt file at your root (e.g., mysite.com/llms.txt) allows AI agents to:
- Understand your site structure in milliseconds
- Avoid scraping unnecessary pages
- Provide better answers to users asking questions about your content
Performance & Cost
Since I built this using Crawlee and Cheerio (no heavy headless browsers needed), it’s incredibly fast and cheap.
How to use it
- Head over to Apify llms.txt Generator
- Paste your sitemap URL
- Run it
- Get your direct download link and host it on your site!
The project is live on the Apify Store. I'd love to hear what you think!
Check out the tool here: https://apify.com/justa/llms-txt-file-generator
Top comments (0)