Ever had an AI crawler ignore your robots.txt? You're not alone.

been digging into how to actually control which parts of our site AI models can access, and llms.txt files are becoming essential. Here's a quick guide to generating one properly.

First, what is llms.txt? It's a simple text file you place at your site root that tells AI crawlers (like OpenAI, Anthropic, etc.) which pages they can fetch and how to interpret your content. Unlike robots.txt which only blocks access, llms.txt allows granular permission control.

I built a small Node.js script to generate a clean llms.txt from a sitemap. Here's the core logic:

const fs = require('fs');
const { XMLParser } = require('fast-xml-parser');

async function generateLlmTxt(sitemapUrl) {
  const response = await fetch(sitemapUrl);
  const xml = await response.text();
  const parser = new XMLParser();
  const sitemap = parser.parse(xml);

  let output = '# LLMs.txt - AI Crawler Permissions\n';
  output += '# Generated for mywebsite.com\n\n';
  output += '# Allowed paths\n';

  // Only include non-blocked paths
  const allowedPaths = sitemap.urlset.url
    .filter(entry => !entry.loc.includes('/private'))
    .map(entry => `Allow: ${entry.loc}`)
    .join('\n');

  output += allowedPaths;

  fs.writeFileSync('llms.txt', output);
  console.log('llms.txt generated!');
}

The key fields you want in your llms.txt:

• Allow: Specific URLs or patterns AI can access
• Disallow: Explicitly blocked paths
• MaxDepth: How deep crawlers can traverse
• RateLimit: Max requests per second

But manually editing these files gets tedious fast. That's where tools come in. I've been using the SERPSpur LLM.txt Generator to automate this. You just paste your sitemap URL, configure your rules visually, and it spits out a valid llms.txt with all the proper formatting and edge cases handled.

Here's a pro tip: always include a Crawl-Delay: 5 directive to prevent hammering your server. And for dynamic sites, use Disallow: /*?* to block query parameters unless absolutely necessary.

The coolest feature? You can set different permissions for different bot UAs. Want to let GPT-4 access your blog but block image scraping? Easy.

# In your llms.txt config
User-agent: GPTBot
Allow: /blog/
Disallow: /images/

User-agent: CCBot
Allow: /docs/
Disallow: /

Once generated, drop the file at https://yoursite.com/llms.txt and test with curl -I https://yoursite.com/llms.txt. You'll see it served with the correct Content-Type.

The best part? Most modern AI training pipelines check for this file. It's becoming as standard as robots.txt for serious webmasters. Start simple, iterate based on your analytics, and you'll have fine-grained control over how AI interacts with your content.

Comments

[Victoria]

Have you considered using a middleware library like user-agents to dynamically detect and categorize crawlers? It could reduce the manual upkeep of bot lists.

[Victoria]

Interesting approach using regex to block bots—simple and effective. But have you considered how this might affect legitimate AI tools like accessibility bots? How do you handle those edge cases?

[Burhan]

Interesting approach, but I've found that many AI bots now spoof user agents to bypass these checks. Have you considered rate-limiting or honeypot traps as a fallback for the ones that slip through?