DEV Community: 500wango

llms.txt best practices: facts, canonical URLs, what to leave out

500wango — Sun, 04 Oct 2026 14:36:54 +0000

With AI search engines (Perplexity, ChatGPT Search, Claude) shifting how content is discovered, the proposed llms.txt standard (from llmstxt.org) has gained massive attention.

Think of llms.txt as a Markdown-based parallel to robots.txt and sitemap.xml. Instead of telling web spiders where to crawl, it serves clean, token-efficient, machine-readable facts and
canonical documentation directly to LLMs.

However, after inspecting hundreds of newly deployed llms.txt files, we noticed developers making common mistakes that actually harm how models interpret their sites.

Here are the practical best practices for structuring your discovery file.

### 1. Where and How to Serve It

• Path: Must be served at the root of your domain: https://yourdomain.com/llms.txt

• MIME type: text/plain; charset=utf-8 (Do not serve it as text/html or application/octet-stream).

• HTTP status: Must return 200 OK without requiring authentication, cookies, or redirect chains.

### 2. Standard Syntax Anatomy

An effective llms.txt follows a simple, hierarchical Markdown structure:

Your Brand / Project Name

│ A concise 1-2 sentence description of what the project actually does. Avoid buzzwords and marketing fluff.

## Core Documentation

• Quickstart Guide https://yourdomain.com/docs/quickstart: Step-by-step setup in under 5 minutes.

• API Reference https://yourdomain.com/docs/api: REST API endpoints and authentication rules.

• Pricing https://yourdomain.com/pricing: Official tiers, free limits, and billing terms.

## Key Facts

• Founded: 2024

• Pricing Model: Freemium ($0 / $29 / $99 per month)
• Supported Frameworks: Node.js, Python, Go

### 3. What NOT to Include (The Top 3 Pitfalls)

Stale Pricing or Claims: Never hardcode pricing numbers into llms.txt that you don't update religiously. If your live page says $49/mo but your llms.txt says $29/mo, AI engines flag this contradiction as a trust failure. Always link to your live pricing page.
Marketing Slogans: Phrases like "The world's leading revolutionary AI platform" take up valuable context window tokens and offer zero factual grounding. Stick to strict technical capabilities and verifiable facts.
Private or Gated Routes: Do not list staging URLs, internal admin panels, or authenticated dashboard links. Crawlers will only fetch HTTP 200 public resources.

### 4. Optional: Full Context via llms-full.txt

If your project has comprehensive API docs or whitepapers that fit within a ~100k token context, you can also publish a companion file at /llms-full.txt containing the full concatenated
text. In your main llms.txt, link to it at the bottom:

• Full Documentation /llms-full.txt: Complete combined documentation for offline context.

### Want to Generate or Validate Your Draft?

If you want to quickly scaffold a clean llms.txt file from your homepage metadata or validate your existing syntax:

👉 Free llms.txt Generator & Validator https://citeaura.com/llms-txt-tool

Are you already experimenting with llms.txt on your projects? Do you see genuine crawler hits on it in your server logs yet?

Is GPTBot Silently Blocked by Your Cloudflare WAF? How to Test and Fix AI Crawler Access

500wango — Sun, 04 Oct 2026 14:08:12 +0000

You launch a site, set up your robots.txt to welcome AI search bots, and move on:

User-agent: GPTBot                                                                                                                                                                     
Allow: /

A few weeks later, you check your server access logs or wonder why your content never gets cited in ChatGPT Search or Perplexity. You find zero crawler hits.

The culprit is almost never your robots.txt. It is usually an edge security rule or web application firewall (WAF) blocking the crawler before the request ever touches your origin server.

Here is what is happening under the hood and how to test it.

──────

The Edge Challenge Problem

Most modern web architectures sit behind Cloudflare, Fastly, or AWS CloudFront.

When you enable features like Cloudflare Super Bot Fight Mode or aggressive rate-limiting:

• The edge evaluates the incoming HTTP request.

• OpenAI, Anthropic, and Perplexity use dynamic IP subnets that change frequently.

• If the WAF cannot instantly verify the crawler via reverse DNS or trusted ASN matching, it serves an HTTP 403 Forbidden or a Cloudflare JavaScript Challenge (Turnstile).

Because an automated crawler cannot solve an interactive browser challenge, it simply fails and drops the page from its index.

How to Test Your Live Site with cURL

You do not need to wait weeks to know if you are affected. Run this command from your terminal:

curl -I -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yourdomain.com

Look at the HTTP status header:

• HTTP/2 200 OK: You are in the clear. The crawler can read your HTML.

• HTTP/2 403 Forbidden: Your WAF or hosting provider is actively blocking OpenAI.

• cf-mitigated: challenge: Cloudflare is intercepting the crawler with an interactive bot challenge.

──────

How to Allow AI Crawlers Safely

If you find that GPTBot is being blocked, do not turn off your entire WAF. Instead, create a targeted bypass rule in Cloudflare:

Go to Security -> WAF -> Custom Rules.
Create a rule named Allow Verified AI Crawlers.
Set the condition: • (cf.client.bot and http.user_agent contains "GPTBot")
Set Action to Skip: • Check All remaining custom rules • Check Super Bot Fight Mode (or Bot Management)

This ensures legitimate OpenAI search crawlers can fetch public content while your protected admin and API endpoints stay secure.

Need an Instant Sanity Check?

If you don't have terminal access or want to check multiple bots (GPTBot, ClaudeBot, PerplexityBot, ByteSpider) in 5 seconds, we built a lightweight, no-signup checker that runs these

cURL validations for you:

Free AI Crawler Access Checker https://citeaura.com/crawler-check

How are you handling AI search crawlers in your current infra? Are you whitelisting them or keeping them blocked by default?

Is GPTBot Silently Blocked by Your Cloudflare WAF? How to Test and Fix AI Crawler Access

500wango — Sun, 04 Oct 2026 13:58:18 +0000

You launch a site, set up your `robots.txt` to welcome AI search bots, and move on:


txt                                                                                                                                                                                 
    User-agent: GPTBot                                                                                                                                                                     
    Allow: /                                                                                                                                                                               

  A few weeks later, you check your server access logs or wonder why your content never gets cited in ChatGPT Search or Perplexity. You find zero crawler hits.                            

  The culprit is almost never your robots.txt. It is usually an edge security rule or web application firewall (WAF) blocking the crawler before the request ever touches your origin      
  server.                                                                                                                                                                                  

  Here is what is happening under the hood and how to test it.                                                                                                                             
  ──────                                                                                                                                                                                   
  ### 1. The Edge Challenge Problem                                                                                                                                                        

  Most modern web architectures sit behind Cloudflare, Fastly, or AWS CloudFront.                                                                                                          

  When you enable features like Cloudflare Super Bot Fight Mode or aggressive rate-limiting:                                                                                               

  • The edge evaluates the incoming HTTP request.                                                                                                                                          
  • OpenAI, Anthropic, and Perplexity use dynamic IP subnets that change frequently.                                                                                                       
  • If the WAF cannot instantly verify the crawler via reverse DNS or trusted ASN matching, it serves an HTTP 403 Forbidden or a Cloudflare JavaScript Challenge (Turnstile).              

  Because an automated crawler cannot solve an interactive browser challenge, it simply fails and drops the page from its index.                                                           
  ──────                                                                                                                                                                                   
  ### 2. How to Test Your Live Site with cURL                                                                                                                                              

  You do not need to wait weeks to know if you are affected. Run this command from your terminal:                                                                                          

    curl -I -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yourdomain.com                                          

  Look at the HTTP status header:                                                                                                                                                          

  • HTTP/2 200 OK: You are in the clear. The crawler can read your HTML.                                                                                                                   
  • HTTP/2 403 Forbidden: Your WAF or hosting provider is actively blocking OpenAI.                                                                                                        
  • cf-mitigated: challenge: Cloudflare is intercepting the crawler with an interactive bot challenge.                                                                                     
  ──────                                                                                                                                                                                   
  ### 3. How to Allow AI Crawlers Safely                                                                                                                                                   

  If you find that GPTBot is being blocked, do not turn off your entire WAF. Instead, create a targeted bypass rule in Cloudflare:                                                         

  1. Go to Security -> WAF -> Custom Rules.                                                                                                                                                
  2. Create a rule named Allow Verified AI Crawlers.                                                                                                                                       
  3. Set the condition:
      • (cf.client.bot and http.user_agent contains "GPTBot")
  4. Set Action to Skip:
      • Check All remaining custom rules
      • Check Super Bot Fight Mode (or Bot Management)


  This ensures legitimate OpenAI search crawlers can fetch public content while your protected admin and API endpoints stay secure.
  ──────
  ### 4. Need an Instant Sanity Check?

  If you don't have terminal access or want to check multiple bots (GPTBot, ClaudeBot, PerplexityBot, ByteSpider) in 5 seconds, we built a lightweight, no-signup checker that runs these  
  cURL validations for you:

  👉 Free AI Crawler Access Checker https://citeaura.com/crawler-check

  How are you handling AI search crawlers in your current infra? Are you whitelisting them or keeping them blocked by default?