Why your robots.txt is blocking AI crawlers (and how to fix it)

#ai #webdev #seo #tutorial

A User-agent: * rule in robots.txt blocks every bot that doesn't have its own entry — including the AI crawlers that power ChatGPT Search, Perplexity, and Google AI Overviews.

The bots that feed live AI answers have their own user-agent strings: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Perplexity-User, and Google-Extended. If your robots.txt has a wildcard Disallow: without explicit rules for these seven, they inherit the restriction — and your content won't appear in AI citations.

The fix

Add explicit Allow: / entries before your wildcard group:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /private/

Specific rules always win over the wildcard (RFC 9309), so Googlebot and Bingbot are unaffected.

Diagnose your robots.txt in one call

curl -X POST https://citeready-api.sprytools.com/v1/audit \
  -H "content-type: application/json" \
  -d '{"url":"https://yoursite.com"}'

The AI Crawler Access category reports a pass, warn, or fail for each bot individually — including which user-agents are blocked and the exact fix to apply.

const res = await fetch('https://citeready-api.sprytools.com/v1/audit', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ url: 'https://yoursite.com' }),
});
const { categories } = await res.json();
const aiAccess = categories.find(c => c.id === 'ai_access');
console.log(aiAccess.score, aiAccess.checks);