DEV Community

Cover image for Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It)
acetrondi
acetrondi Subscriber

Posted on

Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It)

#seo #ai #webdev #nextjs

Linux / macOS

Status Code 

curl -sI -A "GPTBot" https://yoursite.com/ | head -1
Enter fullscreen mode Exit fullscreen mode

Windows PowerShell

Status Code 

(Invoke-WebRequest -Uri "https://yoursite.com/" -UserAgent "GPTBot").StatusCode
Enter fullscreen mode Exit fullscreen mode

Windows Command Prompt (CMD)

Full Headers 

curl -I -A "GPTBot" https://yoursite.com/
Enter fullscreen mode Exit fullscreen mode

Note: Replace yoursite.com with your domain

If that returns anything other than 200, AI engines cannot read your site and your robots.txt may look perfectly permissive while it happens. Audits of several thousand sites found roughly 27% block at least one major AI crawler, most of them unintentionally, at the CDN or WAF layer rather than in robots.txt.

Why this happens

Bot-protection rules ship with sensible-sounding defaults: block unknown user agents, rate-limit non-browser traffic, challenge anything without JavaScript. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot all trip at least one of those on common configurations. Your marketing team sees a clean robots.txt; your edge quietly returns 403.

The five-minute audit

for UA in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot PerplexityBot Bingbot; do

printf "%-16s %s\n" "$UA" "$(curl -sI -A "$UA" https://yoursite.com/ | head -1)"

done

Then check your edge logs for 403s grouped by user agent that's where the truth lives.

The second failure: client-side rendering
Several AI crawlers don't execute JavaScript. If your H1 and opening paragraph arrive via hydration, they see an empty shell. Server-render anything you want quoted view-source: is the test, not DevTools.

Allowing crawlers explicitly (Next.js App Router) 

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {

  const disallow = ['/admin', '/api/']

  return {

    rules: [

      { userAgent: '*', allow: '/', disallow },

      { userAgent: 'GPTBot', allow: '/', disallow },

      { userAgent: 'OAI-SearchBot', allow: '/', disallow },

      { userAgent: 'ClaudeBot', allow: '/', disallow },

      { userAgent: 'PerplexityBot', allow: '/', disallow },

      { userAgent: 'Google-Extended', allow: '/', disallow },

    ],

    sitemap: 'https://yoursite.com/sitemap.xml',

  }

}
Enter fullscreen mode Exit fullscreen mode

robots.txt is necessary but not sufficient the CDN rule is the one that actually bites.

Does being crawlable matter?

It's the precondition, not the strategy. Being retrieved is won with ordinary search ranking; being cited is won with extractable structure direct answers, explicit definitions, sourced statistics. But none of that runs if the crawler gets a 403 at the door.

I'm publishing a live experiment on this: 20 fixed questions asked to ChatGPT and Gemini daily, with raw answers, cited-source leaderboards and the retrieved-but-not-cited gap published openly connectingdots.live/experiment

Top comments (1)

Collapse
 
acetrondi profile image
acetrondi •

comments for any suggestion and best practices