Getting a site online is easy now, but getting found takes much longer, and more of that happens inside AI assistants, which can only mention pages they managed to fetch. Whether they can fetch yours is something you can check in about five minutes, and the answer doesn't always match what your robots.txt suggests.
Blocking can happen in two places. robots.txt is a request that well-behaved bots read before fetching anything, while your host, CDN or firewall can refuse a bot outright even when robots.txt says it's welcome. It's easy to fix the first and never look at the second, so these steps look at both.
One thing to know before you start: the crawlers that collect pages for training or indexing (GPTBot, ClaudeBot, PerplexityBot) are separate from the ones that fetch a page when someone asks an assistant a question (ChatGPT-User, Claude-User, Perplexity-User), and each family has its own user agent and its own robots.txt group. The checks below cover both.
1. Read what robots.txt says to each bot
curl -s https://yourdomain.com/robots.txt | grep -i -A2 -E "^user-agent: (gptbot|chatgpt-user|claudebot|claude-user)"
Here is the real output for our own site, run on 5 November 2026 at about 11:23 UTC:
User-Agent: GPTBot
Allow: /
Disallow: /app$
--
User-Agent: ChatGPT-User
Allow: /
Disallow: /app$
--
User-Agent: ClaudeBot
Allow: /
Disallow: /app$
--
User-Agent: Claude-User
Allow: /
Disallow: /app$
Each bot has its own group, and each group allows the whole site and disallows only the exact URL /app (the $ anchors the match to the end of the path, so /app/anything is covered by a separate Disallow: /app/ line). If a bot you care about has no group of its own, it falls back to the User-agent: * group, so read that one too.
Google-Extended is a robots.txt token only. It controls whether Google uses your content for Gemini, it doesn't change Search or AI Overviews, and it never appears as a user agent in your logs, so there is nothing to test with curl.
2. Ask the server what it answers to each bot
Most robots.txt checks stop before this step, which is where the second kind of blocking shows up. Save the script below as ua-check.sh. It requests your homepage once as a normal browser and once with each bot's user agent, follows redirects, and prints only the status codes.
#!/usr/bin/env bash
# usage: bash ua-check.sh https://example.com/
SITE="${1:-https://example.com/}"
check() { # name, user agent
printf '%-16s %s\n' "$1" "$(curl -sL --max-time 15 -o /dev/null -w '%{http_code}' -A "$2" "$SITE")"
}
check "browser" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36"
check "GPTBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)"
check "OAI-SearchBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)"
check "ChatGPT-User" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)"
check "ClaudeBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
check "Claude-User" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)"
check "PerplexityBot" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
check "Perplexity-User" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"
Run against our site just after step 1:
$ bash ua-check.sh https://growthradar.app/
browser 200
GPTBot 200
OAI-SearchBot 200
ChatGPT-User 200
ClaudeBot 200
Claude-User 200
PerplexityBot 200
Perplexity-User 200
You want every bot line to match the browser line. If a bot gets a 403 or 429 where the browser gets a 200, something in front of your app is refusing it, and the same goes for a 200 whose body is a "checking your browser" page. To catch that hidden challenge, send the same full user-agent string from the script and read the first few hundred bytes of the body. If the browser line fails too, the result is inconclusive, because your server may be rejecting curl itself. The user-agent strings above are the published ones at the time of writing and operators do change them, so confirm against each operator's docs if a result looks odd.
3. Check llms.txt if you publish one
curl -sL -o /dev/null -w "%{http_code}\n" https://yourdomain.com/llms.txt
Ours returned 200. The file is optional, the format is described at llmstxt.org, and no major AI provider has said it relies on it, so treat it as housekeeping. If you do publish one, it should load as plain text with a 200 and not redirect to your homepage.
What this can't tell you
A curl with a crawler's user agent sent from your own connection is not what the real crawler experiences. Many firewalls and CDNs also check the source IP against the ranges the bot's operator publishes, so a request that only claims to be GPTBot from a home or office address can be treated differently from the genuine one, in either direction. A clean result means this URL wasn't blocked by user agent at the moment you ran it, but it doesn't prove the real bot gets through, and it says nothing about rate limits or rules on other paths. Your own access logs, filtered to the operators' published IP ranges, are the closer evidence.
Which bots does your own server answer differently from a browser? If you find one, we'd like to hear what was blocking it.
Disclosure: AI helped draft this article. We ran every command ourselves and pasted the output as it came back.
Top comments (2)
The caveat about source IPs is the part most checklists skip, so I am glad you spelled it out. Two small additions for the curl step: print the response size next to the status code, since a 200 that is far smaller than the browser response is usually a challenge page, and dump the X-Robots-Tag header per user agent. I have seen CDN rules send noindex or noai headers to bot user agents only, which a robots.txt read will never show.
The user wants a comment to post under this video. Must follow developer instructions: casual YouTube comment style, short, 1-2 sentences, possibly fragment. Must start with a specific reaction or question about THIS video. No generic praise. Should be about checking AI crawlers reading site. Could ask about tools or mention something. Must comply with style rules: no quotes, no labels, no hashtags, no markdown. No double hyphens, no em-dash, no curly quotes. Use straight quotes only if needed.