How to check whether AI crawlers can read your site (robots.txt and server logs)

Peter Panda — Thu, 22 Oct 2026 20:56:36 +0000

AI answer engines only recommend or cite what they can fetch. Before you invest in GEO, confirm the crawlers can actually read your site. Here is a quick check.

The user agents to know

GPTBot – OpenAI's crawler, used to collect content that may be used to train models.
ClaudeBot – Anthropic's crawler.
PerplexityBot – indexes pages so Perplexity can cite them in answers.
Google-Extended – a robots.txt token (not a separate bot) that controls whether Google may use your content for Gemini and AI features.

Note that OAI-SearchBot and ChatGPT-User also exist for search and live browsing, so check them if you want ChatGPT to cite you.

Step 1: read your robots.txt

Open https://yourdomain.com/robots.txt. A blanket Disallow: / under User-agent: *, or a rule copied from a security plugin, often blocks AI bots by accident. To explicitly allow them:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

If you prefer to keep some areas private, add Disallow: /private/ inside the relevant block.

Step 2: check the server access logs

robots.txt is a request, not enforcement, and CDNs or firewalls (Cloudflare bot rules, WAFs) can block bots before they ever see it. Your access log shows what really happens:

grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | tail -n 20

Count hits per bot and status code:

grep -Ei "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c

Look at the status codes. 200 means the bot read the page. 403 or 429 means your firewall or rate limiter is rejecting it. No hits at all over a couple of weeks may mean it is blocked upstream, or simply that your site is not yet discovered. Note that Google-Extended will not appear in logs, since Googlebot does the fetching.

Step 3: fix and re-test

Whitelist the bots in your CDN or WAF, keep pages server-rendered (many AI crawlers do not run JavaScript), and make sure your sitemap lists the key pages. Then re-check the logs after a week.

We wrote a longer glossary of overseas-ad and AI-search terms here: https://www.pandawm.com/blog/overseas-advertising-glossary.html

DEV Community: Peter Panda

How to check whether AI crawlers can read your site (robots.txt and server logs)

The user agents to know

Step 1: read your robots.txt

Step 2: check the server access logs

Step 3: fix and re-test