DEV Community

Cover image for Audit your robots.txt AI block list: a script that checks what your rules actually do
Calin V.
Calin V.

Posted on • Originally published at wpghost.com

Audit your robots.txt AI block list: a script that checks what your rules actually do

#security #wordpress #webdev #devops

Your robots.txt has an AI block in it. You pasted it at some point, it looked right, and you have not read it since. This post is a script that tells you what it is actually enforcing, because in my experience the answer is usually "not what you think" in one of four specific ways.

The short version of the problem: every major vendor now runs three bots, not one.

Vendor Training Search / retrieval User-triggered fetch
Anthropic ClaudeBot Claude-SearchBot Claude-User
OpenAI GPTBot OAI-SearchBot ChatGPT-User
Perplexity PerplexityBot (same bot indexes) Perplexity-User
Google Google-Extended Googlebot n/a

Blocking the training bot costs you nothing. Blocking the search bot removes you from that vendor's answers. Most pasted block lists hit both, because they match on a vendor prefix instead of on individual tokens.

And the promises differ per vendor, which is the part that decides your implementation:

  • Anthropic states all three of its crawlers honour robots.txt, including the user-triggered Claude-User.
  • OpenAI documents that robots.txt rules may not apply to ChatGPT-User. So a robots.txt entry is a credible control across Anthropic's whole family, and is not sufficient for OpenAI's fetcher. Same three-tier shape, different enforceable surface.

1. Dump what you are actually serving

Not the file on disk. What the origin returns. On WordPress these differ constantly: with no physical robots.txt, core generates a virtual one, so editing the file you think exists is a silent no-op.

BASE="https://example.com"

# what the world sees
curl -s "$BASE/robots.txt" | grep -iE 'user-agent|disallow' | sed 's/^/  /'

# is it physical or virtual? a physical file usually carries an ETag / Last-Modified
curl -sI "$BASE/robots.txt" | grep -iE 'etag|last-modified|x-cache|cf-cache-status'
Enter fullscreen mode Exit fullscreen mode

If your edits are not in that first output, stop here. Nothing else you configured is running either.

2. Flag deprecated tokens

Anthropic retired Claude-Web and anthropic-ai. They are harmless to keep, but their presence dates the file: a rule set naming them was written in 2024 and has not been reread since, which means every other decision in it is also a 2024 answer.

curl -s "$BASE/robots.txt" \
  | grep -iE 'claude-web|anthropic-ai|cohere-ai|omgili' \
  && echo "^^ deprecated tokens present - this file needs a full reread"
Enter fullscreen mode Exit fullscreen mode

3. Test enforcement per bot, in both directions

This is the part that matters, and the assertion nobody writes is the second one: the search bot must still get a 200.

#!/usr/bin/env bash
# ai-crawler-audit.sh - verify per-bot policy against a live origin
BASE="${1:?usage: ai-crawler-audit.sh https://example.com}"

# agent | expected: BLOCK or ALLOW
BOTS=(
  "ClaudeBot|BLOCK"
  "Claude-SearchBot|ALLOW"
  "Claude-User|ALLOW"
  "GPTBot|BLOCK"
  "OAI-SearchBot|ALLOW"
  "ChatGPT-User|ALLOW"
  "PerplexityBot|BLOCK"
  "Google-Extended|BLOCK"
  "Googlebot|ALLOW"
)

fail=0
for entry in "${BOTS[@]}"; do
  ua="${entry%%|*}"; want="${entry##*|}"
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua/1.0" "$BASE/")
  case "$want:$code" in
    BLOCK:40[0-9]|ALLOW:200) status="ok" ;;
    *) status="MISMATCH"; fail=1 ;;
  esac
  printf '%-18s want=%-5s got=%-3s  %s\n' "$ua" "$want" "$code" "$status"
done
exit "$fail"
Enter fullscreen mode Exit fullscreen mode

Edit the BOTS array to match the policy you actually want, then run it in CI or a cron. The point is that the expectations are declared in one place and checked, instead of living in someone's memory.

Two failure modes it catches immediately:

  • ClaudeBot returns 200. Your rule is not running. Usual causes: virtual robots.txt, a CDN answering in front of the origin rule, or a host that ignores .htaccess.
  • Claude-SearchBot returns 403. Your pattern is too broad. A regex of Claude matches all three Anthropic tokens, so you refused training and deleted yourself from Claude's answers in one line. ## 4. Fix the too-broad pattern

Anchor on full tokens, one per bot you actually mean. The clever compact regex is how the over-block happens.

apache (.htaccess):

<IfModule mod_rewrite.c>
  RewriteEngine On
  # training crawlers only - retrieval bots deliberately absent
  RewriteCond %{HTTP_USER_AGENT} (ClaudeBot|GPTBot|PerplexityBot|CCBot) [NC]
  RewriteRule .* - [F,L]
</IfModule>
Enter fullscreen mode Exit fullscreen mode

nginx:

map $http_user_agent $ai_training_bot {
    default              0;
    "~*ClaudeBot"        1;
    "~*GPTBot"           1;
    "~*PerplexityBot"    1;
    "~*CCBot"            1;
}

server {
    if ($ai_training_bot) { return 403; }
}
Enter fullscreen mode Exit fullscreen mode

Note what is not in either list: no Claude-SearchBot, no OAI-SearchBot, no Googlebot. That omission is the policy.

5. Verify the caller is who it claims

The user-agent string is public and forgeable, so treat it as a claim rather than an identity. Anthropic and OpenAI both publish crawler IP ranges, which makes the claim checkable: a request presenting as ClaudeBot from an address outside the published set is provably lying.

# pull the IPs your log attributed to a given bot, then check them against
# the vendor's published prefixes before trusting the label
awk '/ClaudeBot/ {print $1}' /var/log/nginx/access.log | sort -u | head -50
Enter fullscreen mode Exit fullscreen mode

Confirm the current endpoint and JSON schema for each vendor's published ranges before wiring this into automation. Both vendors have changed the format at least once.

Where a plugin earns its place, and where it does not

Everything above is a hand-maintained rule set, and hand-maintained is a real option: a competent sysadmin's nginx map and a plugin-managed rule produce the identical 403. The difference is upkeep. Tokens get added, renamed, and deprecated, syntax differs across Apache and nginx, and a botched .htaccess edit takes the whole site down rather than one rule.

That maintenance gap is the honest case for something like WP Ghost, which keeps a list of 30+ AI and scraper user agents with per-bot toggles and applies them at a server-level firewall ahead of PHP, or for equivalent edge-managed bot rules if you already run everything through a CDN. Pick based on how many sites you maintain and who is responsible when a vendor adds a fourth bot.

What none of them do, including that one: unmask a scraper that lies. A user-agent policy governs the population that identifies itself. A residential proxy pool presenting a Chrome string is a rate-limiting and traffic-shaping problem with a different toolkit, and any tool advertising "blocks bad bots" as one checkbox is collapsing two populations that need different answers. Consent enforcement and extraction defence are separate projects; the audit above is squarely the first one.

Top comments (0)