DEV Community

Cover image for ClaudeBot, Claude-SearchBot, Claude-User: Which One to Block (and What Blocking Costs You)
Furqan Khalid
Furqan Khalid

Posted on

ClaudeBot, Claude-SearchBot, Claude-User: Which One to Block (and What Blocking Costs You)

#claude #ai #seo #webdev

A lot of sites "blocked Claude" in 2024 by adding two lines to robots.txt:

User-agent: ClaudeBot
Disallow: /
Enter fullscreen mode Exit fullscreen mode

In late 2026 those two lines do something narrower than most people think. Anthropic now documents three separate user agents, each with a different job. That rule only stops one of them, and the two it misses are the ones that decide whether Claude can find and cite your pages when someone asks it a question.

This post covers what each bot does, which ones to allow for the policy you actually want, and how to check your logs and your rules.

The three bots, in Anthropic's words

From Anthropic's help article "Does Anthropic crawl data from the web, and how can site owners block the crawler?":

User agent What Anthropic says it does What blocking it does
ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content" signals "that the site's future materials should be excluded" from training datasets
Claude-SearchBot "navigates the web to improve search result quality for users" "prevents our system from indexing your content for search optimization"
Claude-User "supports Claude AI users" "prevents our system from retrieving your content in response to a user query"

So:

  • ClaudeBot is about training.
  • Claude-SearchBot is about being findable when Claude searches.
  • Claude-User is about being readable when a person's question leads Claude to your URL.

Anthropic says all three respect robots.txt, and each one needs its own User-agent group. A rule for ClaudeBot does nothing for Claude-SearchBot.

Older guides also list Claude-Web and anthropic-ai. They aren't in Anthropic's current article, so leaving old rules for them is harmless, but don't count on them doing anything.

Pick a policy, then copy the rules

Policy A: "Don't train on my content, but let Claude cite me"

This is what most companies with docs, blogs or product pages want.

# robots.txt
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /
Enter fullscreen mode Exit fullscreen mode

The Allow groups are technically optional (no rule means allowed), but writing them down makes the intent obvious to the next person who edits the file. It also protects you from a catch-all User-agent: * group with a Disallow further down that you forgot about.

Policy B: "Keep Claude out of my private areas only" 

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /app/
Disallow: /account/
Disallow: /api/
Enter fullscreen mode Exit fullscreen mode

Several User-agent lines can share one group. This is the right shape for SaaS sites where marketing pages should be public and the logged-in app should never be fetched.

Policy C: "Block everything from Anthropic" 

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /
Enter fullscreen mode Exit fullscreen mode

Be clear about the cost. With Claude-User blocked, a user who pastes your URL into Claude and asks "what does this page say?" gets a refusal or a guess. With Claude-SearchBot blocked, your pages are far less likely to show up as sources when Claude searches the web.

Slowing it down instead of blocking

If the problem is load rather than principle, Anthropic says it supports the non-standard Crawl-delay directive:

User-agent: ClaudeBot
Crawl-delay: 1
Enter fullscreen mode Exit fullscreen mode

That's usually a better first step than a full block when a crawler is hammering a small server.

Remember: robots.txt is per host

Anthropic's article reminds you to add rules "for each subdomain". docs.example.com/robots.txt and www.example.com/robots.txt are separate files. A common mistake is a careful policy on www and nothing on docs, or the reverse.

for host in www.example.com docs.example.com blog.example.com; do
  echo "== $host"
  curl -s "https://$host/robots.txt" | grep -iA3 -E "claude|anthropic" || echo "(no Claude rules)"
done
Enter fullscreen mode Exit fullscreen mode

Test your rules before you ship them

Python's standard library can parse robots.txt. This script answers "can each Anthropic bot fetch these URLs?" for a live site:

# check_claude_robots.py
import sys
from urllib.robotparser import RobotFileParser
from urllib.parse import urlparse

BOTS = ["ClaudeBot", "Claude-SearchBot", "Claude-User"]

def check(urls):
    parsers = {}
    for url in urls:
        origin = "{0.scheme}://{0.netloc}".format(urlparse(url))
        if origin not in parsers:
            rp = RobotFileParser(origin + "/robots.txt")
            rp.read()
            parsers[origin] = rp
        rp = parsers[origin]
        verdicts = ["ok " if rp.can_fetch(bot, url) else "NO " for bot in BOTS]
        print(" | ".join(f"{b}:{v}" for b, v in zip(BOTS, verdicts)), url)

if __name__ == "__main__":
    check(sys.argv[1:])
Enter fullscreen mode Exit fullscreen mode 
python check_claude_robots.py \
  https://example.com/ \
  https://example.com/pricing \
  https://docs.example.com/getting-started
Enter fullscreen mode Exit fullscreen mode

urllib.robotparser is simpler than the matching Anthropic actually runs. It doesn't support every wildcard pattern, for example. Treat it as a smoke test: it catches "I blocked the wrong bot" and "I forgot the docs subdomain", which are the mistakes that actually happen.

Run it in CI against your staging robots.txt and fail the build if a URL you care about comes back NO for Claude-SearchBot or Claude-User.

Check what's really in your logs

Anyone can send User-Agent: ClaudeBot. Anthropic publishes the IP ranges its bots use at https://claude.com/crawling/bots.json and says that a request from an IP on that list is really from Anthropic. (It also warns that blocking by IP "may not work correctly". Use the list to verify traffic, not to firewall it.)

This script reads a combined-format access log, counts requests per Claude bot, and checks each source IP against the published list:

# verify_claude_traffic.py
import ipaddress, json, re, sys, urllib.request
from collections import Counter

BOTS_JSON = "https://claude.com/crawling/bots.json"
UA_RE = re.compile(r"(ClaudeBot|Claude-SearchBot|Claude-User)", re.I)
LINE_RE = re.compile(r'^(\S+) .*?"(?:GET|HEAD|POST) (\S+)[^"]*" (\d{3}) .*"([^"]*)"$')

def load_networks():
    data = json.load(urllib.request.urlopen(BOTS_JSON, timeout=10))
    nets = []
    def walk(node):  # don't depend on the exact JSON shape: find anything CIDR-like
        if isinstance(node, dict):
            for v in node.values(): walk(v)
        elif isinstance(node, list):
            for v in node: walk(v)
        elif isinstance(node, str):
            try: nets.append(ipaddress.ip_network(node, strict=False))
            except ValueError: pass
    walk(data)
    return nets

def main(path):
    nets = load_networks()
    hits, spoofed, statuses, paths = Counter(), Counter(), Counter(), Counter()
    with open(path, encoding="utf-8", errors="replace") as f:
        for line in f:
            m = LINE_RE.match(line.strip())
            if not m: continue
            ip, url, status, ua = m.groups()
            bot = UA_RE.search(ua)
            if not bot: continue
            name = bot.group(1)
            ok = any(ipaddress.ip_address(ip) in n for n in nets)
            (hits if ok else spoofed)[name] += 1
            if ok:
                statuses[(name, status)] += 1
                paths[(name, url)] += 1
    print(f"Loaded {len(nets)} Anthropic networks\n")
    for name in sorted(set(hits) | set(spoofed)):
        print(f"{name:18} verified={hits[name]:6}  unverified={spoofed[name]:6}")
    print("\nStatus codes (verified only):")
    for (name, status), n in sorted(statuses.items()):
        print(f"  {name:18} {status} x{n}")
    print("\nTop paths (verified only):")
    for (name, url), n in paths.most_common(15):
        print(f"  {n:5}  {name:18} {url}")

if __name__ == "__main__":
    main(sys.argv[1])
Enter fullscreen mode Exit fullscreen mode 
python verify_claude_traffic.py /var/log/nginx/access.log
Enter fullscreen mode Exit fullscreen mode

What to look for:

  • Lots of unverified hits. That's someone else using Claude's name. Rate-limit them like any other scraper.
  • Claude-User hits returning 403 or 503. Your WAF or bot-protection layer is blocking real user-initiated fetches, which robots.txt never asked for. This is very common behind aggressive "block AI bots" toggles.
  • Claude-SearchBot spending its visits on junk such as faceted URLs, ?sort= variants or old tag pages. Fix that with canonicals and internal linking, the same way you would for Googlebot.

Check your CDN's "block AI bots" switch too

Several CDNs and hosting platforms now ship one-click AI-bot blocking. Those switches often work at the edge and ignore your robots.txt entirely. If your robots.txt says Allow for Claude-SearchBot but your logs show it never reaching the origin, look at the CDN's bot settings before anything else.

Quick checklist

  • [ ] Decide separately on training (ClaudeBot), search (Claude-SearchBot) and user fetches (Claude-User)
  • [ ] One User-agent group per bot, or one shared group that names all three
  • [ ] Same policy on every subdomain's robots.txt
  • [ ] Rules tested with a script, ideally in CI
  • [ ] Logs checked against claude.com/crawling/bots.json
  • [ ] CDN/WAF bot settings match your robots.txt

Next in the series: Build a "Will Claude Cite Me?" checker with the Claude web search tool.

Top comments (1)

Collapse
 
citedy profile image
Dmitry Sergeev •

We need to output a short comment, casual, about the video content. Should start with a specific reaction or question about this video, not generic praise. Maybe ask about cost differences, or mention confusion. Keep it short, one sentence. Use lowercase start. No quotes, no hashtags. No URLs. No double hyphens. Should be casual. Let's produce: "anyone know how much extra crawl budget you actually lose when you block Claude-User vs Claude-SearchBot?" That's good. Ensure no punctuation errors, start