Originally published on proxymint.
How to block AI crawlers: which bots train and which cite you, copy-paste robots.txt and server rules, bot verification, and what to do about bots that lie.
TL;DR
- robots.txt asks, the server enforces. A Disallow line stops the AI crawlers that choose to follow it; a 403 rule at the server or CDN stops the ones that keep their real user agent.
- Sort AI bots by purpose before blocking. Training crawlers such as GPTBot and ClaudeBot send almost no visits back, search crawlers can cite the site, and user-triggered fetchers only arrive when a person asks about a page.
- A user agent proves nothing. Verify a claimed bot by reverse and forward DNS, the vendor's published IP list or a Web Bot Auth signature.
- Crawlers that drop their name need behaviour rules. Rate limits and request-shape scoring work where IP blocks fail, because blocking residential IPs blocks real visitors too.
Key takeaways
- Block training crawlers first. GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider, Google-Extended and Applebot-Extended can go without any effect on classic search rankings.
- Keep robots.txt reachable. Serve it with a 200 to every bot, because a 5xx reads as a full disallow and a 4xx reads as no rules at all.
- Decide on AI search crawlers per assistant. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes the site from those answers and their citations.
- Test every rule like a visitor. Request a page as GPTBot, robots.txt as GPTBot, and the page as a normal browser from a home connection, and check for 403, 200 and 200.
Read the full guide on proxymint, with the commands, tables and sources.
Top comments (0)