DEV Community

webdecoy
webdecoy

Posted on Originally published at webdecoy.com

How to Block AI Crawlers with robots.txt (and in WordPress)

Originally published on webdecoy.com.

robots.txt is the first thing to change if you don't want AI companies training on your site. It takes five minutes, it is free, and the large AI crawlers say they follow it. This tutorial gives you the rules to copy, shows how to add them in WordPress, and covers how to refuse the bots that ignore robots.txt.

The robots.txt rules to block AI crawlers

Paste this at the end of your robots.txt. It asks the best-known AI training crawlers to stay off your entire site:

# Block AI training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: AI2Bot
Disallow: /
Enter fullscreen mode Exit fullscreen mode

What each one is:

Token Who runs it What it does
GPTBot OpenAI Collects content for training OpenAI's models
ClaudeBot Anthropic Collects content for training Claude
CCBot Common Crawl Builds the open web dataset many AI models are trained on
Google-Extended Google Controls whether Gemini can train on your pages. Not a separate crawler, and it doesn't affect Google Search
Applebot-Extended Apple Controls whether Apple Intelligence can train on what Applebot crawls. Apple search features are unaffected
Meta-ExternalAgent Meta Collects content for training Meta's AI models
Bytespider ByteDance Crawler from TikTok's parent company, collecting data for its AI models
Amazonbot Amazon Crawls for Amazon services, including Alexa
cohere-ai Cohere Collects content for Cohere's models
AI2Bot Allen Institute for AI Collects content for open research models

A few rules about the format:

  • Several User-agent lines can share one Disallow. They form one group, and the rule applies to every name in it.
  • There is no wildcard for "all AI bots". User-agent: * means every crawler, including Googlebot. Each AI crawler has to be named.
  • Names are not case sensitive, but the token must match what the crawler looks for. ChatGPT is not a crawler token. GPTBot is.
  • robots.txt must sit at the root of your domain, as in https://example.com/robots.txt. A copy in a subfolder is ignored.

Training crawlers vs AI search crawlers

Before you add more names, decide what you want from AI search. AI companies run different bots for different jobs:

  • Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect content to train models.
  • AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages so ChatGPT search, Claude and Perplexity can quote and link to you. Blocking them takes you out of those answers.
  • Assistants that fetch for a person (ChatGPT-User, Claude-User, Perplexity-User) load a page when someone asks the assistant to read it.

A common setup is to block training and keep AI search, so you can still get cited and get clicks. The rules above do that. To leave AI search as well, add a second group:

# Also block AI search and assistants
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /
Enter fullscreen mode Exit fullscreen mode

To block AI crawlers from part of a site, such as a members area or a docs folder, replace Disallow: / with the path:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /members/
Disallow: /docs/
Enter fullscreen mode Exit fullscreen mode

How to add the rules in WordPress

WordPress serves robots.txt in one of two ways, and which one you have decides where the rules go.

1. Check for a real file. Look in your site's web root (the folder with wp-config.php) for robots.txt. If it exists, your server sends that file and WordPress never runs. Open it over SFTP or your host's file manager and paste the rules at the end.

2. No file? WordPress generates it. Open https://yoursite.com/robots.txt in a browser. If you see content but there is no file on disk, WordPress is building it. Then you have three options.

Option A: WebDecoy (automatic, stays current)

WebDecoy for WordPress is free on WordPress.org. Install it, then go to Settings > Good Bots > Automatic robots.txt and tick AI training crawler. Save.

WebDecoy adds a marked block with every AI training crawler it knows by name, 36 in the current release, compared with the 10 in the list above. The list updates when the plugin updates, so new crawlers are added without you editing anything. Your sitemap lines and other plugins' rules stay as they are, and the settings page shows the exact rules it added.

You can tick other categories too, such as AI search crawlers, SEO tools or commercial scrapers. If your site has a physical robots.txt file, WebDecoy tells you and shows the rules to copy into it.

Option B: Your SEO plugin

Yoast SEO and Rank Math both include a robots.txt editor. Paste the rules from above into it. You will need to add new crawlers by hand as they appear.

Option C: A code snippet

Add this to a small custom plugin or your theme's functions.php. It appends the rules to the robots.txt WordPress generates:

add_filter( 'robots_txt', function ( $output, $public ) {
    $bots = array(
        'GPTBot', 'ClaudeBot', 'CCBot', 'Google-Extended', 'Applebot-Extended',
        'Meta-ExternalAgent', 'Bytespider', 'Amazonbot', 'cohere-ai', 'AI2Bot',
    );

    $output .= "\n# Block AI training crawlers\n";
    foreach ( $bots as $bot ) {
        $output .= "User-agent: {$bot}\n";
    }
    $output .= "Disallow: /\n";

    return $output;
}, 10, 2 );
Enter fullscreen mode Exit fullscreen mode

Test your robots.txt

  1. Fetch it from the command line and check your rules are at the end:
   curl -s https://yoursite.com/robots.txt
Enter fullscreen mode Exit fullscreen mode
  1. If you use a CDN or a host cache, purge the cached robots.txt so crawlers get the new version.
  2. In Google Search Console, open the robots.txt report under Settings to confirm Google fetched it without errors, and that Googlebot is still allowed.

Crawlers re-read robots.txt on their own schedule, usually within a day.

Bots that ignore robots.txt

robots.txt is a request. The named crawlers from OpenAI, Anthropic, Google, Apple and Common Crawl say they follow it, but nothing forces a bot to read it, and scrapers often don't.

To refuse them rather than ask, the block has to happen on your server. In WebDecoy for WordPress:

  1. Go to Settings > Good Bots and turn on Block AI crawlers. This refuses AI training crawlers and AI agents, including ones that ignore robots.txt. Search engines and AI search crawlers such as OAI-SearchBot and PerplexityBot stay allowed.
  2. WebDecoy starts in monitor mode, which counts what it would block without blocking anything. Check the Detections page, then turn monitor mode off on the Blocking tab.
  3. If you use a page cache and the AI Traffic page offers Count crawlers on cached pages, turn it on, so AI crawlers reach WebDecoy instead of getting a cached copy.

The AI Traffic page then shows which AI crawlers visit and how often, so you can see the rules working.

Quick checklist

  • Add the training crawler group to robots.txt.
  • Decide on AI search crawlers: allow them to stay in AI answers, or add the second group.
  • Keep Googlebot and Bingbot allowed.
  • Test with curl and Search Console.
  • Block at the server for bots that ignore robots.txt.

Install WebDecoy from WordPress.org to set up AI crawler rules in one click.

Top comments (0)