DEV Community

Tom Herbin
Tom Herbin

Posted on

How to Block AI Crawlers from Scraping Your Website in 2026

#tutorial #beginners #webdev #security

Your content is feeding AI models — without your permission

Every day, AI companies send bots to crawl websites and ingest content for training data. If you run a blog, portfolio, or any content-heavy site, chances are your work has already been scraped multiple times. Most website owners don't even realize it's happening.

Why robots.txt alone isn't enough anymore

The traditional approach to controlling crawlers relies on robots.txt. You add a few Disallow rules, and well-behaved bots respect them. The problem? The landscape has changed dramatically.

In 2025-2026, dozens of new AI crawlers appeared — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, and many others. Each requires its own specific User-agent directive. Miss one, and your content is still being scraped. Worse, some crawlers don't consistently respect robots.txt at all.

Manually keeping track of every AI crawler's user-agent string is a full-time job. The list changes monthly as new models launch and companies rebrand their bots.

Step 1: Identify which crawlers are hitting your site

Before blocking anything, you need visibility. Check your server access logs for known AI crawler user-agent strings:

GPTBot/1.0
ClaudeBot/1.0
CCBot/2.0
Google-Extended
Bytespider
anthropic-ai
cohere-ai
Enter fullscreen mode Exit fullscreen mode

If you're on shared hosting without log access, tools like Cloudflare's bot analytics (free tier) can give you a rough picture.

Step 2: Decide what to block and what to allow

Not all crawlers are harmful. Googlebot and Bingbot drive your search traffic — blocking them kills your SEO. The key distinction:

  • Search engine crawlers (Googlebot, Bingbot): keep these allowed
  • AI training crawlers (GPTBot, CCBot, Bytespider): block if you don't want your content used for training
  • AI search crawlers (ChatGPT-User, PerplexityBot): your call — blocking these means your site won't appear in AI-powered search answers

There's no one-size-fits-all answer. A news site might want AI search visibility. A fiction writer probably doesn't want their novels in training datasets.

Step 3: Implement the blocks correctly

Here's a comprehensive robots.txt snippet for blocking major AI training crawlers:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /
Enter fullscreen mode Exit fullscreen mode

But robots.txt is just the first layer. For stronger protection, consider:

  • HTTP headers: The X-Robots-Tag: noai header is gaining adoption
  • Meta tags: <meta name="robots" content="noai, noimageai"> for page-level control
  • Rate limiting: Throttle suspicious crawl patterns at the server level

Step 4: Monitor and maintain your crawler rules

This is where most people drop off. You set up your blocks, forget about them, and six months later three new AI crawlers are happily scraping your site.

Set a quarterly reminder to review your crawler blocks. Check resources like the Dark Visitors directory for newly identified AI bots.

For those who want to skip the manual maintenance, tools like CrawlShield automate this process — they keep an updated database of AI crawlers and generate the right blocking rules for your site, so you don't have to track every new bot yourself.

The bottom line

Blocking AI crawlers from scraping your website requires more than a basic robots.txt edit. Identify which bots are visiting, decide your policy, implement multi-layered blocks, and commit to maintaining them. Your content is yours — you should decide who gets to use it.

Top comments (0)