DEV Community

Cover image for Stop AI scrapers without blocking Google: verified crawlers + a tarpit
North Dev
North Dev

Posted on

Stop AI scrapers without blocking Google: verified crawlers + a tarpit

#security #ai #webdev #devops

AI crawlers are hammering your site, ignoring robots.txt, and running up your bandwidth. The tempting fix — block by User-Agent — also blocks the real Googlebot you want, because anyone can claim any UA. Here's the approach that separates the two.

User-Agent is a claim, not an identity

User-Agent: Googlebot means nothing — a scraper sets it in one line. So a UA allowlist lets impostors in, and a UA blocklist eventually catches a real crawler you cared about. You need to verify.

Forward-confirmed reverse DNS

The one check that actually works for the big crawlers: FCrDNS.

  1. Take the client IP, do a reverse DNS (PTR) lookup → a hostname.
  2. Check the hostname ends in the crawler's domain (.googlebot.com, .search.msn.com).
  3. Resolve that hostname forward (A/AAAA) and confirm it maps back to the original IP.

A spoofer can set a UA but can't make 66.249.x.x reverse to crawl-*.googlebot.com and forward back. Verified crawlers pass through untouched. (Range-only crawlers like DuckDuckBot need a published IP-range list instead.)

Tarpit the unverified — without hurting humans

For traffic that fails verification and ignores your rules, serve a tarpit: a cheap, bounded maze of generated links that wastes a scraper's crawl budget. Keep it honest:

  • Advertise the trap paths as Disallow in robots.txt. Compliant bots self-exclude; only bots that ignore robots fall in. That's what makes it ethical rather than hostile.
  • Fail open for humans. A normal visitor who stumbles onto a trap path shouldn't get mazed — gate on behavior, not on being unverified.
  • Non-destructive, synthetic content. No real data, size-capped, rate-limited.

Honest limits

This raises a scraper's cost; it isn't a wall. A determined operator rotating residential IPs defeats per-IP logic. The point is to make your site not worth the trouble.

I packaged the verify-good-bots + tarpit-the-rest logic into one Go binary, crawlgate — reverse proxy or middleware, forward-confirmed verification for Google/Bing/Yandex, rate limiting, and a non-destructive tarpit. Respects robots.txt and real crawlers.

Top comments (0)