DEV Community

Cover image for How to verify a real Googlebot (User-Agent isn't enough)
North Dev
North Dev

Posted on

How to verify a real Googlebot (User-Agent isn't enough)

#seo #security #devops #webdev

Your access log is full of Googlebot. Some of it is Google. A lot of it isn't. Here's how to tell them apart — because the User-Agent string is worthless for this, and blocking the fakes the wrong way will deindex you.

The User-Agent is a claim anyone can make

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) is a header. A scraper sets it in one line to inherit Googlebot's reputation (and your allowlist). So:

  • A UA allowlist lets every impostor straight in.
  • A UA blocklist eventually catches the real Googlebot — and now you're not in the index.

You have to verify identity, not read a label.

Forward-confirmed reverse DNS (the method Google itself documents)

  1. Take the client IP. Do a reverse DNS (PTR) lookup → a hostname.
  2. Check the hostname is under Google's domain: it must end in .googlebot.com or .google.com.
  3. Resolve that hostname forward (A/AAAA) and confirm it resolves back to the original IP.

All three must hold. A spoofer can forge the UA but can't make 66.249.66.1 reverse to crawl-*.googlebot.com and forward-resolve back to 66.249.66.1.

# client IP 66.249.66.1
dig -x 66.249.66.1 +short        # -> crawl-66-249-66-1.googlebot.com.
dig crawl-66-249-66-1.googlebot.com +short   # -> 66.249.66.1  ✓ match
Enter fullscreen mode Exit fullscreen mode

The suffix check in step 2 is not optional — a PTR record is controlled by whoever owns the IP's reverse zone, so evil-googlebot.com.attacker.net or a lookalike must fail. Confirm the dot-bounded suffix, then forward-confirm.

Bing, and the range-only crawlers

Bing (.search.msn.com), Apple, and Yandex support the same FCrDNS check. Some crawlers (e.g. DuckDuckBot) don't publish reverse DNS and instead publish an IP range list — for those, verify the IP is in the official, published ranges, and refresh that list.

Don't punish humans on the way

Once you can verify good bots, the temptation is to tarpit or block everything unverified. Keep it honest: a real person hitting your site isn't a verified crawler either. Gate on behavior (ignoring robots, abusive rate) — not on "failed bot verification" — so you slow abusive scrapers without breaking normal visitors.

I put forward-confirmed verification (Google/Bing/Yandex/Apple), rate limiting, and an ethical tarpit for the unverified-and-abusive into one Go binary, crawlgate — reverse proxy or middleware, respects robots.txt and real crawlers.

Top comments (0)