Filtering a server log for the user agent string "Googlebot" feels like it should be the whole job. It isn't. A meaningful share of the traffic claiming to be Googlebot in a raw log is a scraper, a low-quality SEO tool, or an outright bot that set its user agent to "Googlebot" specifically because that string gets past naive filtering and sometimes earns preferential treatment from rate limiters. This walkthrough covers the actual verification step, and how to build it into a repeatable check rather than a one-time manual lookup.
Step 1: Pull the Raw Requests Matching the User Agent String
Start with a straightforward filter against your access log, matching any user agent containing "Googlebot," case-insensitive. Depending on log format, this is a single grep command against a flat file or an equivalent query against your log aggregation platform. Keep the full log line at this stage, including the source IP address, since that's the field the next step actually depends on.
At this point you have a superset that includes real Googlebot traffic and every impostor pretending to be it. Don't treat any conclusions drawn from this list as reliable yet.
Step 2: Run a Reverse DNS Lookup on Each Source IP
For every distinct IP address in your filtered set, run a reverse DNS lookup. A genuine Googlebot request will resolve to a hostname ending in googlebot.com or google.com. Any IP that resolves to something else, or that fails to resolve at all, is not verified Googlebot traffic, regardless of what its user agent string claims.
This is the step most naive filtering skips entirely, and it's the single biggest gap between "traffic that says it's Googlebot" and "traffic that actually is."
Step 3: Run a Forward Lookup to Confirm the Hostname Maps Back
A reverse lookup alone isn't quite sufficient, since DNS records can be spoofed in ways that pass a reverse check but not a full round trip. Google's own documentation specifies the complete verification method: after the reverse lookup resolves to a googlebot.com or google.com hostname, perform a forward DNS lookup on that hostname and confirm it resolves back to the original IP address you started with. If it doesn't match, discard the request from your verified set.
This two-step round trip is what makes the verification reliable rather than just probable, and it's cheap enough computationally that there's no good reason to skip it once you're scripting this instead of doing it by hand.
Step 4: Automate the Verification, Don't Spot-Check It
Doing this verification manually on a handful of suspicious-looking IPs is a reasonable first pass, but the real value comes from running it automatically against every distinct IP in your filtered set, every time you process a new log window. Most scripting languages have built-in or easily available DNS resolution libraries, so this isn't a heavy lift to automate once you've written it correctly the first time.
Cache the verification results per IP for a reasonable window, a day or two is usually fine, since Google's crawler IP ranges don't change minute to minute, and re-verifying the same IP on every single log line wastes DNS lookups for no additional accuracy.
Step 5: Segment by Googlebot Variant, Not Just "Googlebot"
Once you have a verified set, don't collapse it into a single bucket. The user agent string itself tells you which variant made the request: standard desktop Googlebot, Googlebot Smartphone, Googlebot Image, and Googlebot Video each carry distinct signatures. Keeping these separate matters because they crawl different things and at different rates, and a pattern that looks healthy in aggregate can hide a real gap in one specific variant, most commonly Googlebot Smartphone given how central mobile-first indexing is to how Google actually evaluates a site today.
Step 6: Cross-Check Against Known Googlebot IP Ranges as a Sanity Check
Google publishes downloadable IP range lists for its crawlers as an additional verification layer, useful as a fast sanity check or a fallback in environments where live DNS lookups aren't practical for every request (extremely high log volume, for instance). This shouldn't replace the DNS round-trip method described above as your primary verification, since IP ranges do shift over time and a stale downloaded list will eventually produce false negatives, but it's a reasonable supplementary check.
Step 7: Watch for the Inverse Problem Too
Verification usually gets framed as "catching fake Googlebot traffic," but the inverse mistake is just as costly: discarding real Googlebot requests because a verification script has a bug, an expired DNS cache, or an overly strict matching pattern. If your verified Googlebot request count drops sharply between two log windows with no corresponding change in Search Console's crawl stats, check the verification logic itself before concluding Google actually crawled your site less.
Step 8: Consider a Tool That Automates the Whole Chain
Writing and maintaining the reverse-plus-forward DNS verification script yourself is a reasonable choice for a team comfortable owning that code long-term, but it's not the only option. Screaming Frog's Log File Analyser implements this exact verification method internally, importing a raw log and automatically separating confirmed Googlebot traffic from everything else without requiring a custom script at all. For teams that just need the verified output and don't want to own the DNS-lookup code, this is often the faster path to a trustworthy dataset.
Step 9: Remember CDN-Fronted Traffic Changes the Picture
If your site sits behind a CDN, the verification approach described above still applies, but it needs to run against the CDN's edge logs rather than your origin server's logs, since a meaningful share of requests never reach the origin at all once caching is involved. Cloudflare and most other CDN providers expose bot classification and edge-level logging as part of their platform, which can serve as a useful cross-check against your own DNS-based verification, though it shouldn't fully replace verifying independently if the accuracy of your crawl analysis actually matters for a decision you're about to make.
A Quick Sanity Check Worth Running Once
Before trusting a new verification pipeline for anything important, run it against a log window you can independently confirm, a period right after publishing new content, for example, where you'd expect to see Googlebot activity if crawling is working normally. If your verified dataset shows zero or near-zero real Googlebot traffic during a window where you have other evidence crawling was happening, the bug is almost certainly in the verification script, not in Google's crawling behavior. This single check catches most implementation mistakes before they quietly corrupt every analysis built on top of the dataset afterward.
Putting It Together
A verified Googlebot dataset is the prerequisite for every other piece of log file analysis: crawl budget waste detection, sitemap-versus-crawl comparisons, and response time correlation all depend on the underlying data actually being Googlebot and not noise pretending to be it. Skipping verification doesn't just add error bars to the analysis, it can flip conclusions entirely, since impostor bots often hit very different URL patterns than the real crawler does.
Our broader guide on building a full log file analysis workflow covers what to do once you have this verified dataset in hand, including how to turn crawl waste findings into actual fixes. For teams whose crawl data and indexing results have drifted apart in ways that are hard to diagnose from Search Console alone, 137Foundry's technical SEO service works through this kind of log-level audit directly.
Top comments (0)