Is that Googlebot real? Verify crawler hits in your access logs with Python

Hanin Mp — Fri, 23 Oct 2026 08:10:13 +0000

Any client can send Googlebot/2.1 in its user agent. If you build crawl-budget charts or "Googlebot is hitting our 404s" tickets from a raw log filter on that string, some of your numbers may describe scrapers, not Google.

This tutorial builds a small Python script that tags every request claiming to be Googlebot as verified or spoofed. It uses Google's published IP ranges first, because that check is fast and works offline once you have the file, and offers reverse DNS as an optional fallback.

What you need

Python 3.9 or newer (standard library only)
An access log in Apache or Nginx combined format
Google's crawler range file (the script downloads it, or you can pass a local copy)

How Google says to verify its crawlers

Google's documentation describes two methods:

Manual, with DNS. Reverse-resolve the IP, check the hostname ends in googlebot.com, google.com or googleusercontent.com, then forward-resolve that hostname and confirm it returns the original IP.
Automatic, with IP ranges. Match the IP against JSON files of CIDR ranges that Google publishes for each crawler category.

Google splits its fetchers into three groups, each with its own reverse DNS mask and range file:

Group	Example	Range file
Common crawlers	Googlebot	`common-crawlers.json`
Special-case crawlers	AdsBot	`special-crawlers.json`
User-triggered fetchers	Site Verifier	`user-triggered-fetchers.json` and related files

For Googlebot you want common-crawlers.json:

https://developers.google.com/static/crawling/ipranges/common-crawlers.json

The file has a creationTime and a prefixes list. Each entry holds either an ipv4Prefix or an ipv6Prefix:

{
  "creationTime": "2026-10-22T14:46:04.000000",
  "prefixes": [
    {"ipv6Prefix": "2001:4860:4801:10::/64"},
    {"ipv4Prefix": "66.249.66.0/27"}
  ]
}

The script

Save this as verify_googlebot.py.

#!/usr/bin/env python3
"""Tag "Googlebot" requests in an access log as verified or spoofed.

Method 1 (default): match the client IP against Google's published ranges.
Method 2 (--dns): for IPs that fail method 1, run a forward-confirmed
reverse DNS check as described in Google's documentation.

Usage:
    python verify_googlebot.py access.log
    python verify_googlebot.py access.log --ranges ranges.json --csv out.csv
    python verify_googlebot.py access.log --dns
"""
import argparse
import csv
import ipaddress
import json
import re
import socket
import sys
import urllib.request
from collections import Counter
from functools import lru_cache

RANGES_URL = "https://developers.google.com/static/crawling/ipranges/common-crawlers.json"
ALLOWED_SUFFIXES = (".googlebot.com", ".google.com", ".googleusercontent.com")

# Apache/Nginx "combined" format
LOG_RE = re.compile(
    r'^(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] '
    r'"(?P<method>[A-Z]+) (?P<path>\S+)[^"]*" '
    r'(?P<status>\d{3}) (?P<bytes>\S+) '
    r'"(?P<referer>[^"]*)" "(?P<ua>[^"]*)"'
)


def load_ranges(source):
    """Load a Google range file from a URL or a local path."""
    if source.startswith(("http://", "https://")):
        with urllib.request.urlopen(source, timeout=30) as resp:
            data = json.load(resp)
    else:
        with open(source, encoding="utf-8") as fh:
            data = json.load(fh)
    networks = []
    for entry in data["prefixes"]:
        prefix = entry.get("ipv4Prefix") or entry.get("ipv6Prefix")
        if prefix:
            networks.append(ipaddress.ip_network(prefix))
    return networks, data.get("creationTime", "unknown")


def make_range_checker(networks):
    """Return a cached function: is this IP string inside Google's ranges?"""
    cache = {}

    def check(ip_str):
        if ip_str not in cache:
            try:
                ip = ipaddress.ip_address(ip_str)
                cache[ip_str] = any(
                    ip in net for net in networks if net.version == ip.version
                )
            except ValueError:
                cache[ip_str] = False
        return cache[ip_str]

    return check


@lru_cache(maxsize=None)
def fcrdns_ok(ip_str):
    """Forward-confirmed reverse DNS: IP -> hostname -> same IP."""
    try:
        host = socket.gethostbyaddr(ip_str)[0].lower().rstrip(".")
    except (socket.herror, socket.gaierror, OSError):
        return False
    if not host.endswith(ALLOWED_SUFFIXES):
        return False
    try:
        infos = socket.getaddrinfo(host, None)
    except (socket.gaierror, OSError):
        return False
    resolved = {ipaddress.ip_address(i[4][0]) for i in infos}
    return ipaddress.ip_address(ip_str) in resolved


def main():
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("logfile")
    ap.add_argument("--ranges", default=RANGES_URL, help="URL or local JSON path")
    ap.add_argument("--dns", action="store_true", help="fall back to FCrDNS")
    ap.add_argument("--csv", help="write per-IP results to this CSV file")
    args = ap.parse_args()

    networks, created = load_ranges(args.ranges)
    in_ranges = make_range_checker(networks)
    print(f"Loaded {len(networks)} prefixes (file creationTime: {created})")

    claimed = Counter()
    verified = Counter()
    status_by_state = {"verified": Counter(), "spoofed": Counter()}
    unparsed = 0

    with open(args.logfile, encoding="utf-8", errors="replace") as fh:
        for line in fh:
            m = LOG_RE.match(line)
            if not m:
                unparsed += 1
                continue
            if "googlebot" not in m["ua"].lower():
                continue
            ip = m["ip"]
            claimed[ip] += 1
            ok = in_ranges(ip)
            if not ok and args.dns:
                ok = fcrdns_ok(ip)
            state = "verified" if ok else "spoofed"
            if ok:
                verified[ip] += 1
            status_by_state[state][m["status"]] += 1

    total = sum(claimed.values())
    ok_total = sum(verified.values())
    print(f"Requests claiming Googlebot: {total}")
    print(f"  verified: {ok_total}")
    print(f"  spoofed:  {total - ok_total}")
    print(f"Unparsed lines: {unparsed}")
    for state, counts in status_by_state.items():
        print(f"Status codes ({state}): {dict(sorted(counts.items()))}")

    if args.csv:
        with open(args.csv, "w", newline="", encoding="utf-8") as out:
            w = csv.writer(out)
            w.writerow(["ip", "claimed_hits", "state"])
            for ip, n in claimed.most_common():
                w.writerow([ip, n, "verified" if ip in verified else "spoofed"])
        print(f"Wrote {args.csv}")


if __name__ == "__main__":
    sys.exit(main())

Why it is built this way

Ranges first. A set of CIDR checks is quick and needs no network once the file is loaded. Reverse DNS costs two lookups per IP and can time out, so it only runs for IPs that failed the range check, and only with --dns.
ipaddress handles IPv4 and IPv6. The net.version == ip.version guard avoids comparing an IPv6 address with an IPv4 network.
Per-IP caching. A busy log repeats the same crawler IPs thousands of times. The range checker and fcrdns_ok both cache their answers.
Streaming. The log is read line by line, so a multi-gigabyte file does not need to fit in memory.

Run it

python verify_googlebot.py access.log --csv googlebot_ips.csv

Add --dns to give IPs outside the ranges a second chance through reverse DNS. To work offline or pin a version of the ranges, download the JSON yourself and pass --ranges common-crawlers.json.

Expected output

I tested the script on a nine-line synthetic log containing real-looking Googlebot lines, a Googlebot-claiming scanner, a normal browser and one malformed line. For the test I used a reduced range file with four prefixes copied from Google's file (creation time 2026-10-22). Results:

Loaded 4 prefixes (file creationTime: 2026-10-22T14:46:04.000000)
Requests claiming Googlebot: 7
  verified: 4
  spoofed:  3
Unparsed lines: 1
Status codes (verified): {'200': 3, '404': 1}
Status codes (spoofed): {'200': 1, '404': 2}
Wrote out.csv

ip,claimed_hits,state
66.249.66.1,2,verified
45.33.32.156,2,spoofed
35.247.243.245,1,verified
2001:4860:4801:10::5,1,verified
66.249.99.1,1,spoofed

This is a correctness check on synthetic data, not a benchmark. I have not run it against production logs, so the proportion of spoofed traffic you see on a real site will differ.

The --dns path was tested with mocked gethostbyaddr and getaddrinfo calls only, not live DNS. It returned True for a hostname ending in googlebot.com that resolved back to the same IP and False for an unrelated hostname.

Common errors and gotchas

Every hit shows as spoofed. Check which IP your log records. Behind a CDN, load balancer or reverse proxy the first column is often the proxy, not the client. Log the real client IP (for example from X-Forwarded-For or your CDN's client-IP header) and make sure that header is only trusted from your own proxy.

An old range URL in a tutorial or script. Older guides point to googlebot.json under developers.google.com/search/apis/ipranges/. When I fetched that old URL, it returned the same content as common-crawlers.json, but do not rely on legacy paths. Use the current file listed in Google's verification page.

Other Google fetchers are missing. common-crawlers.json covers common crawlers. AdsBot belongs to the special-case group, and user-triggered fetchers have their own files. This script's filter looks for googlebot in the user agent, so it does not look at AdsBot lines at all. Extend the filter and load the other range files if you need those.

Log format mismatch. The regex expects the combined format. If your server adds fields (request time, host, upstream), adjust LOG_RE. The Unparsed lines counter tells you how many lines were skipped, so watch it.

Stale range file. The file carries a creationTime. If you cache it, refresh it regularly rather than shipping one copy forever.

Limitations and trade-offs

A passing IP check shows the request came from Google's published infrastructure. It says nothing about whether your page rendered correctly or got indexed.
Reverse DNS is accurate but slow at scale. Ranges are fast but only as fresh as your copy of the file.
Counting spoofed hits does not tell you why they are there. Some are scrapers, some are SEO tools that send a Googlebot user agent. Do not block by user agent alone, and test any firewall rule before deploying it, so you do not block real crawlers.
This covers Googlebot. Other crawlers, including AI bots, have their own verification methods, and vendors differ in what they publish. Check each vendor's documentation.

Where to use the output

Filter your crawl analysis to verified hits before you count crawl frequency, status codes or most-crawled directories. Then look at the spoofed list on its own: repeated 404s on admin paths from a "Googlebot" IP outside Google's ranges are a security signal, not a crawl-budget one.

Sources and further reading

Conclusion

User agents are claims. IP ranges and DNS are evidence. Adding a verification step takes about 100 lines of Python and means your crawl conclusions rest on real Googlebot requests.

Hanin MP is an SEO specialist in Sharjah, UAE, working on technical SEO, GEO/AEO and entity building for client projects across the UAE, GCC and India. Writer at The SEO Central.

DEV Community: Hanin Mp