seotto.com

Posted on Nov 4

How to read your server logs like Googlebot's diary (with a short Python script)

#seo #python #logs #webdev

Search Console tells you what Google thinks about your site. Your access logs tell you what Googlebot actually did: which URLs it requested, how often, what status codes it got, and how long your server took to respond.

If you've never looked, you'll probably find at least one surprise: a crawler stuck in filter URLs, a section it never visits, or a stream of 404s you didn't know about.

This guide walks through doing it with standard tools and a short Python script. No paid log analyser needed.

Step 1: Get the logs

You need raw access logs from your web server, load balancer or CDN. The common "combined" format looks like this:

66.249.66.1 - - [02/Nov/2026:10:14:03 +0000] "GET /blog/some-post HTTP/1.1" 200 18234 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Where to find them:

Nginx: usually /var/log/nginx/access.log (rotated files as .1, .2.gz, and so on).
Apache: /var/log/apache2/access.log or /var/log/httpd/access_log.
CDNs and hosting platforms: often a log export or "log push" feature. Check your provider's docs.

Get at least two weeks if you can. Googlebot's behaviour varies day to day.

Step 2: Filter to Googlebot, quickly

A first look with grep:

zcat -f /var/log/nginx/access.log* | grep -i "googlebot" > googlebot.log
wc -l googlebot.log

That's your candidate set. But don't trust it yet.

Step 3: Verify it's really Google

Anyone can send a request with "Googlebot" in the user agent. Scrapers do it all the time. Google's documented way to verify is:

Reverse DNS lookup on the IP. The hostname should end in googlebot.com or google.com.
Forward DNS lookup on that hostname. It should resolve back to the same IP.

Google also publishes its crawler IP ranges as JSON files, which is faster for large logs. The DNS method is easy to script:

import socket
from functools import lru_cache

@lru_cache(maxsize=10000)
def is_real_googlebot(ip: str) -> bool:
    try:
        host = socket.gethostbyaddr(ip)[0]
    except OSError:
        return False
    if not (host.endswith(".googlebot.com") or host.endswith(".google.com")):
        return False
    try:
        return ip in socket.gethostbyname_ex(host)[2]
    except OSError:
        return False

The cache matters: Googlebot uses a limited set of IPs, so you'll only resolve each one once.

Step 4: Parse and summarise

Here's the full script. It reads a combined-format log, keeps verified Googlebot requests, and prints the views I find most useful.

import re, sys
from collections import Counter, defaultdict
from urllib.parse import urlsplit

LINE = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] '
    r'"(?P<method>\S+) (?P<path>\S+) [^"]*" (?P<status>\d{3}) \S+ '
    r'"[^"]*" "(?P<ua>[^"]*)"'
)

statuses = Counter()
sections = Counter()
params = Counter()
paths = Counter()
days = defaultdict(int)

for raw in open(sys.argv[1], errors="replace"):
    m = LINE.match(raw)
    if not m or "googlebot" not in m["ua"].lower():
        continue
    if not is_real_googlebot(m["ip"]):
        continue
    url = urlsplit(m["path"])
    statuses[m["status"]] += 1
    sections["/" + url.path.strip("/").split("/")[0]] += 1
    paths[url.path] += 1
    if url.query:
        params[url.query.split("=")[0]] += 1
    days[m["time"].split(":")[0]] += 1

print("Status codes:", statuses.most_common())
print("\nTop sections:", sections.most_common(15))
print("\nTop query params:", params.most_common(10))
print("\nMost-crawled paths:", paths.most_common(20))
print("\nRequests per day:")
for d, n in sorted(days.items()):
    print(f"  {d}  {n}")

(Paste is_real_googlebot from step 3 at the top.) Run it:

python3 gbot.py googlebot.log

Step 5: What to look for

The numbers aren't interesting on their own. Comparisons are.

Status codes

A healthy site is mostly 200s and some 304s (not modified). Watch for:

Lots of 404s. Googlebot is following links, old sitemaps or external links to pages that don't exist. Find the top ones and redirect or fix the source.
Any 5xx. Server errors during crawling can slow Google down. Correlate them with the times in your logs.
Lots of 301s. Your internal links might point at old URLs. Update links to go straight to the final URL.

Sections

Compare where Googlebot spends time with where your important pages are. If /tag/ gets more crawl requests than /products/, that's a signal worth acting on.

Query parameters

This is where "crawl traps" show up: sort orders, filters, session IDs and tracking parameters creating near-infinite URL variations. If a parameter like sort or color shows up thousands of times, consider whether those URLs should be crawlable at all.

Pages it never visits

Make a list of your important URLs (from your CMS or sitemap) and diff it against the paths in the log:

cut -d' ' -f7 googlebot.log | cut -d'?' -f1 | sort -u > crawled.txt
sort -u important-urls.txt > important.txt
comm -23 important.txt crawled.txt   # important but not crawled

Pages in that output weren't requested during your log window. They may be deep in the site, weakly linked or seen as low value.

Response time

If your log format includes request time (Nginx $request_time), add it to the regex and look at the slowest URLs for Googlebot. Slow responses for crawlers usually mean slow responses for people too.

Step 6: Make it a habit

Running this once is useful. Running it monthly, and comparing, is where the value is: you'll see a crawl shift after a redesign, a new parameter trap after a feature launch, or a section going quiet.

If you'd rather not maintain scripts, a site auditing tool like seotto.com can watch the crawlable side of your site (status codes, links, indexability), and your logs fill in the crawler's side of the story. The two together give you the full picture.

Quick reference

Get 2+ weeks of raw access logs.
Filter on user agent, then verify with reverse + forward DNS.
Summarise by status, section, query parameter and day.
Diff important URLs against crawled URLs.
Repeat monthly and compare.

DEV Community