seotto.com

Posted on Nov 8 Edited on Nov 8

How long until Googlebot sees your change? Measure recrawl latency from your own access logs

#seo #logs #devops #python

You've fixed the broken canonical tags on 300 product pages. The PR is merged, the deploy is green. Then someone asks the obvious question: when will Google notice?

The usual answers are "a few days", "a few weeks", or "it depends". All true, none useful. The real answer is sitting in your access logs: you can measure how often crawlers revisit each part of your site, and how long after a deploy they first fetch the changed URLs.

This post is an experiment you can run on your own site. There are no universal numbers here; recrawl rates depend on the site, the section and how often pages change. That's the point: measure yours.

What we're measuring

Two numbers per page type (product pages, blog posts, category pages, docs):

Revisit interval: the typical gap between consecutive crawler fetches of the same URL.
Time to first fetch after a change: for URLs you changed at a known moment, how long until the crawler requested each one again.

The first tells you what to expect in general. The second tells you what actually happened after your deploy.

Step 1: Get crawler requests out of the logs

Any combined log format works. First, filter to the crawler you care about. User agents can be faked, so verify the IPs. Google documents how to verify Googlebot by reverse DNS (the hostname should end in googlebot.com or google.com, and a forward lookup should return the same IP), and publishes its crawler IP ranges.

# crawler_hits.py: access.log -> TSV of (timestamp, path) for verified Googlebot
import re, sys, socket, functools
from datetime import datetime

LINE = re.compile(r'(\S+) \S+ \S+ \[([^\]]+)\] "(?:GET|HEAD) (\S+) [^"]*" (\d{3}) .*"([^"]*)"$')

@functools.lru_cache(maxsize=None)
def verified(ip):
    try:
        host = socket.gethostbyaddr(ip)[0]
        return host.endswith((".googlebot.com", ".google.com")) and \
               ip in socket.gethostbyname_ex(host)[2]
    except OSError:
        return False

for line in open(sys.argv[1], errors="ignore"):
    m = LINE.search(line)
    if not m or "Googlebot" not in m.group(5):
        continue
    ip, ts, path, status, _ = m.groups()
    if status != "200" or not verified(ip):
        continue
    t = datetime.strptime(ts, "%d/%b/%Y:%H:%M:%S %z")
    print(f"{t.isoformat()}\t{path.split('?')[0]}")

DNS lookups are slow, which is why the cache matters. For big logs, verify against the published IP ranges instead.

Step 2: Group URLs into page types

Revisit intervals differ a lot between sections, so averaging across the whole site hides everything. Map paths to types with a few rules:

def page_type(path):
    if path.startswith("/product/"): return "product"
    if path.startswith("/blog/"):    return "blog"
    if path.startswith("/c/"):       return "category"
    if path == "/":                  return "home"
    return "other"

Step 3: Compute revisit intervals

# revisit.py: TSV from step 1 -> median revisit interval per page type
import sys, statistics
from collections import defaultdict
from datetime import datetime
from page_types import page_type

hits = defaultdict(list)
for line in open(sys.argv[1]):
    ts, path = line.rstrip("\n").split("\t")
    hits[path].append(datetime.fromisoformat(ts))

gaps = defaultdict(list)
for path, times in hits.items():
    times.sort()
    for a, b in zip(times, times[1:]):
        gaps[page_type(path)].append((b - a).total_seconds() / 3600)

for t, g in sorted(gaps.items()):
    print(f"{t:10} urls_with_2+_hits={sum(1 for p in hits if page_type(p)==t and len(hits[p])>1):6} "
          f"median_gap_h={statistics.median(g):7.1f}  p90_gap_h={statistics.quantiles(g, n=10)[-1]:7.1f}")

Use at least several weeks of logs. With only a few days, you'll only see the frequently crawled URLs and miss the slow ones entirely.

Also count the URLs with zero or one hit. Those are pages crawled less often than your log window, and often the most interesting finding.

Step 4: Measure time to first fetch after your change

Take the list of URLs you changed and the deploy timestamp, then find the first verified crawler hit after it:

from datetime import datetime, timezone
deploy = datetime(2026, 11, 3, 10, 0, tzinfo=timezone.utc)
changed = set(open("changed_urls.txt").read().split())
first = {}
for line in open("googlebot.tsv"):
    ts, path = line.rstrip("\n").split("\t")
    t = datetime.fromisoformat(ts)
    if path in changed and t >= deploy and path not in first:
        first[path] = t
waits = sorted((first[p] - deploy).total_seconds() / 3600 for p in first)
print(f"fetched {len(first)}/{len(changed)}; median wait {waits[len(waits)//2]:.1f}h" if waits else "none yet")

Rerun it daily. The "fetched X of Y" line becomes your progress bar for the fix.

Reading the results

Some patterns to look for, rather than numbers to compare against:

Large differences between page types. It's common for a homepage and top categories to be fetched far more often than deep pages. If your most valuable pages are in the slow group, internal linking and sitemap freshness are worth looking at.
A long tail with few hits. Pages the crawler barely visits are pages where any fix will take a long time to be seen.
Fetched, but not reflected yet. A fetch isn't the same as the index being updated. Logs show the request; Search Console's URL Inspection shows what's indexed.

Can you speed it up?

Within limits, yes: submit an updated sitemap with honest lastmod dates for changed URLs, make sure changed pages are linked from frequently crawled pages, and request indexing for a handful of the most important URLs through URL Inspection. None of these guarantee timing, and the measurement above tells you whether they helped.

At seotto.com we prefer this kind of measurement to rules of thumb: your logs tell you what crawlers actually do on your site, not what a forum thread says they do on average.

DEV Community