Crawl budget probably isn't your problem: read your server logs before you blame it

A 400-page site wasn't getting new posts indexed. The team read three articles about crawl budget, blocked their tag pages in robots.txt and trimmed the sitemap. Nothing changed. When someone finally opened the access logs, Googlebot had fetched every new post within a day of publishing. The pages were crawled fine. They just weren't being indexed.

Crawl budget is real. It's also the most over-diagnosed problem in technical SEO.

TL;DR

• Search engines have said crawl budget mainly matters for very large sites (hundreds of thousands of URLs and up) or sites with lots of rapidly changing URLs.
• If your site has a few thousand pages and new pages don't show up, the cause is usually discovery, duplication or quality, not crawl capacity.
• Your access logs tell you the truth: which URLs real crawlers fetch, how often and with what status codes.
• Verify crawler IPs. Plenty of scrapers claim to be Googlebot.
• My opinion: nobody should change robots.txt for "crawl budget" without first looking at a week of logs.

Step 1: pull crawler hits from your logs

For a standard nginx or Apache combined log:

grep -h 'Googlebot' /var/log/nginx/access.log* \
  | awk '{print $1, $4, $7, $9}' > googlebot.txt
wc -l googlebot.txt

Fields: IP, timestamp, path, status. If you're behind a CDN, use the CDN's logs or make sure your origin logs the real client IP.

Step 2: verify it's really Googlebot

The user agent is just a string. Real Googlebot IPs reverse-resolve to googlebot.com or google.com, and the forward lookup matches:

import socket

def is_google(ip):
    try:
        host = socket.gethostbyaddr(ip)[0]
    except OSError:
        return False
    if not host.endswith((".googlebot.com", ".google.com")):
        return False
    try:
        return ip in socket.gethostbyname_ex(host)[2]
    except OSError:
        return False

Google also publishes its crawler IP ranges as JSON, which is faster for large logs. On many sites, a surprising share of "Googlebot" traffic turns out to be fake, which inflates any crawl analysis built on user-agent matching alone.

Step 3: answer the questions that matter

With verified hits in hand, a short script gives you most of what you need:

from collections import Counter
from datetime import datetime

hits = []  # (ip, ts, path, status) for verified Googlebot only
for line in open("googlebot.txt"):
    ip, ts, path, status = line.split()
    if is_google(ip):
        hits.append((ts.lstrip("["), path, status))

print("hits per day:")
days = Counter(datetime.strptime(t[:11], "%d/%b/%Y").date() for t, _, _ in hits)
for d, n in sorted(days.items()):
    print(f"  {d} {n}")

print("status codes:", Counter(s for _, _, s in hits).most_common())

sections = Counter("/" + p.split("/")[1].split("?")[0] for _, p, _ in hits)
print("top sections:", sections.most_common(10))

params = sum(1 for _, p, _ in hits if "?" in p)
print(f"URLs with query strings: {params} of {len(hits)}")

(Reverse DNS on every line is slow. Cache results per IP for real logs.)

Reading the results

New pages fetched within a day or two of publishing? Crawling is fine. If they're not indexed, look at duplication, thin content, canonicals and internal links instead. Why did my pages disappear from Google? walks through those causes.

New pages never fetched? That's discovery, not budget. Are they in the sitemap? Linked from anywhere crawlable? Sitemap "Couldn't fetch" in Search Console covers one common reason the sitemap isn't helping.

Most hits going to parameter URLs, faceted filters or calendar pages? Now you have a real crawl efficiency problem, usually self-inflicted by infinite URL spaces. Fix the URL generation and internal links first, then consider robots.txt rules, carefully. How to check if robots.txt is blocking Google shows how to test a rule before shipping it.

Lots of 5xx or very slow responses to Googlebot? Crawlers back off from struggling servers. That does reduce crawling, and the fix is server health, not robots.txt.

Lots of 301s and 404s? Crawls spent on redirects and dead URLs. Update internal links to final URLs and clean the sitemap.

The robots.txt trap

Blocking sections in robots.txt to "save budget" can backfire. Blocked URLs can still be indexed (without content) if they're linked, and canonical or noindex signals on blocked pages can't be seen at all. Blocking is the right tool for genuinely infinite URL spaces, not for "less important" pages on a small site.

Make it routine

Run the log summary weekly and keep the numbers. Trends matter more than snapshots: a sudden drop in daily crawler hits, or a jump in 5xx, often shows up in logs days before Search Console reports anything. Pair it with a change monitor such as seotto.com for on-page tags, and with the structured data check from the previous post, and you'll catch most indexing problems from the server side first.

My take

"Crawl budget" became a catch-all explanation for "Google isn't indexing my pages", because it sounds technical and it's not your fault. Logs usually show something less comfortable: the pages were crawled and judged not worth indexing, or never linked well enough to be found. Both are fixable, but not with robots.txt.

Over to you

Do you look at raw access logs for crawler behaviour, or rely on Search Console's crawl stats report? And if you run a large site where crawl budget really is the bottleneck, what actually moved the needle for you?