You shipped 40,000 product pages. Three months later, Search Console says half of them are "Discovered – currently not indexed."
That's often a crawl budget problem: Googlebot is spending its time on URLs you don't care about, and never getting round to the ones you do.
This guide walks through how to audit a site for crawl budget issues using Search Console, your server logs and a short Python script.
First: do you actually have a crawl budget problem?
Most sites don't. Google's own crawl budget guide says it mainly matters for:
Very large sites (around a million or more unique pages) with content that changes regularly
Medium-to-large sites (10,000+ pages) with content that changes very frequently
Sites with a large share of URLs stuck in "Discovered – currently not indexed"
If you run a 200-page marketing site and pages aren't being indexed, the cause is almost always quality or internal linking, not crawl budget.
Crawl budget in one paragraph
Google decides how much to crawl based on two things:
Crawl capacity limit: how much crawling your server can handle without slowing down. Fast, healthy responses raise it; slow responses and server errors lower it.
Crawl demand: how much Google wants to crawl, based on how popular and how fresh your URLs seem.
Crawl budget gets wasted when that crawling goes to URLs that shouldn't exist, don't change, or don't matter.
Step 1: Read the Crawl Stats report
In Search Console go to Settings → Crawl stats. The Crawl Stats report shows 90 days of Googlebot activity. Check these in order:
What to check
Healthy
Warning sign
Total crawl requests
Stable or growing with the site
Sudden drops
Average response time
Low and stable
Rising or spiky
Host status
No issues
robots.txt fetch, DNS or server connectivity problems
By response
Mostly 200
High share of 404, 301 or 5xx
By purpose
Healthy mix of discovery and refresh
Almost no discovery on a growing site
By file type
Mostly HTML
Lots of JSON, JS or other resources you didn't expect
A rising response time plus falling crawl requests usually means your server is throttling Googlebot. That's a performance problem, not an SEO one.
Step 2: Check index coverage
Go to Indexing → Pages and look at the reasons pages aren't indexed:
Discovered – currently not indexed: Google knows the URL but hasn't crawled it. On big sites, this is the classic crawl budget symptom.
Crawled – currently not indexed: Google crawled it and chose not to index it. That's usually a quality signal, not a budget one.
Duplicate without user-selected canonical / Alternate page with proper canonical: large numbers here often point to parameter or faceted-URL bloat.
Click into each reason and look at the example URLs. Patterns jump out quickly: ?sort=, ?sessionid=, /tag/, /page/347/.
Step 3: Analyse your server logs
Search Console shows you a sample. Your logs show everything Googlebot actually requested. This is where most crawl waste is found.
Two important rules:
Verify Googlebot. Plenty of scrapers fake the Googlebot user agent. Google recommends you verify Googlebot with a reverse DNS lookup, then a forward lookup to confirm the IP matches.
Use at least 2–4 weeks of logs so you see real patterns, not a single day's noise.
Here's a script that parses a standard combined-format access log (the Apache and Nginx default), verifies Googlebot, and summarises where its crawl is going:
import re
import socket
import sys
from collections import Counter
from urllib.parse import urlsplit, parse_qs
LOG_LINE = re.compile(
r'(?P\S+) \S+ \S+ [[^]]+] "(?P\S+) (?P
r'(?P\d{3}) \S+ "[^"]" "(?P[^"]*)"'
)
_dns_cache = {}
def is_real_googlebot(ip: str) -> bool:
"""Reverse DNS, then forward-confirm, as Google recommends."""
if ip in _dns_cache:
return _dns_cache[ip]
try:
host = socket.gethostbyaddr(ip)[0]
ok = host.endswith((".googlebot.com", ".google.com")) and \
ip in socket.gethostbyname_ex(host)[2]
except (socket.herror, socket.gaierror):
ok = False
_dns_cache[ip] = ok
return ok
def main(logfile: str) -> None:
statuses, sections, params = Counter(), Counter(), Counter()
total = with_params = fake = 0
with open(logfile, encoding="utf-8", errors="ignore") as f:
for line in f:
m = LOG_LINE.match(line)
if not m or "Googlebot" not in m["ua"]:
continue
if not is_real_googlebot(m["ip"]):
fake += 1
continue
total += 1
url = urlsplit(m["path"])
statuses[m["status"]] += 1
first = url.path.strip("/").split("/")[0] or "(home)"
sections["/" + first] += 1
if url.query:
with_params += 1
params.update(parse_qs(url.query).keys())
if not total:
print("No verified Googlebot requests found.")
return
print(f"Verified Googlebot requests: {total} (ignored {fake} fake)")
print(f"URLs with parameters: {with_params} ({with_params / total:.0%})\n")
print("By status:", dict(statuses.most_common()))
print("\nTop sections:")
for s, n in sections.most_common(15):
print(f" {n:>7} {s}")
print("\nTop query parameters:")
for p, n in params.most_common(15):
print(f" {n:>7} {p}")
if name == "main":
main(sys.argv[1])
Run it with python crawl_audit.py access.log. Reverse DNS lookups are slow, but the cache means each IP is only checked once.
What you're looking for in the output:
A high percentage of parameter URLs. Sort orders, filters, tracking and session parameters are the most common crawl sink.
Sections that get crawled far more than they're worth, like tag archives, internal search results or calendar pages.
Lots of 301s and 404s. Every redirect or dead URL is a wasted request.
Important sections with very few hits. These are the pages Google is neglecting.
For multi-gigabyte logs, a dedicated log analyser or loading the logs into a database or BigQuery will be faster than a single Python process.
Step 4: Crawl the site yourself and compare
Run a crawler (Screaming Frog, Sitebulb or an open-source alternative) and compare three lists:
URLs in your XML sitemaps: what you want indexed
URLs your crawler finds via internal links: what your site exposes
URLs Googlebot requests in your logs: what Google actually spends time on
The gaps between these lists are the audit findings:
In logs but not in sitemaps → crawl waste
In sitemaps but rarely in logs → pages Google isn't prioritising
Found by the crawler but not in sitemaps → orphan sitemaps or uncontrolled URL generation
The most common crawl budget wasters
Faceted navigation and parameters
Filter combinations (?color=red&size=m&sort=price) can generate millions of near-duplicate URLs. Google has specific guidance on faceted navigation. The usual fix is blocking unimportant filter combinations in robots.txt and not linking to them crawlably.
noindex used to save crawl budget
A common misconception: noindex does not stop crawling. Google has to crawl the page to see the tag. If you want to stop Google requesting a URL pattern at all, use robots.txt. Never block a page in robots.txt that you're also trying to noindex, though: Google won't see the tag.
Redirect chains
A → B → C → D costs multiple requests for one page. Point internal links and redirects straight at the final URL.
Soft 404s and infinite spaces
Empty category pages returning 200, endless calendar pages, or internal search result pages that link to more search results. Return proper 404/410 status codes and stop linking to infinite paths.
Slow or erroring servers
If average response time is high or you see 5xx errors in Crawl Stats, Google will reduce its crawl rate. Caching, a CDN and fixing slow database queries often increase crawl volume without touching anything SEO-related.
Stale sitemaps
Sitemaps full of redirected, noindexed or 404 URLs, or lastmod dates that change on every build, teach Google to trust your sitemap less. Only list canonical, indexable 200 URLs and only update lastmod when content actually changes.
A quick audit checklist
Confirm the site is large or fast-changing enough for crawl budget to matter
Review Crawl Stats: response time, host status, response codes, file types
Review "Discovered – currently not indexed" volume and URL patterns
Analyse 2–4 weeks of server logs with verified Googlebot only
Measure the share of crawl going to parameter, faceted and low-value URLs
Compare sitemap URLs, crawlable URLs and Googlebot-requested URLs
Fix redirect chains, soft 404s and infinite URL spaces
Block unimportant parameter patterns in robots.txt
Improve server response times
Clean sitemaps: canonical, indexable, 200-status URLs only
Re-check Crawl Stats and index coverage after 4–8 weeks
Wrapping up
Crawl budget audits come down to one question: is Googlebot spending its time on the URLs you want indexed? Search Console tells you the symptoms, your logs tell you the cause, and most of the fixes are ordinary engineering: fewer junk URLs, faster responses and cleaner sitemaps.
If you're working out where crawl and indexing issues sit in the bigger picture, I've written about where technical SEO fits alongside on-page and off-page work.
Have you found a surprising crawl sink in your own logs? I'd like to hear about it in the comments.
Top comments (0)