<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Usman Ghani</title>
    <description>The latest articles on DEV Community by Usman Ghani (@usman_ghani_63768e6fec59e).</description>
    <link>https://dev.to/usman_ghani_63768e6fec59e</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4234915%2F14e12b1a-c51f-4032-83a0-266ce5e32b4a.jpg</url>
      <title>DEV Community: Usman Ghani</title>
      <link>https://dev.to/usman_ghani_63768e6fec59e</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/usman_ghani_63768e6fec59e"/>
    <language>en</language>
    <item>
      <title>How to audit a website for crawl budget issues?</title>
      <dc:creator>Usman Ghani</dc:creator>
      <pubDate>Mon, 09 Nov 2026 10:46:38 +0000</pubDate>
      <link>https://dev.to/usman_ghani_63768e6fec59e/how-to-audit-a-website-for-crawl-budget-issues-393k</link>
      <guid>https://dev.to/usman_ghani_63768e6fec59e/how-to-audit-a-website-for-crawl-budget-issues-393k</guid>
      <description>&lt;p&gt;You shipped 40,000 product pages. Three months later, Search Console says half of them are "Discovered – currently not indexed."&lt;/p&gt;

&lt;p&gt;That's often a crawl budget problem: Googlebot is spending its time on URLs you don't care about, and never getting round to the ones you do.&lt;/p&gt;

&lt;p&gt;This guide walks through how to audit a site for crawl budget issues using Search Console, your server logs and a short Python script.&lt;br&gt;
First: do you actually have a crawl budget problem?&lt;br&gt;
Most sites don't. Google's own crawl budget guide says it mainly matters for:&lt;/p&gt;

&lt;p&gt;Very large sites (around a million or more unique pages) with content that changes regularly&lt;br&gt;
Medium-to-large sites (10,000+ pages) with content that changes very frequently&lt;br&gt;
Sites with a large share of URLs stuck in "Discovered – currently not indexed"&lt;/p&gt;

&lt;p&gt;If you run a 200-page marketing site and pages aren't being indexed, the cause is almost always quality or internal linking, not crawl budget.&lt;br&gt;
Crawl budget in one paragraph&lt;br&gt;
Google decides how much to crawl based on two things:&lt;/p&gt;

&lt;p&gt;Crawl capacity limit: how much crawling your server can handle without slowing down. Fast, healthy responses raise it; slow responses and server errors lower it.&lt;br&gt;
Crawl demand: how much Google wants to crawl, based on how popular and how fresh your URLs seem.&lt;/p&gt;

&lt;p&gt;Crawl budget gets wasted when that crawling goes to URLs that shouldn't exist, don't change, or don't matter.&lt;br&gt;
Step 1: Read the Crawl Stats report&lt;br&gt;
In Search Console go to Settings → Crawl stats. The Crawl Stats report shows 90 days of Googlebot activity. Check these in order:&lt;/p&gt;

&lt;p&gt;What to check&lt;br&gt;
Healthy&lt;br&gt;
Warning sign&lt;br&gt;
Total crawl requests&lt;br&gt;
Stable or growing with the site&lt;br&gt;
Sudden drops&lt;br&gt;
Average response time&lt;br&gt;
Low and stable&lt;br&gt;
Rising or spiky&lt;br&gt;
Host status&lt;br&gt;
No issues&lt;br&gt;
robots.txt fetch, DNS or server connectivity problems&lt;br&gt;
By response&lt;br&gt;
Mostly 200&lt;br&gt;
High share of 404, 301 or 5xx&lt;br&gt;
By purpose&lt;br&gt;
Healthy mix of discovery and refresh&lt;br&gt;
Almost no discovery on a growing site&lt;br&gt;
By file type&lt;br&gt;
Mostly HTML&lt;br&gt;
Lots of JSON, JS or other resources you didn't expect&lt;/p&gt;

&lt;p&gt;A rising response time plus falling crawl requests usually means your server is throttling Googlebot. That's a performance problem, not an SEO one.&lt;br&gt;
Step 2: Check index coverage&lt;br&gt;
Go to Indexing → Pages and look at the reasons pages aren't indexed:&lt;/p&gt;

&lt;p&gt;Discovered – currently not indexed: Google knows the URL but hasn't crawled it. On big sites, this is the classic crawl budget symptom.&lt;br&gt;
Crawled – currently not indexed: Google crawled it and chose not to index it. That's usually a quality signal, not a budget one.&lt;br&gt;
Duplicate without user-selected canonical / Alternate page with proper canonical: large numbers here often point to parameter or faceted-URL bloat.&lt;/p&gt;

&lt;p&gt;Click into each reason and look at the example URLs. Patterns jump out quickly: ?sort=, ?sessionid=, /tag/, /page/347/.&lt;br&gt;
Step 3: Analyse your server logs&lt;br&gt;
Search Console shows you a sample. Your logs show everything Googlebot actually requested. This is where most crawl waste is found.&lt;/p&gt;

&lt;p&gt;Two important rules:&lt;/p&gt;

&lt;p&gt;Verify Googlebot. Plenty of scrapers fake the Googlebot user agent. Google recommends you verify Googlebot with a reverse DNS lookup, then a forward lookup to confirm the IP matches.&lt;br&gt;
Use at least 2–4 weeks of logs so you see real patterns, not a single day's noise.&lt;/p&gt;

&lt;p&gt;Here's a script that parses a standard combined-format access log (the Apache and Nginx default), verifies Googlebot, and summarises where its crawl is going:&lt;/p&gt;

&lt;p&gt;import re&lt;br&gt;
import socket&lt;br&gt;
import sys&lt;br&gt;
from collections import Counter&lt;br&gt;
from urllib.parse import urlsplit, parse_qs&lt;/p&gt;

&lt;p&gt;LOG_LINE = re.compile(&lt;br&gt;
    r'(?P\S+) \S+ \S+ [[^]]+] "(?P\S+) (?P\S+) [^"]&lt;em&gt;" '&lt;br&gt;
    r'(?P\d{3}) \S+ "[^"]&lt;/em&gt;" "(?P[^"]*)"'&lt;br&gt;
)&lt;/p&gt;

&lt;p&gt;_dns_cache = {}&lt;/p&gt;

&lt;p&gt;def is_real_googlebot(ip: str) -&amp;gt; bool:&lt;br&gt;
    """Reverse DNS, then forward-confirm, as Google recommends."""&lt;br&gt;
    if ip in _dns_cache:&lt;br&gt;
        return _dns_cache[ip]&lt;br&gt;
    try:&lt;br&gt;
        host = socket.gethostbyaddr(ip)[0]&lt;br&gt;
        ok = host.endswith((".googlebot.com", ".google.com")) and \&lt;br&gt;
             ip in socket.gethostbyname_ex(host)[2]&lt;br&gt;
    except (socket.herror, socket.gaierror):&lt;br&gt;
        ok = False&lt;br&gt;
    _dns_cache[ip] = ok&lt;br&gt;
    return ok&lt;/p&gt;

&lt;p&gt;def main(logfile: str) -&amp;gt; None:&lt;br&gt;
    statuses, sections, params = Counter(), Counter(), Counter()&lt;br&gt;
    total = with_params = fake = 0&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;with open(logfile, encoding="utf-8", errors="ignore") as f:
    for line in f:
        m = LOG_LINE.match(line)
        if not m or "Googlebot" not in m["ua"]:
            continue
        if not is_real_googlebot(m["ip"]):
            fake += 1
            continue

        total += 1
        url = urlsplit(m["path"])
        statuses[m["status"]] += 1
        first = url.path.strip("/").split("/")[0] or "(home)"
        sections["/" + first] += 1
        if url.query:
            with_params += 1
            params.update(parse_qs(url.query).keys())

if not total:
    print("No verified Googlebot requests found.")
    return

print(f"Verified Googlebot requests: {total} (ignored {fake} fake)")
print(f"URLs with parameters: {with_params} ({with_params / total:.0%})\n")
print("By status:", dict(statuses.most_common()))
print("\nTop sections:")
for s, n in sections.most_common(15):
    print(f"  {n:&amp;gt;7}  {s}")
print("\nTop query parameters:")
for p, n in params.most_common(15):
    print(f"  {n:&amp;gt;7}  {p}")
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;if &lt;strong&gt;name&lt;/strong&gt; == "&lt;strong&gt;main&lt;/strong&gt;":&lt;br&gt;
    main(sys.argv[1])&lt;/p&gt;

&lt;p&gt;Run it with python crawl_audit.py access.log. Reverse DNS lookups are slow, but the cache means each IP is only checked once.&lt;/p&gt;

&lt;p&gt;What you're looking for in the output:&lt;/p&gt;

&lt;p&gt;A high percentage of parameter URLs. Sort orders, filters, tracking and session parameters are the most common crawl sink.&lt;br&gt;
Sections that get crawled far more than they're worth, like tag archives, internal search results or calendar pages.&lt;br&gt;
Lots of 301s and 404s. Every redirect or dead URL is a wasted request.&lt;br&gt;
Important sections with very few hits. These are the pages Google is neglecting.&lt;/p&gt;

&lt;p&gt;For multi-gigabyte logs, a dedicated log analyser or loading the logs into a database or BigQuery will be faster than a single Python process.&lt;br&gt;
Step 4: Crawl the site yourself and compare&lt;br&gt;
Run a crawler (Screaming Frog, Sitebulb or an open-source alternative) and compare three lists:&lt;/p&gt;

&lt;p&gt;URLs in your XML sitemaps: what you want indexed&lt;br&gt;
URLs your crawler finds via internal links: what your site exposes&lt;br&gt;
URLs Googlebot requests in your logs: what Google actually spends time on&lt;/p&gt;

&lt;p&gt;The gaps between these lists are the audit findings:&lt;/p&gt;

&lt;p&gt;In logs but not in sitemaps → crawl waste&lt;br&gt;
In sitemaps but rarely in logs → pages Google isn't prioritising&lt;br&gt;
Found by the crawler but not in sitemaps → orphan sitemaps or uncontrolled URL generation&lt;br&gt;
The most common crawl budget wasters&lt;br&gt;
Faceted navigation and parameters&lt;br&gt;
Filter combinations (?color=red&amp;amp;size=m&amp;amp;sort=price) can generate millions of near-duplicate URLs. Google has specific guidance on faceted navigation. The usual fix is blocking unimportant filter combinations in robots.txt and not linking to them crawlably.&lt;br&gt;
noindex used to save crawl budget&lt;br&gt;
A common misconception: noindex does not stop crawling. Google has to crawl the page to see the tag. If you want to stop Google requesting a URL pattern at all, use robots.txt. Never block a page in robots.txt that you're also trying to noindex, though: Google won't see the tag.&lt;br&gt;
Redirect chains&lt;br&gt;
A → B → C → D costs multiple requests for one page. Point internal links and redirects straight at the final URL.&lt;br&gt;
Soft 404s and infinite spaces&lt;br&gt;
Empty category pages returning 200, endless calendar pages, or internal search result pages that link to more search results. Return proper 404/410 status codes and stop linking to infinite paths.&lt;br&gt;
Slow or erroring servers&lt;br&gt;
If average response time is high or you see 5xx errors in Crawl Stats, Google will reduce its crawl rate. Caching, a CDN and fixing slow database queries often increase crawl volume without touching anything SEO-related.&lt;br&gt;
Stale sitemaps&lt;br&gt;
Sitemaps full of redirected, noindexed or 404 URLs, or lastmod dates that change on every build, teach Google to trust your sitemap less. Only list canonical, indexable 200 URLs and only update lastmod when content actually changes.&lt;br&gt;
A quick audit checklist&lt;br&gt;
Confirm the site is large or fast-changing enough for crawl budget to matter&lt;br&gt;
Review Crawl Stats: response time, host status, response codes, file types&lt;br&gt;
Review "Discovered – currently not indexed" volume and URL patterns&lt;br&gt;
Analyse 2–4 weeks of server logs with verified Googlebot only&lt;br&gt;
Measure the share of crawl going to parameter, faceted and low-value URLs&lt;br&gt;
Compare sitemap URLs, crawlable URLs and Googlebot-requested URLs&lt;br&gt;
Fix redirect chains, soft 404s and infinite URL spaces&lt;br&gt;
Block unimportant parameter patterns in robots.txt&lt;br&gt;
Improve server response times&lt;br&gt;
Clean sitemaps: canonical, indexable, 200-status URLs only&lt;br&gt;
Re-check Crawl Stats and index coverage after 4–8 weeks&lt;br&gt;
Wrapping up&lt;br&gt;
Crawl budget audits come down to one question: is Googlebot spending its time on the URLs you want indexed? Search Console tells you the symptoms, your logs tell you the cause, and most of the fixes are ordinary engineering: fewer junk URLs, faster responses and cleaner sitemaps.&lt;/p&gt;

&lt;p&gt;If you're working out where crawl and indexing issues sit in the bigger picture, I've written about where technical SEO fits alongside on-page and off-page work.&lt;/p&gt;

&lt;p&gt;Have you found a surprising crawl sink in your own logs? I'd like to hear about it in the comments.&lt;/p&gt;

</description>
      <category>python</category>
      <category>seo</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
