DEV Community

appierun
appierun

Posted on

Measure Click Depth for Every Page in Your Sitemap with Python (Standard Library Only): BFS Crawl, Pages Deeper Than 3 Clicks, CSV Report

Click depth is the smallest number of link clicks it takes to get from your home page to a page. The home page is depth 0, anything in the main nav is depth 1, and a blog post that only appears on page 4 of the blog archive might be depth 5 or 6. Nobody plans for a page to end up six clicks deep. It happens when a blog grows one "Next page" link at a time, when docs get nested one level per release, or when the only link to an important page is a button that is not really a link.

This post builds a click depth auditor in Python using only the standard library (urllib, urllib.robotparser, html.parser, xml.etree, csv, concurrent.futures). It reads your sitemap or sitemap index, crawls your site breadth first from the home page, and records the shortest click path to every page. It flags sitemap URLs deeper than a threshold you choose (3 by default), marks pages that can only be reached by paging through an archive, and writes a CSV with the full path for each URL, plus a console summary by depth and by site section and an exit code you can use in CI.

Google Search Central's Help Google understand your ecommerce website structure, SEO Starter Guide and Pagination, incremental page loading, and their impact on Google Search (all three showing "Last updated 2025-12-10 UTC") were opened on 15 Nov 2026, and every Google statement below comes from those three pages. Appie free-tool descriptions are as of 15 Nov 2026, checked on the tool pages that day. The script was tested on Python 3.13.5 against a local fixture site with planted deep pages; the output below is real, from runs on 15 Nov 2026.

Disclosure: Appie.run publishes this article. Two free tools mentioned near the end are ours. The script does not depend on them.

Where this fits

This is another script in a small standard-library series. The previous one found orphan pages by comparing sitemap URLs with internal links: pages nothing links to at all. An earlier post found broken links and redirect chains. This one asks the next question for the pages that are linked: how far from the home page are they, which path gets there, and is that path a real navigation route or a long walk through archive pages?

What Google actually says (and what it does not)

The most direct statement is on the ecommerce site structure page, and it applies to any site with a lot of pages: "Google can use information such as the number of links it needs to follow to reach a page and the number of links to a page to infer the relative importance of a page over the rest of your site." That is click depth and inbound links, named together.

The same page says that structure comes from links, not from URLs: "Google generally doesn't look at the structure of URLs to work out the structure of a site. Instead, it analyzes the linkages between pages to gain insights about the relative importance of different pages on a site." So /docs/getting-started/install/proxy/sso is not deep because it has five folders in its path. It is deep only if it takes five clicks to reach. The script measures clicks and ignores folder depth.

It also gives the fix for an important page: "if you have a best selling product, consider linking to it from the home page or in other content, such as blog posts or newsletters on your site." And it warns about pages that can only be reached through a search box: "Googlebot generally doesn't try to submit searches into a search box as part of crawling a site."

The pagination page explains why archives push pages down. Its first best practice is to "include links from each page to the following page using <a href> tags," and it suggests "linking from all individual pages in a collection back to the first page of the collection." It also notes that Google's crawlers "don't 'click' buttons and generally don't trigger JavaScript functions that require user actions," which is why a "Load more" button can hide everything after the first batch. The page says Google "no longer uses" rel="next" and rel="prev" link tags, so the script ignores them and follows only <a href> links.

The SEO Starter Guide adds a reality check before you start moving pages around: "Don't drop everything and start reorganizing your site right now though: while these suggestions can be helpful long term (especially if you're working on a larger website), search engines will likely understand your pages as they are right now, regardless of how your site is organized." Its internal linking advice is "Every page you care about should have a link from at least one other page on your site," and "There's no magical ideal number of links a given page should contain."

What none of these pages give is a maximum click depth. "Three clicks" is a common rule of thumb in site architecture, not a Google number. The script uses 3 as a default because it is a reasonable line for "a visitor can get here from the nav without hunting," and the flag is --max-depth so you can set your own. Treat a deep page as a question to answer (does anyone need to reach this?), not as an error.

How the crawl works

  1. Sitemap URLs. It reads a urlset or a sitemap index (recursively, gzip supported) and keeps every <loc>. These are the pages the report is about.
  2. Breadth-first crawl. It fetches the home page (or any --start URLs), then every same-host page linked from it, then every page linked from those, one level at a time. Because each level is finished before the next one starts, the first time the crawl sees a URL is its shortest click distance. That is the main reason to use breadth first: a depth-first crawl would report whatever path it happened to take first.
  3. What counts as a link. Only <a href> links on the same host. Links with rel="nofollow" and links on pages whose robots meta says nofollow are not followed, but nofollow links are counted so the report can explain why a page was never reached. <base href> is honoured. robots.txt is respected. Links to files such as PDFs, images or .txt count as reached but are not fetched.
  4. Redirects. A link to a URL that redirects is credited to the final URL, at the depth of the link. A sitemap URL that redirects is reported as redirected and counted under its target.
  5. Shortest paths. After the crawl, every link from a page at depth d to a page at depth d + 1 is a shortest-path link. The script stores how many there are (shortest_paths_in) and prints one full path per URL, for example / > /blog > /blog/page/2 > /blog/post-7.
  6. Pagination. A URL matching /page/N or a page=, p=, pg= or paged= query parameter is treated as a paginated listing. A page is only_via_pagination when every one of its shortest paths runs through one. Those are the pages a visitor only finds by clicking "Next" again and again.
  7. Matching. URLs are compared after removing fragments, lowercasing scheme and host, dropping default ports and ignoring a trailing slash. --drop-query also ignores query strings.

What the script flags

  • Flag: deeper_than_N; Meaning: Shortest path from the start page is more than --max-depth clicks; What to check: Does this page matter enough to link from a hub, a category page or the nav?
  • Flag: only_via_pagination; Meaning: Every shortest path goes through page 2, 3, ... of a listing; What to check: Add numbered page links, category or topic hubs, or related-post links
  • Flag: not_reached; Meaning: In the sitemap but never reached from the start page; What to check: Run an orphan check; the nofollow_inlinks column shows whether nofollow links are the reason
  • Flag: redirected; Meaning: The sitemap URL redirects; What to check: List the final URL in the sitemap instead
  • Flag: http_error; Meaning: The page returned something other than 200; What to check: Fix it or remove it from the sitemap
  • Flag: not_in_sitemap; Meaning: A crawled HTML page the sitemap does not list (CSV only); What to check: Informational; paginated archive pages often land here

Exit codes: 0 when no sitemap URL has an issue other than redirected, 1 when at least one is too deep, only reachable through pagination, unreached or erroring, and 2 when the sitemap cannot be read.

The script

Save as click_depth.py:

#!/usr/bin/env python3
"""Measure click depth from the home page for every URL in a sitemap.

Standard library only. Usage:
    python3 click_depth.py https://example.com/sitemap.xml --max-depth 3 --out depth.csv
Exit codes: 0 = nothing deeper than --max-depth, 1 = deep or unreached pages, 2 = sitemap unreadable.
"""
import argparse
import csv
import gzip
import re
import sys
import time
import urllib.error
import urllib.request
import urllib.robotparser
import xml.etree.ElementTree as ET
from collections import Counter, defaultdict
from concurrent.futures import ThreadPoolExecutor
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit, urlunsplit

UA = "click-depth-audit/1.0 (+python stdlib)"
SKIP_EXT = (".pdf", ".jpg", ".jpeg", ".png", ".gif", ".webp", ".svg", ".zip",
            ".mp4", ".mp3", ".css", ".js", ".xml", ".txt", ".json", ".ico")
PAGINATION = re.compile(r"(/page/\d+/?$)|([?&](page|p|pg|paged)=\d+)", re.I)


def norm(url, drop_query=False):
    """Comparable form of a URL: no fragment, lowercase host, no default port,
    no trailing slash (except the root)."""
    s = urlsplit(url.strip())
    scheme, host = s.scheme.lower(), (s.hostname or "").lower()
    if s.port and not ((scheme == "http" and s.port == 80) or (scheme == "https" and s.port == 443)):
        host = f"{host}:{s.port}"
    path = s.path or "/"
    if len(path) > 1:
        path = path.rstrip("/")
    return urlunsplit((scheme, host, path, "" if drop_query else s.query, ""))


def fetch(url, timeout=15):
    """Return (status, final_url, content_type, body_bytes). Follows redirects."""
    req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept-Encoding": "gzip"})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            body = r.read()
            if r.headers.get("Content-Encoding") == "gzip":
                body = gzip.decompress(body)
            return r.status, r.geturl(), r.headers.get("Content-Type", ""), body
    except urllib.error.HTTPError as e:
        return e.code, e.geturl() or url, "", b""
    except Exception as e:  # DNS, timeout, TLS
        return None, url, str(e), b""


def read_sitemap(url, seen=None):
    """All <loc> values from a urlset or sitemap index (recursive, .gz ok)."""
    seen = seen if seen is not None else set()
    if url in seen:
        return []
    seen.add(url)
    status, _, _, body = fetch(url)
    if status != 200:
        raise ValueError(f"{url} returned {status}")
    if body[:2] == b"\x1f\x8b":
        body = gzip.decompress(body)
    root = ET.fromstring(body)
    locs = [el.text.strip() for el in root.iter() if el.tag.endswith("loc") and el.text]
    if root.tag.endswith("sitemapindex"):
        out = []
        for child in locs:
            out.extend(read_sitemap(child, seen))
        return out
    return locs


class LinkParser(HTMLParser):
    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.links, self.nofollow_page, self.base = [], False, None

    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag == "base" and a.get("href"):
            self.base = a["href"]
        elif tag == "meta" and a.get("name", "").lower() in ("robots", "googlebot"):
            if "nofollow" in a.get("content", "").lower() or "none" in a.get("content", "").lower():
                self.nofollow_page = True
        elif tag == "a" and a.get("href"):
            rel = a.get("rel", "").lower().split()
            self.links.append((a["href"], "nofollow" in rel))


def crawl(starts, host, args, robots):
    """Breadth-first crawl, one depth level at a time, so the first time a page
    is seen is its shortest click distance from the start page(s)."""
    depth, parents, edges, info = {}, defaultdict(set), [], {}
    nofollow_in = Counter()
    alias = {}  # normalized requested URL -> normalized final URL (redirects)
    frontier = []
    for s in starts:
        k = norm(s, args.drop_query)
        depth[k] = 0
        frontier.append(k)
    level = 0
    blocked = set()

    def visit(key):
        status, final, ctype, body = fetch(key)
        return key, status, final, ctype, body

    with ThreadPoolExecutor(max_workers=args.workers) as pool:
        while frontier and level <= args.max_crawl_depth and len(info) < args.limit:
            batch = frontier[: args.limit - len(info)]
            nxt = []
            for key, status, final, ctype, body in pool.map(visit, batch):
                fkey = norm(final, args.drop_query)
                if fkey != key:
                    alias[key] = fkey
                    if fkey in info:            # redirect target already crawled
                        continue
                    depth[fkey] = min(depth.get(fkey, level), level)
                info[fkey] = {"status": status, "html": "html" in ctype.lower()}
                if status != 200 or "html" not in ctype.lower():
                    continue
                p = LinkParser()
                p.feed(body.decode("utf-8", "replace"))
                if p.nofollow_page:
                    continue
                base = urljoin(final, p.base) if p.base else final
                for href, nofollow in p.links:
                    if href.startswith(("mailto:", "tel:", "javascript:", "#")):
                        continue
                    target = urljoin(base, href)
                    t = urlsplit(target)
                    if t.scheme not in ("http", "https") or (t.hostname or "").lower() != host:
                        continue
                    tkey = norm(target, args.drop_query)
                    tkey = alias.get(tkey, tkey)
                    if nofollow:
                        nofollow_in[tkey] += 1
                        continue
                    edges.append((fkey, tkey))
                    if tkey in depth:
                        continue
                    if robots and not robots.can_fetch(UA, target):
                        blocked.add(tkey)
                        continue
                    depth[tkey] = level + 1
                    if t.path.lower().endswith(SKIP_EXT):
                        info[tkey] = {"status": None, "html": False}
                        continue
                    nxt.append(tkey)
            if args.delay:
                time.sleep(args.delay)
            frontier, level = sorted(set(nxt)), level + 1
    # Redirect sources share the depth of their target; resolve edges through aliases.
    edges = [(alias.get(a, a), alias.get(b, b)) for a, b in edges]
    for a, b in edges:
        if a != b and a in depth and b in depth and depth[a] + 1 == depth[b]:
            parents[b].add(a)
    return depth, parents, edges, info, alias, blocked, nofollow_in


def is_pagination(url):
    return bool(PAGINATION.search(url))


def build_report(sitemap, depth, parents, edges, info, alias, nofollow_in, args):
    inlinks = Counter(b for a, b in set(edges) if a != b)
    # A page is "only via pagination" when every shortest path to it goes
    # through a paginated listing (page 2, 3, ...).
    only_pag = {}
    for key in sorted(depth, key=lambda k: depth[k]):
        ps = parents.get(key)
        if not ps:
            only_pag[key] = False
            continue
        only_pag[key] = all(is_pagination(p) or only_pag.get(p, False) for p in ps)

    def path_of(key):
        chain, cur, guard = [key], key, 0
        while parents.get(cur) and guard < 100:
            cur = sorted(parents[cur])[0]
            chain.append(cur)
            guard += 1
        return " > ".join(urlsplit(c).path or "/" for c in reversed(chain))

    in_map = {norm(u, args.drop_query) for u in sitemap}
    rows = []
    keys = list(dict.fromkeys([norm(u, args.drop_query) for u in sitemap] +
                              [k for k, v in info.items() if v["html"] and v["status"] == 200]))
    for key in keys:
        final = alias.get(key, key)
        d = depth.get(final)
        st = info.get(final, {}).get("status")
        issues = []
        if final != key:
            issues.append("redirected")
        if d is None:
            issues.append("not_reached")
        elif d > args.max_depth:
            issues.append(f"deeper_than_{args.max_depth}")
        if d is not None and only_pag.get(final):
            issues.append("only_via_pagination")
        if st is not None and st != 200:
            issues.append("http_error")
        if key not in in_map:
            issues.append("not_in_sitemap")
        rows.append({
            "url": key,
            "in_sitemap": "yes" if key in in_map else "no",
            "status": st if st is not None else "",
            "depth": "" if d is None else d,
            "shortest_paths_in": len(parents.get(final, ())),
            "inlinks": inlinks.get(final, 0),
            "nofollow_inlinks": nofollow_in.get(final, 0),
            "section": "/" + (urlsplit(key).path.strip("/").split("/")[0] if urlsplit(key).path.strip("/") else ""),
            "path": path_of(final) if d is not None else "",
            "issues": ";".join(issues),
        })
    rows.sort(key=lambda r: (r["depth"] == "", -(r["depth"] or 0) if r["depth"] != "" else 0, r["url"]))
    return rows


def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("sitemap")
    ap.add_argument("--start", action="append", help="start URL(s); default is the site root")
    ap.add_argument("--max-depth", type=int, default=3, help="flag pages deeper than this (default 3)")
    ap.add_argument("--max-crawl-depth", type=int, default=25, help="stop crawling past this level")
    ap.add_argument("--limit", type=int, default=5000, help="max pages to fetch")
    ap.add_argument("--workers", type=int, default=4)
    ap.add_argument("--delay", type=float, default=0.0, help="seconds to pause between levels")
    ap.add_argument("--drop-query", action="store_true", help="treat ?a=1 and ?a=2 as one page")
    ap.add_argument("--no-robots", action="store_true", help="ignore robots.txt (your own staging only)")
    ap.add_argument("--out", default="click_depth.csv")
    args = ap.parse_args()

    try:
        sitemap = read_sitemap(args.sitemap)
    except Exception as e:
        print(f"Could not read sitemap: {e}")
        return 2
    s = urlsplit(args.sitemap)
    root = f"{s.scheme}://{s.netloc}/"
    host = (s.hostname or "").lower()
    starts = args.start or [root]
    robots = None
    if not args.no_robots:
        robots = urllib.robotparser.RobotFileParser(urljoin(root, "/robots.txt"))
        try:
            robots.read()
        except Exception:
            robots = None

    print(f"Sitemap: {len(sitemap)} URLs from {args.sitemap}")
    depth, parents, edges, info, alias, blocked, nofollow_in = crawl(starts, host, args, robots)
    rows = build_report(sitemap, depth, parents, edges, info, alias, nofollow_in, args)
    fetched = sum(1 for v in info.values() if v["status"] is not None)
    print(f"Crawled {fetched} pages breadth first from {', '.join(starts)}")
    if blocked:
        print(f"Skipped {len(blocked)} URLs disallowed by robots.txt")

    in_map = [r for r in rows if r["in_sitemap"] == "yes"]
    # Redirecting sitemap URLs are reported, but counted under their target.
    direct = [r for r in in_map if "redirected" not in r["issues"]]
    hist = Counter(r["depth"] for r in direct if r["depth"] != "")
    print("\nClick depth of sitemap URLs:")
    for d in sorted(hist):
        flag = "  <- deeper than %d" % args.max_depth if d > args.max_depth else ""
        print(f"  depth {d:>2}: {hist[d]:>4} {'#' * min(hist[d], 40)}{flag}")
    unreached = sum(1 for r in direct if r["depth"] == "")
    if unreached:
        print(f"  not reached: {unreached:>2}")

    sections = defaultdict(list)
    for r in direct:
        if r["depth"] != "":
            sections[r["section"]].append(r["depth"])
    print("\nBy section (sitemap URLs reached):")
    w = max([len(x) for x in sections] + [7]) + 2
    print(f"  {'section':<{w}}{'pages':>6}{'max':>5}{'deep':>6}")
    for sec in sorted(sections, key=lambda x: (-max(sections[x]), x)):
        ds = sections[sec]
        print(f"  {sec:<{w}}{len(ds):>6}{max(ds):>5}{sum(d > args.max_depth for d in ds):>6}")

    flagged = [r for r in in_map if r["issues"]]
    print(f"\nFlagged sitemap URLs ({len(flagged)}), deepest first:")
    for r in flagged:
        extra = f"  ({r['nofollow_inlinks']} nofollow links)" if r["nofollow_inlinks"] and r["depth"] == "" else ""
        print(f"  {str(r['depth']) or '-':>2}  {r['issues']:<34} {urlsplit(r['url']).path or '/'}{extra}")
    extra_pages = sum(1 for r in rows if r["in_sitemap"] == "no")
    if extra_pages:
        print(f"  (+{extra_pages} crawled pages not in the sitemap; see the CSV)")

    with open(args.out, "w", newline="", encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=list(rows[0].keys()) if rows else ["url"])
        w.writeheader()
        w.writerows(rows)
    print(f"\nReport: {args.out}")
    benign = {"", "redirected", "not_in_sitemap"}
    bad = any(set(r["issues"].split(";")) - benign for r in in_map)
    return 1 if bad else 0


if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode

Testing it against a local fixture

I built a small site for a made-up field data app, "Fieldnote," with a sitemap index pointing at two child sitemaps (35 URLs). Every page shares a header nav (Home, Features, Pricing, Blog, Docs, About) and a footer with a Privacy link and a Careers link marked rel="nofollow". The planted problems:

  • The blog has 12 posts, three per listing page, and the listing pages only link to the next page (/blog to /blog/page/2 to /blog/page/3 to /blog/page/4). The exception is /blog/post-1, which the features page also links to.
  • The docs nest one level per page: /docs to /docs/getting-started to /install to /install/proxy to /install/proxy/sso.
  • /pricing/enterprise is linked from the pricing page only through <a onclick="go('/pricing/enterprise')">, with no href. The only real link to it is at the bottom of the SSO docs page.
  • Case studies: the home page features /customers/acme, the customers index is only linked from the about page, and /customers/initech is only linked from the Globex case study.
  • /old-integrations is in the sitemap and returns a 301 to /integrations; the home page links to the old URL.
  • /careers is only linked through the nofollow footer link, and /webinar-2025 is in the sitemap with no links to it.
  • robots.txt disallows /admin/, which the home page links to.

The fixture is generated by a short script, and served with a handler that maps clean URLs to .html files and answers the one redirect:

"""Serve ./site with clean URLs (/pricing -> pricing.html) and one redirect."""
import functools
import http.server
import os

REDIRECTS = {"/old-integrations": "/integrations"}


class Handler(http.server.SimpleHTTPRequestHandler):
    def do_GET(self):
        path = self.path.split("?")[0].split("#")[0]
        if path in REDIRECTS:
            self.send_response(301)
            self.send_header("Location", REDIRECTS[path])
            self.end_headers()
            return
        path = path.rstrip("/") or "/"
        if os.path.isdir("site" + path):
            path = path.rstrip("/") + "/index.html"
        elif not os.path.splitext(path)[1]:
            path += ".html"
        self.path = path
        super().do_GET()

    def log_message(self, *args):
        pass


http.server.ThreadingHTTPServer(
    ("127.0.0.1", 8794),
    functools.partial(Handler, directory="site")).serve_forever()
Enter fullscreen mode Exit fullscreen mode

Run the server, then from another terminal:

python3 click_depth.py http://127.0.0.1:8794/sitemap.xml --out click_depth.csv
echo "exit code: $?"
Enter fullscreen mode Exit fullscreen mode

Real output on Python 3.13.5:

Sitemap: 35 URLs from http://127.0.0.1:8794/sitemap.xml
Crawled 35 pages breadth first from http://127.0.0.1:8794/
Skipped 1 URLs disallowed by robots.txt

Click depth of sitemap URLs:
  depth  0:    1 #
  depth  1:    8 ########
  depth  2:    9 #########
  depth  3:    5 #####
  depth  4:    5 #####  <- deeper than 3
  depth  5:    3 ###  <- deeper than 3
  depth  6:    1 #  <- deeper than 3
  not reached:  2

By section (sitemap URLs reached):
  section         pages  max  deep
  /pricing            2    6     1
  /blog              13    5     5
  /docs               6    5     2
  /customers          4    4     1
  /integrations       3    2     0
  /about              1    1     0
  /features           1    1     0
  /privacy            1    1     0
  /                   1    0     0

Flagged sitemap URLs (15), deepest first:
   6  deeper_than_3                      /pricing/enterprise
   5  deeper_than_3;only_via_pagination  /blog/post-2
   5  deeper_than_3;only_via_pagination  /blog/post-3
   5  deeper_than_3                      /docs/getting-started/install/proxy/sso
   4  deeper_than_3;only_via_pagination  /blog/post-4
   4  deeper_than_3;only_via_pagination  /blog/post-5
   4  deeper_than_3;only_via_pagination  /blog/post-6
   4  deeper_than_3                      /customers/initech
   4  deeper_than_3                      /docs/getting-started/install/proxy
   3  only_via_pagination                /blog/post-7
   3  only_via_pagination                /blog/post-8
   3  only_via_pagination                /blog/post-9
   1  redirected                         /old-integrations
   -  not_reached                        /careers  (35 nofollow links)
   -  not_reached                        /webinar-2025
  (+3 crawled pages not in the sitemap; see the CSV)

Report: click_depth.csv
exit code: 1
Enter fullscreen mode Exit fullscreen mode

The CSV has 38 rows: the 35 sitemap URLs plus three crawled archive pages the sitemap does not list. These are the first ten rows and the last two, sorted deepest first:

url,in_sitemap,status,depth,shortest_paths_in,inlinks,nofollow_inlinks,section,path,issues
http://127.0.0.1:8794/pricing/enterprise,yes,200,6,1,1,0,/pricing,/ > /docs > /docs/getting-started > /docs/getting-started/install > /docs/getting-started/install/proxy > /docs/getting-started/install/proxy/sso > /pricing/enterprise,deeper_than_3
http://127.0.0.1:8794/blog/post-2,yes,200,5,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/page/4 > /blog/post-2,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/blog/post-3,yes,200,5,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/page/4 > /blog/post-3,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/docs/getting-started/install/proxy/sso,yes,200,5,1,1,0,/docs,/ > /docs > /docs/getting-started > /docs/getting-started/install > /docs/getting-started/install/proxy > /docs/getting-started/install/proxy/sso,deeper_than_3
http://127.0.0.1:8794/blog/page/4,no,200,4,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/page/4,deeper_than_3;only_via_pagination;not_in_sitemap
http://127.0.0.1:8794/blog/post-4,yes,200,4,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/post-4,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/blog/post-5,yes,200,4,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/post-5,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/blog/post-6,yes,200,4,1,1,0,/blog,/ > /blog > /blog/page/2 > /blog/page/3 > /blog/post-6,deeper_than_3;only_via_pagination
http://127.0.0.1:8794/customers/initech,yes,200,4,1,1,0,/customers,/ > /about > /customers > /customers/globex > /customers/initech,deeper_than_3
http://127.0.0.1:8794/docs/getting-started/install/proxy,yes,200,4,1,1,0,/docs,/ > /docs > /docs/getting-started > /docs/getting-started/install > /docs/getting-started/install/proxy,deeper_than_3
...
http://127.0.0.1:8794/careers,yes,,,0,0,35,/careers,,not_reached
http://127.0.0.1:8794/webinar-2025,yes,,,0,0,0,/webinar-2025,,not_reached
Enter fullscreen mode Exit fullscreen mode

What the run shows:

  • The deepest page is a sales page, not a docs page. /pricing/enterprise sits one click below the pricing page for a visitor who clicks the "Talk to sales" button, but that button is an onclick handler with no href, so the crawl does not count it. The only real link is at the bottom of a docs page five clicks down, which makes the enterprise plan six clicks from the home page. The path column shows the route the crawler had to take.
  • Eight of the 12 blog posts are only_via_pagination. Posts 10 to 12 are on the first listing page (depth 2), post 1 is linked from the features page (depth 2), and everything else needs one more "Next" click per three posts. Posts 2 and 3 are five clicks deep.
  • The docs chain adds one click per level. Four of the six docs URLs are fine; the proxy and SSO pages are at 4 and 5.
  • The case studies are uneven. Acme is one click away because the home page features it. Initech is four clicks away because the only link to it is inside another case study.
  • /old-integrations is reported once, as redirected, and is not counted in the depth histogram, because the crawl credits that link to /integrations.
  • /careers was never reached, and the CSV shows why: 35 nofollow links and no followed ones. /webinar-2025 has no links of any kind. Both need the orphan check from the previous post, not a depth fix.
  • /admin/ was skipped because robots.txt disallows it.

After fixing the fixture

I changed three things in the fixture: every blog listing page now links to every other listing page (numbered pages instead of only "Next"), the pricing page links to /pricing/enterprise with a real <a href>, and the getting-started docs page links directly to the proxy and SSO pages. Same command, real output:

Sitemap: 35 URLs from http://127.0.0.1:8794/sitemap.xml
Crawled 35 pages breadth first from http://127.0.0.1:8794/
Skipped 1 URLs disallowed by robots.txt

Click depth of sitemap URLs:
  depth  0:    1 #
  depth  1:    8 ########
  depth  2:   10 ##########
  depth  3:   12 ############
  depth  4:    1 #  <- deeper than 3
  not reached:  2

By section (sitemap URLs reached):
  section         pages  max  deep
  /customers          4    4     1
  /blog              13    3     0
  /docs               6    3     0
  /integrations       3    2     0
  /pricing            2    2     0
  /about              1    1     0
  /features           1    1     0
  /privacy            1    1     0
  /                   1    0     0

Flagged sitemap URLs (12), deepest first:
   4  deeper_than_3                      /customers/initech
   3  only_via_pagination                /blog/post-2
   3  only_via_pagination                /blog/post-3
   3  only_via_pagination                /blog/post-4
   3  only_via_pagination                /blog/post-5
   3  only_via_pagination                /blog/post-6
   3  only_via_pagination                /blog/post-7
   3  only_via_pagination                /blog/post-8
   3  only_via_pagination                /blog/post-9
   1  redirected                         /old-integrations
   -  not_reached                        /careers  (35 nofollow links)
   -  not_reached                        /webinar-2025
  (+3 crawled pages not in the sitemap; see the CSV)

Report: click_depth_after.csv
exit code: 1
Enter fullscreen mode Exit fullscreen mode

Every docs page is now within three clicks, the enterprise page dropped from 6 to 2, and posts 2 and 3 went from 5 to 3. The blog posts are still flagged only_via_pagination, which is correct: numbered page links shorten the walk, but a reader still has to page through the archive to find post 2. Topic hubs, "related posts" links, or a link from a relevant product page would give those posts a second route. Initech is still at 4 because I left it alone.

When the sitemap URL is wrong, the script stops early:

Could not read sitemap: http://127.0.0.1:8794/missing.xml returned 404
exit code: 2
Enter fullscreen mode Exit fullscreen mode

Running it on your own sitemap

Point it at your sitemap. On a large site, keep the worker count low, add a delay between levels, and cap the crawl on a first pass:

python3 click_depth.py https://example.com/sitemap.xml --workers 2 --delay 0.5 --limit 2000 --out depth.csv
Enter fullscreen mode Exit fullscreen mode

Some options worth knowing:

  • --max-depth 4 changes the flag line. Pick the number for your site and keep it the same between runs so the CSVs are comparable.
  • --start https://example.com/blog/ measures depth from a section hub instead of the home page. Repeat --start to begin from several pages at once (for example, home plus the docs index if docs live on the same host and most visitors land there).
  • --drop-query treats ?utm_source=... and sort or filter parameters as the same page. Leave it off if your pagination uses ?page=, or the archive pages collapse into one URL.
  • --max-crawl-depth (default 25) stops the crawl from following an endless calendar or filter trail forever.
  • --no-robots is for your own staging site only. On a live site, let robots.txt decide.

If the crawl reaches far fewer pages than your sitemap lists, check whether your navigation is rendered by JavaScript. The script reads only the HTML your server returns and does not run JavaScript. A site that builds its menu in the browser will look much deeper (or unreachable) to this script than it does to a person.

If you want a quick look before running a crawl, two free tools on our site can help. Sitemap Analytics and Insights reads a urlset (pasted or by URL) and counts path depth, file extensions, http versus https and query strings. That is URL folder depth, not click depth, and as the Google page says, folder structure is not how Google works out site structure, so use it to spot over-nested URL patterns and use the script for clicks. The Website URL Extractor collects same-host links from up to 80 HTML pages or until 12 seconds pass, and skips a host whose robots.txt disallows all crawling, which is enough to see which URLs a small site actually links to. Both ask for a captcha before each run; the script has no page cap and gives you the path for every URL.

Fixing what it finds

Work from the CSV, sorted by depth:

  1. Decide which deep pages matter. A five-click changelog entry from 2022 may be fine. A five-click pricing or signup page usually is not. Mark the pages that should be close to the home page before changing anything.
  2. Turn button "links" into real links. Any deeper_than_N page that people reach in a browser through a button, a JavaScript menu or a search box is a candidate. Google's pages are clear that crawlers follow <a href> and do not click buttons or submit searches.
  3. Shorten archives. Link every listing page to the others (or at least to the first page and a few around the current one), as the pagination page recommends for linking back to the first page of a collection.
  4. Add hubs, not more nav items. Topic or category pages that link to every post on a subject give old posts a route that does not depend on publish date. A "related posts" block does the same at the post level.
  5. Flatten long chains. If a docs section nests five levels deep, link the section index to every page in it.
  6. Link the pages you want to promote from the home page. That is the ecommerce structure page's own example for a best seller, and it works for a flagship case study or an integration page too.
  7. Rerun and compare. Keep the CSV from each run. A page that jumps from depth 2 to depth 5 after a redesign is usually a nav item someone removed.

Extending it

  • Track depth over time. Save the CSV per deploy and diff the depth column. Fail CI only when a page that was within the threshold moves past it, so existing deep pages do not block every build.
  • Join with analytics or Search Console. Export clicks per page, join on url, and sort deep pages by traffic. A deep page with traffic is worth a better route; a deep page with none may be a candidate to merge or retire.
  • Count JavaScript-only anchors. Add a counter in LinkParser for <a> tags that have onclick or routerLink and no href (the orphan post does this), and print which pages have them. In the fixture, that would have pointed straight at the pricing page.
  • Weight by section. Use a different --max-depth per section by running the script with --start set to each section hub, or by post-processing the CSV by the section column.

FAQ

What is a good maximum click depth?
Google's pages do not give one. They say Google can use "the number of links it needs to follow to reach a page" to infer importance, which means shallower pages look more important relative to the rest of the site. Three clicks is a common rule of thumb; choose the number that fits your site's size and keep it fixed between runs.

Is click depth the same as URL depth?
No. /a/b/c/d can be one click from the home page, and /pricing/enterprise was six clicks away in the fixture. Google's ecommerce page says it "generally doesn't look at the structure of URLs to work out the structure of a site." The script only measures clicks.

Why breadth first instead of following links as they come?
Breadth first finishes every page at depth 1 before looking at depth 2, so the first time a page is found is its shortest distance. A depth-first crawl can reach a page through a long route first and report the wrong number.

Does a sitemap fix deep pages?
A sitemap helps a crawler find URLs, and Google's ecommerce page recommends one when "it's not possible to link to all pages." It does not change how many clicks a person needs, and it does not add links between your pages. The script reads the sitemap only to know which URLs to report on.

Can I run it in CI?
Yes. It exits 1 when a sitemap URL is too deep, only reachable through pagination, unreached or erroring, 0 when clean and 2 when the sitemap cannot be read. Run it against a staging build with --start pointed at the staging home page.

Wrap-up

Click depth is a site structure signal you can measure directly, and Google names it in its own documentation. A breadth-first crawl from the home page gives you the shortest path to every page in your sitemap, and the path column shows exactly where a route gets long: an archive that only has "Next" links, a docs tree that adds a level per release, a sales page behind a button. Fix the pages that matter, link them from hubs or the home page, and rerun the script after every redesign so a removed nav item shows up as a depth jump before anyone notices the traffic.

RG, Founder, Appie.run

Top comments (0)