Most sites do not have an information hierarchy. They have a navigation menu, a sitemap a plugin generated three years ago, and a pile of pages whose relationships exist only in the head of whoever built it. If you are writing a crawler that has to emit a structured summary of a site — a table of contents, an llms.txt, a doc index — you cannot read the structure. You have to infer it.
I have written this crawler more than once now. Here is what turns out to matter.
Three structure signals, and how much to trust each
There are three sources of hierarchy on a normal website, and they disagree constantly.
The sitemap is the most complete and the least meaningful. sitemap.xml gives you every URL, but it is a flat bag, and <priority> is noise almost nobody sets deliberately. What it is good for is a denominator: if your crawl found 40 pages and the sitemap lists 900, your crawl is broken.
The nav is the most meaningful and the least complete. Whatever a human put in the header or sidebar is a curated statement of "these are the things that matter," and it gives you grouping for free — a <ul> under a heading in a sidebar is a section. The catch is that nav typically covers 5–20% of the URL space, and on marketing sites it is often pure funnel (Pricing, Book a Demo) rather than content structure.
URL path depth is the cheap fallback, and right more often than it deserves to be. /docs/api/auth/tokens really does usually mean tokens is under auth is under api. The failure mode is flat-by-design sites where everything is /p/<slug>.
My rule: nav is ground truth where it exists, URL path fills gaps, sitemap validates coverage. Never let path depth override an explicit nav grouping.
def infer_parent(url, nav_index, url_tree):
if url in nav_index: # human-curated, trust it
return nav_index[url].section
parts = urlparse(url).path.strip("/").split("/")
for depth in range(len(parts) - 1, 0, -1):
candidate = "/" + "/".join(parts[:depth])
if candidate in url_tree: # ancestor we actually crawled
return candidate
return ROOT # give up honestly rather than invent a parent
That last branch matters more than it looks. The temptation is to always produce a tidy tree, so you invent a parent from a URL segment that was never a real page — and now your output claims /blog/2024/ is a section when it 404s. A flat node attached to root is worse-looking and more correct.
SPA nav: don't render the whole site
A lot of nav is rendered client-side, so fetch() returns a shell with an empty <nav>. The obvious fix is to run everything through headless Chrome. That fix costs you roughly two orders of magnitude in throughput, and you do not need it for most pages.
The approach that has held up is a two-tier crawl with escalation. Fetch plain HTML first; escalate to a rendered fetch only when a heuristic says the static HTML is content-free:
def needs_render(html):
text = extract_visible_text(html)
if len(text) < 200: return True # shell, no content
if count_internal_links(html) == 0: return True # no nav at all
if has_root_mount(html) and len(text) < 1000:
return True # <div id="root"></div> plus a spinner
return False
Critically, you usually only need to render one page to recover the nav, because the nav is the same everywhere. I render the homepage, one deep page (to catch section-only sidebar nav), and anything that trips the heuristic. Three renders instead of nine hundred.
Also: check for an embedded state blob before reaching for a browser. Next.js dumps __NEXT_DATA__, Nuxt dumps __NUXT__, and plenty of frameworks inline their route manifest. Parsing JSON out of a script tag is free and gives cleaner structure than scraping rendered DOM.
Deduping near-identical URLs
This is where naive crawlers produce garbage. The same document is reachable as /docs/auth, /docs/auth/, /docs/auth/index.html, /docs/auth?utm_source=twitter, and /docs/auth#tokens — while /docs/auth?page=2 looks identical to all of those and is not a duplicate.
Cheap normalization handles the first tier: lowercase host, strip fragment and trailing slash, sort query params, drop a denylist of tracking params (utm_*, fbclid, gclid, ref). Do not strip all query params — you will collapse paginated and filtered pages carrying real distinct content.
Then honor what the site tells you. If <link rel="canonical"> points elsewhere, the owner is explicitly declaring the duplicate; follow it. Same for rel="alternate" hreflang — translations are language variants, not duplicates, and must never be flattened together.
For what survives, content-level dedup catches the rest. Full-text hashing fails on one differing footer timestamp, so hash the shape: normalized H1 + ordered H2s + rounded body length.
def structure_fingerprint(doc):
heads = [normalize(h) for h in doc.headings if h.level <= 2]
bucket = round(len(doc.text) / 500) # tolerate small diffs
return sha1("|".join(heads + [str(bucket)]))
Identical fingerprints mean pick one — prefer the shortest URL, then the one appearing in nav, then the sitemap. Deterministic tie-breaks matter: if the winner flips between runs, your diffs become useless.
The whole pipeline is roughly: sitemap for coverage → static fetch with selective render escalation → normalize and canonicalize → structure-fingerprint dedup → merge nav grouping over the path-derived tree → emit. When I finally wrapped this into something usable rather than a folder of scripts, it became aicrawlable.com, which crawls a URL and emits spec-shaped llms.txt out the other end — but the interesting part was always the inference, not the file format.
The part nobody warns you about
Hierarchy inference degrades badly and silently. A crawler that produced a beautiful nested tree last month produces a flat list this month because the site moved its nav into a client-rendered mega-menu. Nothing errors. It still exits 0.
So instrument the shape of your own output: tree depth, orphan ratio, dedup collapse rate, render escalation rate. Alert on deltas, not absolutes. A jump from 4% orphans to 60% is the signal that your nav extraction broke — and it is the only signal you get.
Top comments (0)