I run a small SEO tooling site, and on 2026-08-24 I pointed my crawler at it. The plan was routine: check internal health, check for orphans, move on.
The crawl returned zero orphan candidates. That result is why I'm writing this. It wasn't proof that the site was perfect; the audit had asked the wrong question. The broken links were a false alarm, and the green "everything is fine" result was the real bug.
What I ran, and what it actually measured
The setup was deliberately plain, so anyone can reproduce the shape of it:
- Seed URL:
https://rankforge.cc - Breadth-first traversal of internal links, respecting
robots.txt - Requested page cap: 100
- 1 second delay between requests
- Started 2026-08-24T09:04:12Z
Two details matter before the numbers. The requested cap was 100 but the script fetched 159 pages: after traversal, it performs bounded extra link checks. So "pages fetched" and "BFS budget" are different metrics.
The bigger issue: the orphan check fetched sitemap_index.xml and nothing else.
The numbers
Raw crawler output first, then a separate probe I ran immediately afterward. I'm keeping these visually separated on purpose, because mixing a tool's output with your own follow-up checks is how an audit quietly becomes fiction.
| Metric | Value | Source |
|---|---|---|
| Crawl start time | 2026-08-24T09:04:12Z | raw crawl |
| Pages fetched | 159 | raw crawl |
| Unique URLs queued | 158 | raw crawl |
| HTTP 200 | 147 | raw crawl |
| HTTP 404 | 12 | raw crawl |
| Total internal links recorded | 4,883 | raw crawl |
| Homepage internal links | 53 | raw crawl |
| Homepage external links | 2 | raw crawl |
| Broken-link entries (source → target pairs) | 22 | raw crawl |
| Unique broken targets | 12 | raw crawl |
| Sitemap URLs seen by crawler | 0 | raw crawl |
| Orphan candidates reported | 0 | raw crawl |
robots.txt |
HTTP 200 | direct probe, immediately after crawl |
sitemap_index.xml |
HTTP 404 | direct probe, immediately after crawl |
sitemap.xml |
HTTP 200, 142 <loc> entries |
direct probe, immediately after crawl |
robots.txt disallowed ten app prefixes, including /api/, /dashboard/, and /search-performance/. A robots-aware crawl correctly stayed out of them; compare that boundary with the sitemap before judging coverage.
The false alarm: 12 broken links that aren't broken links
Twelve 404s on a 159-page crawl looks like a real finding. It wasn't. Every single one was a Cloudflare email-obfuscation URL under /cdn-cgi/l/email-protection, linked from /contact, /pricing, /privacy, /terms, /security, and /docs/crawler. Cloudflare rewrites mailto links into that endpoint plus a hex-encoded fragment; browser JavaScript decodes it back into an address. A plain HTTP fetch of that path is supposed to fail. The site is behaving exactly as configured.
The 22-versus-12 gap is another small lesson: the crawler logs one entry per source→target pair, and ten entries recorded the source as ? because the target was found during the bounded extra-check phase. The same obfuscated addresses were counted from multiple pages. Reporting "22 broken links" would inflate a non-issue by nearly 2x.
The rule for any crawler: put infrastructure URLs in their own bucket. /cdn-cgi/*, analytics beacons, tracking pixels, and mailto:/tel: rewrites should be reported separately from content 404s. Dropping them hides regressions; mixing them into the headline trains you to ignore the report.
The real trap: sitemap_index.xml vs sitemap.xml
Here's the part that actually cost me something.
The orphan check works by diffing pages found in the sitemap against pages reached by crawling internal links. Anything in the sitemap but unreachable by link is an orphan candidate. Sound logic — and it depends entirely on having a sitemap to diff against.
My crawler only requested sitemap_index.xml. That's the WordPress/Yoast convention, and it's a reasonable default if you mostly audit WordPress sites. On my own site that URL is a 404. So the crawler recorded sitemap_url_count: 0, diffed 158 crawled URLs against an empty set, and reported zero orphan candidates.
Zero orphans was structurally impossible to avoid. The check couldn't have found anything.
The direct probe immediately afterward got https://rankforge.cc/sitemap.xml at HTTP 200 with 142 <loc> entries. There was a sitemap the whole time — the audit just never looked at it. Note carefully what this does and does not tell you: it does not mean the site has no sitemap, and the zero result does not mean the site has no orphans. It means the orphan check never ran. Those 142 URLs versus 158 crawled URLs is a diff I still owe myself, and the overlap is not something you can eyeball from the counts alone.
If your tool can't distinguish "I looked and found none" from "I couldn't look," every clean report is ambiguous.
The corrected workflow
1. crawl internal links from seed, robots-aware, record every URL + status
2. fetch robots.txt, parse ALL "Sitemap:" directives
3. if none: try /sitemap.xml AND /sitemap_index.xml AND /sitemap-index.xml
4. for each sitemap found:
if it is an index -> recurse into every child <loc>
else -> collect every <loc>
5. FAIL LOUDLY if total sitemap URL count == 0
(do not emit "0 orphans" — emit "sitemap not discovered")
6. normalize both sets: scheme, host, trailing slash, case,
strip fragments, decide on query params explicitly
7. diff: sitemap_urls - crawled_urls = orphan candidates
8. verify each candidate: fetch it, confirm 200, confirm no internal
link actually points to it, check it isn't robots-disallowed
Step 5 is the whole fix. An empty discovery set must be an error state, never an input to a diff. Step 6 is where most homegrown orphan checkers leak false positives — https://site.com/page and https://site.com/page/ are the same page to you and different strings to a set difference.
What 4,883 internal links actually tell me
Not much about quality, and I want to be honest about that. 4,883 internal links across 159 pages averages ~31 links per page, which mostly reflects a shared header, footer, and nav. The homepage's 53 internal links and 2 external links are the same story: that's template, not editorial linking.
Link count is a volume metric. It says nothing about anchor relevance, placement, or whether a page has inbound links beyond the global nav. The crawl flagged the homepage as the only page with zero inbound internal links — an artifact of it being the BFS seed, not a finding.
So the next passes are specific: strip nav/footer links and recount in-body links only; check which of the 142 sitemap URLs are reachable only through nav; and run the corrected sitemap diff before I make any claim about orphans at all.
Takeaway
The dramatic result (12 broken links) was noise; the boring result (0 orphans) was the bug. When an audit returns nothing, ask whether the check ran. Then use the orphan page checker, while confirming which sitemap was fetched, how many <loc> entries it returned, and whether 404s are content or infrastructure.
Ever shipped a "clean" audit that was actually blind, not clean? Drop it below.
Top comments (0)