You run a link checker, it comes back green, and you move on. The myth here is subtle: a clean report means "no broken links among the links I found", not "no broken links". The gap between those two is where users still hit dead ends.
Here are seven blind spots, why checkers miss them, and how to close each one.
1. Links that only exist after JavaScript runs
Most crawlers, including many CI link checkers, parse the raw HTML. If your nav, footer or "related articles" block is rendered client-side, those links don't exist in the HTML they read.
Test it: compare link counts in raw vs rendered HTML.
// count-links.mjs
import { chromium } from "playwright";
const url = process.argv[2];
const raw = await (await fetch(url)).text();
const rawCount = (raw.match(/<a\s[^>]*href=/gi) || []).length;
const b = await chromium.launch(); const p = await b.newPage();
await p.goto(url, { waitUntil: "networkidle" });
const rendered = await p.$$eval("a[href]", as => as.length);
console.log({ rawCount, rendered });
await b.close();
If rendered is much larger, use a checker that renders pages, or point a Playwright-based check at the important templates.
2. Links behind a login
Account pages, dashboards, customer docs and admin screens are invisible to an anonymous crawler, and they're often where the stalest links live ("see our help article on..." pointing at a help center you migrated two years ago).
Close it: run an authenticated crawl with a test account, or at least extract hardcoded URLs from templates and check them directly:
grep -rhoE 'https?://[^"'"'"' )>]+' src/ templates/ | sort -u > urls.txt
3. Links that aren't <a href>
Checkers usually follow anchors. They often skip:
-
<form action="...">: a contact form posting to a dead endpoint. -
<link rel="canonical">,hreflangand<link rel="alternate">. -
<img srcset>,<source>,<video poster>, CSSurl(). -
<meta http-equiv="refresh">and JavaScriptwindow.locationredirects. -
data-hrefattributes used by custom components.
Close it: check your tool's settings for "check resources/assets", and grep templates for the attributes it doesn't cover.
4. Links the checker was told to skip
Every mature config grows an exclusion list: "skip linkedin.com, it always 403s", "skip /api/, too slow", "skip external links in CI". Each exclusion was reasonable at the time. Together they can hide a lot.
Close it: review the skip list quarterly. Better still, split checks: block CI on internal links only, and run external links in a separate scheduled job that reports without blocking, so you don't need to exclude them.
5. Links that "work" but go somewhere wrong
The checker sees a 200 and reports success. But the link:
- redirects to the target site's homepage (the page you meant is gone),
- lands on a login page,
- hits a parked domain full of ads,
- serves a soft 404 ("We couldn't find that page", status 200).
Close it: flag redirects whose final URL is a homepage, or a different domain from the original.
from urllib.parse import urlparse
def suspicious(original, final):
o, f = urlparse(original), urlparse(final)
return (o.netloc != f.netloc) or (o.path not in ("", "/") and f.path in ("", "/"))
6. Pages the crawler never reached
A crawler finds pages by following links from a start URL. Orphan pages, which nothing links to, are never crawled, so their broken links are never checked. Landing pages from old campaigns are the classic case.
Close it: seed the crawl with your sitemap and your analytics' top landing pages, not just the homepage.
7. Links that break between checks
A weekly check is a snapshot. A vendor reorganizes their docs on Tuesday, your check runs Sunday, and for five days the link is broken while the last report says it's fine.
Close it: match frequency to importance. Check critical pages (pricing, signup, top docs) daily, the rest weekly. A scheduled crawler such as linkbruno.com handles the "run it repeatedly" part, and the cadence decision is still yours.
A better way to read a clean report
When the report says zero, ask:
- Did it render JavaScript?
- Did it see logged-in pages?
- Which link types and attributes did it check?
- What was on the skip list?
- Did it flag suspicious redirects, or only status codes?
- Where did the crawl start?
- When did it run?
If you can answer all seven, the green report means something. If not, it means "no broken links that this tool could see". That's still useful, as long as you know which one you've got.
If you only fix one thing this week
Start with blind spot #6. Export your top 50 landing pages from analytics, add your sitemap URLs, and run your existing checker against that combined list instead of starting from the homepage. It needs no new tooling, it usually turns up a batch of pages nobody has crawled in months, and those pages already get real visitors, so every fix there matters. Then put the seed list in version control so the next check starts from the same place. Once that's in place, tackle the JavaScript gap (#1) on your two or three most important templates, because nav and footer links are repeated on every page and one bad link there multiplies across the whole site.
Related reading
- HEAD vs GET for link checking: HEAD is faster, and it lies more than you'd think
- How to find broken links with Playwright (a test you can drop into CI)
- Should a broken link fail your CI build? Block internal links, warn on external ones
- How to find and fix broken images on a website
Which blind spot has bitten you? I'm betting on #4: the skip list nobody remembers writing. Tell me in the comments what's on yours.
Top comments (0)