Louis Wright

Posted on Oct 16

23% of News Pages Have a Broken Link. Catch Yours Before Deploy

#webdev #ai #javascript #devops

This article was written with AI assistance and reviewed by the author.

In May 2024, Pew Research Center reported that 23% of news webpages in its sample contained at least one broken link, and 21% of government webpages did too [1]. Across a decade of Common Crawl snapshots, a quarter of pages that existed between 2013 and 2023 were already inaccessible by October 2023 [1]. Link rot is not a niche SEO footnote. It is a measurable property of the public web.

Yet many static sites still discover 404s the hard way: after deploy, after a CDN purge, after a user files a ticket. Source review and local preview often miss the problems that only appear in the built output on a case-sensitive host.

That is the gap static-link-sentinel is built for. It crawls dist/, build/, out/, public/, or _site/ before you ship, maps URLs the way Netlify, Vercel, GitHub Pages, and Cloudflare Pages do, and fails CI when links, assets, anchors, or sitemap entries do not resolve.

The build is not the source tree

A Markdown link that looks fine in the editor can still point at a page the bundler never emitted. An image path that works on a default macOS volume can 404 on Linux. Amazon S3 documents that object key names are case sensitive: Logo.png and logo.png are different objects [2]. Local APFS defaults often hide that mismatch. Production hosts and object stores do not.

Google's crawl guidance is equally blunt about the cost of pretending missing pages still exist. Soft 404s (error-like or empty pages that still return HTTP 200) keep getting crawled and waste crawl budget; Google recommends returning a real 404 or 410 for permanently removed content and avoiding long redirect chains [3][4]. Broken internal links and stale sitemap entries feed exactly that mess.

So the useful check is not "does this string look like a URL in the repo?" It is "given the files that will actually be served, does every reference resolve the way the host will resolve it?"

What static-link-sentinel actually checks

static-link-sentinel is a zero-dependency CLI and library for Node 18+. Point it at a built directory (or let it auto-detect common output folders). It walks HTML, CSS, web manifests, sitemap*.xml, and robots.txt, plus optional Markdown/MDX sources.

Internal findings include:

broken-link and missing-asset: pages, scripts, stylesheets, images, and related refs with no matching file
case-mismatch: a path that only matches when letter case is ignored (works on many Mac volumes, fails on Linux/CDN origins)
missing-fragment: #anchors with no matching id or name
sitemap-url / robots-sitemap: sitemap or robots entries that are not in the build
mixed-content and related HTTPS warnings when a site URL is known
optional orphan-page reporting with --orphans

With --external, unique outbound URLs are checked (HEAD with GET fallback), including redirect chains hop by hop, with caching for CI reruns.

URL resolution understands clean URLs, index.html, trailing-slash policy, basePath, Netlify _redirects, and vercel.json redirects/rewrites/cleanUrls. Exit codes are 0 (pass), 1 (findings at or above --fail-on), and 2 (usage error). Output can be pretty CLI, JSON, SARIF 2.1.0, or Markdown. Vite and webpack plugins and a GitHub Action are included.

None of that invents a live server. The tool maps paths to files the same way those hosts do.

A labelled demo on a tiny dist

The following is example/demo data, not a production site: a three-page build with intentional typos, a case-mismatched image path, a missing CSS background, a bad fragment, and a sitemap URL for a page that was never built.

npx static-link-sentinel dist --site-url https://demo.example

Authentic output from that demo run:

static-link-sentinel: dist/ 3 page(s), 1 stylesheet(s), 7 file(s), 15 internal reference(s)

about.html
  ✖ error Broken internal link [broken-link]  line 12
          <a href="/pricing/"> resolves to no file in the build
          fix: Remove the link or create the page.

assets/style.css
  ✖ error Missing asset [missing-asset]  line 2
          ../img/bg.png resolves to no file in the build
          fix: Check the build output and asset paths (publicPath/base).

index.html
  ✖ error Broken internal link [broken-link]  line 16
          <a href="/blog/post-l/"> resolves to no file in the build
          fix: did you mean /blog/post-1/?
  ✖ error Path case mismatch [case-mismatch]  line 18
          <a href="/Blog/Post-1/"> only matches blog/post-1/index.html with different case (fails on Linux/CDN hosts)
          fix: Use /blog/post-1/ or rename the file
  ✖ error Path case mismatch [case-mismatch]  line 20
          <img src="/img/Logo.png"> only matches img/logo.png with different case (fails on Linux/CDN hosts)
          fix: Use /img/logo.png or rename the file
  ✖ error Missing asset [missing-asset]  line 22
          <img src="/img/hero-missing.png"> resolves to no file in the build
          fix: Check the build output and asset paths (publicPath/base).
  ⚠ warn  Missing #fragment anchor [missing-fragment]  line 24
          <a href="/about#tem">: no id="tem" in about.html
          fix: did you mean #team?

sitemap.xml
  ✖ error sitemap.xml URL not in build [sitemap-url]  line 6
          <loc>https://demo.example/gone/</loc> has no matching page in the build
          fix: Regenerate the sitemap or remove the entry

3 page(s) checked: 7 error(s), 1 warning(s), 0 info

Notice the "did you mean" hints: typo links and near-miss fragments get a Levenshtein-style suggestion instead of a bare failure. The case-mismatch lines are the ones source review usually misses, because the file "looks present" on a case-insensitive laptop.

Wire it into CI after the build

The cheapest habit is a postbuild script:

npm i -D static-link-sentinel

{
  "scripts": {
    "postbuild": "static-link-sentinel dist --site-url https://example.com"
  }
}

Or the GitHub Action after npm run build:

- uses: MrLouisWright/static-link-sentinel@v1
  with:
    dir: dist
    site-url: https://example.com
    fail-on: error
    sarif: links.sarif

Framework recipes follow the same pattern: run after vite build, next build (static export to out/), astro build, Eleventy, or Hugo, and pass --base-path when the site is served under a subpath. Vite and webpack plugins can attach the check to the bundler lifecycle so a bad build never exits green.

For outbound links, enable --external on a schedule or on push (not necessarily every PR) and cache .link-cache.json between runs so unchanged URLs are not re-fetched every time.

Limits worth knowing

Only static markup and assets are visible. Client-side routers and content injected by JavaScript after load are out of scope, and SPA fallbacks that rewrite every path to index.html make every route look "present." Soft 404s on remote hosts can still return HTTP 200. HTML is tokenized with an offset-preserving regex approach rather than a full HTML5 parser, so badly malformed markup may be misread. Those are honest boundaries, not marketing fine print.

Check the bytes you will serve

Pew's figures show broken links are common even on news and government pages [1]. Case-sensitive object stores will not forgive a capital letter that a laptop filesystem ignored [2]. Search systems treat soft 404s and redirect chains as crawl waste [3][4]. The practical response for a static site is simple: after the build, crawl the output, fail the job when references do not resolve, and fix the path before the CDN ever sees it.