A build step that refuses to ship broken links

Most teams check for broken links the way they check smoke detector batteries: once a quarter, with a tool that crawls the live site and spits out a spreadsheet nobody reads before the next release. By the time the audit runs, the broken link has been live for weeks and possibly already flagged by a user who did not bother reporting it.

The fix is to move the check earlier and make it a gate instead of a report. Crawl the generated output before it deploys, fail the build on errors that are the site's fault, and warn on the ones that are not.

Crawl the build output, not the live site

Checking the deployed site catches problems after they ship. Checking the static output the build just produced catches them before, and it is faster too: no network latency, no CDN cache to fight, just files on disk.

// scripts/crawl-links.mjs
import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs';
import path from 'node:path';

function collectHtmlFiles(dir, files = []) {
  for (const entry of readdirSync(dir)) {
    const full = path.join(dir, entry);
    if (statSync(full).isDirectory()) {
      collectHtmlFiles(full, files);
    } else if (entry.endsWith('.html')) {
      files.push(full);
    }
  }
  return files;
}

function extractLinksAndIds(html) {
  const links = [...html.matchAll(/href="([^"]+)"/g)].map((m) => m[1]);
  const ids = new Set([...html.matchAll(/\sid="([^"]+)"/g)].map((m) => m[1]));
  return { links, ids };
}

A regex is enough here. The build output is markup this same pipeline generated, not arbitrary HTML from the web, so the shape of href and id attributes is known and consistent.

Internal links are the site's fault, external links are not

The distinction that matters is who can fix the problem. An internal link pointing at a page that does not exist in the build is a bug in the site: someone renamed a route, deleted a page, or typo'd a path. That should fail the build. An external link returning a 404 might be a bug on someone else's server, or a site that restructured its URLs months ago without telling anyone. Worth knowing, but not a reason to block a deploy that has nothing to do with it.

function classifyLink(href, currentFile, distRoot) {
  if (href.startsWith('#')) return { kind: 'anchor', target: currentFile, fragment: href.slice(1) };
  if (/^https?:\/\//.test(href)) return { kind: 'external', url: href };
  if (href.startsWith('mailto:') || href.startsWith('tel:')) return { kind: 'skip' };

  const [pathPart, fragment] = href.split('#');
  const resolved = path.join(path.dirname(currentFile), pathPart || '');
  const candidate = resolved.endsWith('.html') ? resolved : path.join(resolved, 'index.html');
  return { kind: 'internal', target: candidate, fragment };
}

Anchors get checked against the target page's own id set, whether that page is the current one or a different internal page. A link to /pricing#faq is only correct if /pricing/index.html actually contains id="faq".

External checks: cache them, do not hammer someone else's server

Re-fetching every external link on every build is slow and rude. A site with two hundred outbound links does not need two hundred fresh requests for a commit that only fixes a typo in the footer. Cache the result, keyed by URL, and only re-check entries older than a set window.

import { existsSync, readFileSync, writeFileSync } from 'node:fs';

const CACHE_PATH = '.link-cache.json';
const CACHE_TTL_MS = 7 * 24 * 60 * 60 * 1000; // one week

function loadCache() {
  return existsSync(CACHE_PATH) ? JSON.parse(readFileSync(CACHE_PATH, 'utf8')) : {};
}

async function checkExternal(url, cache) {
  const cached = cache[url];
  if (cached && Date.now() - cached.checkedAt < CACHE_TTL_MS) {
    return cached.ok;
  }
  try {
    const res = await fetch(url, { method: 'HEAD', signal: AbortSignal.timeout(8000) });
    const ok = res.ok || res.status === 405; // some servers reject HEAD, treat 405 as inconclusive-pass
    cache[url] = { ok, checkedAt: Date.now() };
    return ok;
  } catch {
    cache[url] = { ok: false, checkedAt: Date.now() };
    return false;
  }
}

A HEAD request is enough to know whether a link resolves, and it costs the remote server far less than a full GET. Persisting the cache between runs, as a committed file or a CI cache artifact, means a flaky link gets flagged once, not on every commit for a week.

Fail on the right category

The crawler walks every page, resolves every link, and separates the two buckets before deciding whether the build lives or dies.

let brokenInternal = 0;
let brokenExternal = 0;

// ... after walking all files and resolving every link:
if (brokenInternal > 0) {
  console.error(`${brokenInternal} broken internal link(s). Failing build.`);
  process.exit(1);
}
if (brokenExternal > 0) {
  console.warn(`${brokenExternal} external link(s) did not resolve. Review before next release.`);
}
process.exit(0);

Wired into CI, this runs after the static build and before the deploy step, so a broken internal link never reaches production in the first place:

# .github/workflows/deploy.yml
- name: Build site
  run: npm run build
- name: Check links
  run: node scripts/crawl-links.mjs dist
- name: Deploy
  run: npm run deploy

The team building this kind of pipeline gate for client sites, including the redirect and link checks that go with a redesign, writes about the broader approach at nooralto.com.

Treating link rot as a build-time concern rather than a quarterly chore changes who finds the problem. Instead of a user hitting a dead page, or an audit six weeks late, the commit that introduced the break never merges.

Written by the team at Nooralto, a web and SEO studio working out of Agadir and Paris.

---

Built by Nooralto — Nooralto.