DEV Community: Swift Labs

Broken Link Checking for Docs Sites: Build-Time Checks Plus Live Monitoring

Swift Labs — Wed, 21 Oct 2026 14:08:55 +0000

Documentation sites are hard on links. They're full of cross-references between pages, anchors to specific sections, links to API references, code repositories, changelogs and third-party docs, and all of those move. Pages get renamed when the product changes, versions get archived, and the external projects you link to restructure their own docs. A reader who follows a "see Configuration" link and lands on a 404 in the middle of a task is a frustrated reader, and often becomes a support ticket.

The good news is that docs teams have better tools for this than most websites, because docs are usually built from source files. The catch is that build-time checks only catch some problems. This guide covers both halves: what to check during the build, and what you can only catch by checking the live site.

The three kinds of broken docs links

It helps to separate them, because different tools catch each kind.

Internal page links. A link from one doc page to another that doesn't exist (renamed file, deleted page, typo in a relative path). Your static site generator can catch these at build time, because it knows every page that exists.
Anchor links. A link to /config#retry-options when the heading is now "Retry settings", so the anchor is #retry-settings. The page loads, so nothing returns a 404, but the reader lands at the top of a long page instead of the section they wanted. Some generators can check these. Many link checkers ignore them.
External links. Links to GitHub, package registries, other projects' docs, blog posts and standards. Your build can't know whether these work, so you need a checker that makes real HTTP requests, and those links can break at any time after you deploy.

There's a fourth, sneakier kind: links that work in the build but break in production. Server-level redirects, a base path that's different in production, a CDN rule, or a versioned-docs setup where old versions point at pages that only exist in the latest version. Only a check against the live site finds these.

Layer 1: Turn on your generator's own checks

Start here. It's free and takes a few minutes. Exact option names vary between versions, so check your generator's docs for the version you run.

Docusaurus

Docusaurus checks internal links when you run a production build, and the onBrokenLinks option in docusaurus.config.js controls what happens when it finds one. In current versions the default is 'throw', which fails the build. Keep it that way. There are related options for broken Markdown file links and for broken anchors (onBrokenAnchors, which defaults to a warning in the versions that have it). If you can, set anchors to throw as well, so renamed headings fail the build instead of quietly sending readers to the top of the page.

Note that these checks happen during docusaurus build, not in the dev server, so make sure CI runs a real build on every pull request.

MkDocs

Run mkdocs build --strict in CI. Strict mode turns warnings, including warnings about links to documentation files that don't exist, into a failed build. Recent MkDocs versions also have a validation section in mkdocs.yml that controls how strictly different link problems are treated, including anchors. MkDocs validates links between your Markdown files. It doesn't check external URLs.

Sphinx

Sphinx warns about broken cross-references during a normal build (-W turns warnings into errors, and -n "nitpicky" mode reports missing references more aggressively). For external URLs it ships a dedicated builder: sphinx-build -b linkcheck (or make linkcheck) requests every external link and writes a report. It's slow on large sites, so most teams run it on a schedule rather than on every commit.

Hugo

Hugo's ref and relref shortcodes fail or warn (depending on refLinksErrorLevel) when they point at a page that doesn't exist. Plain Markdown links aren't checked, so teams that care use ref/relref for internal links and an external checker for everything else.

Everything else (GitBook, Mintlify, ReadMe, custom setups)

Hosted docs platforms vary a lot. Some flag broken internal links in their editor, some don't, and few check external links on a schedule. If yours doesn't, you'll rely more on Layers 2 and 3.

Layer 2: Run a link checker in CI

Build checks handle internal links. For external links, add a dedicated link checker to your CI pipeline. Common choices:

lychee, a fast link checker written in Rust that works on Markdown, HTML and plain text. It has an official GitHub Action (lycheeverse/lychee-action) and supports caching, exclusions and retries.
markdown-link-check, a Node tool that checks links in Markdown files and is popular in smaller repos.
Generator-specific actions, such as the MkDocs link-check GitHub Action, which wrap a checker with sensible defaults for that generator.

A few lessons that save a lot of frustration:

Don't block pull requests on external links. An external site going down for ten minutes shouldn't stop someone merging a typo fix. Block PRs on internal link failures (Layer 1). Run external checks on a schedule or as a non-blocking job, and open an issue when something fails.

Expect bot-blocking. Some sites answer automated requests with 403, 429 or other non-200 codes while working fine in a browser. Social networks and some large platforms are known for it. Keep an exclusion list for known false positives, and review it every few months, because "excluded" can quietly turn into "never checked."

Retry before failing. Use your checker's retry settings, or have it check twice before reporting. A single timeout isn't a broken link.

Use caching on a schedule. Checking every external link in a large docs repo on every run is slow and can get you rate-limited. Caching results for a day or so is a reasonable compromise.

Layer 3: Monitor the live site

Layers 1 and 2 check your source. Your readers use the deployed site, and the two differ more than you'd think:

Redirects and rewrites set up in your host, CDN or reverse proxy don't exist in the build environment.
Base paths and versioning. Docs served under /docs/v2/ can behave differently from the local build, especially with versioned docs where old versions link to paths that only exist in newer ones.
Generated content. API references generated from code, embedded changelogs or content pulled in at deploy time may never be seen by the source checker.
Things that break after deploy. Every external link you checked at build time can break next week. If you only check on commits, a docs section nobody edits never gets rechecked.
Assets. Images, downloadable files and embedded media served from a separate bucket or CDN.

A scheduled crawl of the live site catches all of these. It works the way a reader does: it starts at your docs homepage, follows links, and records every link and image that returns an error, along with the page it was found on. Weekly is enough for most docs sites. If you deploy docs many times a day, run it daily and alert only on new breakage, so the alerts don't become noise.

Any crawler that records the "found on" page for each broken link works here: a scheduled lychee run against the deployed URL, a hosted monitor, or a small script of your own. What matters is that it hits the live site, rechecks failures before reporting them, and compares each run with the previous one.

Docs-specific habits that prevent breakage

Redirect when you rename. Most docs platforms and hosts support redirect maps. Make "add a redirect" part of the checklist for renaming or moving any page, the same way you'd update the sidebar. Old docs URLs get bookmarked, linked from forum answers and Stack Overflow, and cited in other projects' docs, so they need to keep working.

Link to stable URLs. For external references, prefer permalinks: tagged release URLs instead of main branch file links (which break when files move), versioned API docs instead of "latest" where the content matters, and DOIs or archived copies for papers and specs.

Link to headings deliberately. If your generator lets you set explicit heading IDs, use them for sections that get linked often. Then changing the heading text doesn't change the anchor.

Keep an eye on archived versions. Versioned docs multiply your link count. Decide how far back you'll maintain links. For versions you've frozen, you can exclude external links from checks and add a banner pointing readers to the current version.

Give link reports an owner. Route scheduled link failures to a real place, such as a docs issue label, a Slack channel or a weekly triage, with a named owner. A report nobody owns gets ignored.

A setup that covers all three layers

Every pull request: production build with the generator's broken-link checks set to fail (internal pages, plus anchors if supported).
Nightly or weekly in CI: lychee or similar against the source for external links, non-blocking, opening an issue when something fails.
Weekly (daily if you deploy often): a crawl of the live docs site, alerting only on new breakage, with the "found on" page included.
Always: redirects for moved pages, stable external URLs, and explicit IDs for frequently linked headings.

Originally published on the DeadLinkPatrol blog. We build DeadLinkPatrol, a scheduled broken-link checker. If you want to see what a live crawl finds on your own site, the free checker scans up to 25 pages without signup.

This article was created with the help of AI and checked for accuracy before publishing.

Search Console "Not Found (404)" Errors: Which to Fix and Which to Ignore

Swift Labs — Mon, 19 Oct 2026 07:34:44 +0000

Open the Page indexing report in Google Search Console on almost any site that has been around for a while and you'll find a "Not found (404)" row, sometimes with hundreds or thousands of URLs. It looks like an emergency. Most of the time it isn't, but a few of those URLs probably do matter, and the trick is to work out which ones.

This guide explains what Google actually says about 404s, gives you a simple triage to sort the list into "fix", "redirect" and "leave alone", and shows how to stop the list from growing.

What Google says about 404s

It's worth starting from Google's own documentation rather than SEO folklore.

In the Search Console help for the Page indexing report, Google describes a "Not found (404)" URL as one that "returned a 404 error when requested," discovered "without any explicit request or sitemap," possibly "as a link from another page, or possibly the page existed before and was deleted." It then says plainly: "404 responses are not necessarily a problem, if the page has been removed without any replacement. If your page has moved, use a 301 redirect to the new location." It also notes that Googlebot "will probably continue to try this URL for some period of time" and "will crawl it less and less often."

Google's developer documentation on HTTP status codes adds that all 4xx errors except 429 are treated the same way: the URL is removed from the index if it was indexed before, "newly encountered 404 pages aren't processed," and "the crawling frequency gradually decreases."

That gives you two practical conclusions:

A 404 is the correct response for a page that's genuinely gone. Returning one doesn't penalize your site.
A 404 on a URL that should work is a real problem. It's not a penalty either, but the page drops out of the index and any traffic or link value it had is lost.

So you're not trying to get the 404 count to zero. You're trying to find the 404s that shouldn't be 404s.

Step 1: Export the list

In Search Console, go to Indexing → Pages, click the Not found (404) row, and export the examples. Search Console shows a sample of affected URLs, not necessarily every one, so treat the export as a starting point rather than a full inventory.

Add two columns to the spreadsheet: Category and Action.

Step 2: Sort every URL into one of five buckets

Go through the list (filter by URL pattern to speed it up) and put each URL into one of these buckets.

Bucket A: Pages that moved and still have an equivalent

A product renamed from /blue-widget to /widgets/blue. A blog post whose slug changed. A service page merged into another service page.

Action: 301 redirect to the closest equivalent. This is exactly what Google's help recommends, and it's the most valuable fix on the list, because these URLs often had rankings, backlinks, or both.

Bucket B: Pages deleted on purpose with no replacement

A discontinued product with no successor. An event from 2019. A job ad that has been filled.

Action: leave it as a 404 (or a 410 if you want to signal "gone for good"). Don't redirect these to the homepage. Google tends to treat irrelevant redirects like that as soft 404s anyway, and users who land on your homepage instead of what they clicked get confused. If the deleted page had good backlinks and there's a genuinely related page, such as the category it belonged to, a redirect there is reasonable.

Bucket C: URLs that never existed

Typos in links on other sites, URLs with junk parameters, spam bots probing /wp-login.php on a site that isn't WordPress, paths with tracking garbage tacked on.

Action: ignore. You don't control other people's typos, and these URLs never had value. The exception is a typo'd URL that has a strong backlink pointing at it. A redirect to the intended page recovers that value.

Bucket D: URLs created by your own broken links

This is the bucket people miss. The URL is in Search Console because your own site links to it. Common causes:

A link edited in one place but not in another (old menu item, footer, sidebar widget)
A relative link missing its protocol: an editor types example.com/page instead of https://example.com/page, and the browser resolves it relative to the current page, so you get /about/example.com/page
A template that builds URLs from data that's sometimes empty (/products/ + nothing)
Links in old blog posts pointing at pages that have since moved

Action: fix the link at the source, and add a redirect if the URL has external value. A redirect alone isn't enough. If your own pages link to a 404, every visitor who clicks it hits a dead end, and internal links pass through a redirect hop for no reason. Find the page that contains the link and edit it.

Search Console's URL Inspection tool can sometimes show you where Google found a URL, but it doesn't reliably list every page that links to it. A crawler that records the source page for every broken link, the "found on" page, is the faster way to find these.

Bucket E: URLs that should work but don't

The URL is correct, the page should exist, and it returns 404 anyway. Typical causes are a deployment that dropped a route, a CMS page set to draft by mistake, a permalink flush that didn't run, or a server rule that's too broad.

Action: fix the page itself. This is a bug, not a content decision. If a Bucket E URL is a revenue page (pricing, checkout, contact, booking), it's the most urgent item on the whole list.

Step 3: Prioritize

With everything bucketed, work in this order:

Bucket E on important pages. Broken pages that should be working.
Bucket D. Your own links pointing at 404s, starting with sitewide elements (menus, footers), because one fix there repairs hundreds of instances.
Bucket A, ordered by value. If you can see which moved URLs had traffic (in analytics or Search Console's performance report) or backlinks (in any backlink tool), redirect those first.
Bucket C typos with strong backlinks.
Everything else in B and C: nothing to do.

On a typical site, steps 1 and 2 involve a fairly short list and do most of the good.

Step 4: Validate and wait

After fixing, you can use Validate fix on the Not found (404) row in Search Console. Expect it to take a while. Google rechecks on its own schedule, and the help docs say Googlebot keeps trying removed URLs "for some period of time." URLs you intentionally left as 404s will stay in the report for a while, and that's fine.

Don't judge your work by whether the 404 count goes down. Judge it by whether the URLs in buckets A, D and E now resolve correctly.

Why Search Console alone isn't enough

Search Console is a good lagging indicator, but it has gaps for this job:

It's slow. URLs show up after Googlebot crawls them, which can be days or weeks after a link broke.
It's a sample. The export won't necessarily include every affected URL.
It only sees what Google crawls. It won't show broken links in pages Google skips, in noindexed sections, or links to external sites that have died. Your visitors still click those.
It mixes good 404s with bad ones. You have to do the triage above every time.

A scheduled link crawler fills these gaps. It works from your side: it crawls your pages, checks every link it finds, and tells you which page each broken link sits on, usually within a day of the break. That turns Bucket D from detective work into a to-do list.

Stopping new 404s from building up

A few habits keep the report small:

Redirect when you rename. Make "add a 301" part of the process for changing any published URL. Most CMSs have redirect plugins or settings.
Scan after big changes. Migrations, redesigns, permalink changes and bulk deletions all need a crawl before and after.
Paste full URLs. Train editors to paste links with https://, or use a CMS link picker for internal links.
Monitor on a schedule. Weekly for most sites, daily for stores and client sites you're paid to maintain, with alerts only for new breakage.
Check your sitemap. A sitemap that lists 404 URLs sends Google mixed signals. Regenerate it after deletions.

The short version

404s for pages that are really gone are normal. Google says so.
Redirect moved pages to their closest equivalent with a 301.
Fix your own links that point to 404s at the source. This is the most commonly missed fix.
Treat a 404 on a page that should exist as a bug, and fix it first.
Ignore typos and junk URLs unless they carry real backlinks.
Use a crawler that records the "found on" page, so you can see where the broken links come from without waiting for Google.

This article was created with the help of AI and checked for accuracy before publishing.