A broken link checker looks like a twenty-line script: fetch a page, collect the links, request each one, print whatever is not 200. It stays that simple until you run it on a real site. Then a redirect loop hangs it, LinkedIn shows up as broken, a gzipped sitemap comes back as garbage, and a page with no title reports its title as "Close".
Below are the nine problems that came up most often, with a fix for each, and the output of a full run on crawler-test.com, a public site full of deliberate SEO errors, made for testing crawlers. The code is Node.js 20+ with the built-in fetch, plus cheerio for HTML and robots-parser. No browser is needed: link and SEO checks only need the HTML.
If you need the result rather than the code, the same checks run as a hosted tool: SEO Audit & Broken Link Checker crawls a site and returns an HTML report, a CSV of broken links and a 0–100 score.
The starting point
// node check.mjs https://your-site.com/
import * as cheerio from 'cheerio';
const base = process.argv[2];
const html = await (await fetch(base)).text();
const $ = cheerio.load(html);
const links = new Set();
$('a[href]').each((_, a) => {
const url = new URL($(a).attr('href'), base);
url.hash = '';
if (url.protocol.startsWith('http')) links.add(url.href);
});
for (const url of links) {
const res = await fetch(url, { method: 'HEAD' });
if (!res.ok) console.log(res.status, url);
}
On the "About" page of python.org it checks 99 links and reports two broken ones: a LinkedIn page and a page on opensource.org, both with status 403. Both open fine in a browser. Each section below fixes one problem like this.
1. fetch hides redirects
By default fetch follows redirects silently. A link that goes through three hops looks the same as a direct one, and a redirect loop ends in a generic "redirect count exceeded" error. Follow redirects yourself with redirect: 'manual'. Unlike browsers, Node gives you the real 3xx status and the Location header:
const UA = 'Mozilla/5.0 (compatible; MyLinkChecker/1.0)';
async function request(url, method) {
return fetch(url, {
method,
redirect: 'manual',
headers: { 'user-agent': UA },
signal: AbortSignal.timeout(10_000),
});
}
async function checkLink(url, maxHops = 10) {
const seen = new Set();
let current = url;
for (let hop = 0; hop <= maxHops; hop++) {
if (seen.has(current)) return { url, broken: true, error: 'REDIRECT_LOOP', hops: hop };
seen.add(current);
let res;
try {
res = await request(current, 'HEAD');
if ([403, 405, 501].includes(res.status)) res = await request(current, 'GET');
await res.body?.cancel();
} catch (err) {
return { url, broken: true, error: errorCode(err), hops: hop };
}
const location = res.headers.get('location');
if (res.status >= 300 && res.status < 400 && location) {
current = new URL(location, current).href;
continue;
}
return { url, status: res.status, finalUrl: current, hops: hop, broken: res.status >= 400 };
}
return { url, broken: true, error: 'TOO_MANY_REDIRECTS', hops: maxHops };
}
On crawler-test.com this tells the cases apart:
| Link | Result |
|---|---|
/redirects/redirect_chain_allowed |
200 after 2 hops |
/redirects/redirect_to_404 |
404 after 1 hop: broken |
/redirects/infinite_redirect |
redirects to itself: REDIRECT_LOOP
|
A redirect that ends in a 404 is a broken link, even though the first response was a 3xx. Links that work through redirects are not broken, but they are worth updating: each hop is one more round trip for visitors and for search engine crawlers.
2. Some servers refuse HEAD
HEAD is cheap because it skips the body, but some servers answer it with 405 Method Not Allowed, 501 or 403 while GET works. The code above retries with GET before calling a link broken.
await res.body?.cancel() matters for GET: without it, Node keeps the connection busy until the body is read or garbage-collected, and a checker with a few hundred links runs out of sockets.
3. Errors without a status code
A dead domain, a reset connection or a timeout does not return a response at all: fetch throws TypeError: fetch failed, and the real reason is in err.cause.code:
function errorCode(err) {
if (err.name === 'TimeoutError') return 'TIMEOUT';
const code = err.cause?.code;
if (code === 'ENOTFOUND' || code === 'EAI_AGAIN') return 'DNS_FAILURE';
if (code === 'ECONNRESET') return 'CONNECTION_RESET';
return 'REQUEST_FAILED';
}
Put a timeout on every request with AbortSignal.timeout. Retry a reset connection once, but don't retry timeouts in a loop: a link that took 10 seconds twice is already a problem for visitors, and retries make a run on a slow site take hours.
4. A 403 from bot protection is not a broken link
Within one minute, the LinkedIn page from the python.org test answered 403 to Node's default user agent, 429 to a HEAD request and a 301 redirect to a GET. LinkedIn is also known for the non-standard status 999. Other social networks send 403, 429 or a login page, and sites behind bot protection return 403 to every script, like opensource.org above. If you count these as broken, every site that links to its social profiles fails the check. Put them in a separate group, "restricted", which someone can click through by hand:
const BOT_WALLS = /(^|\.)(linkedin|x|twitter|facebook|instagram|tiktok)\.com$/;
function verdict(r) {
if (!r.broken) return r.hops ? 'redirect' : 'ok';
if (BOT_WALLS.test(new URL(r.url).hostname) && r.error !== 'DNS_FAILURE') return 'restricted';
if ([401, 403, 429, 451, 999].includes(r.status)) return 'restricted';
if (r.error === 'TIMEOUT') return 'timeout';
return 'broken';
}
5. Read robots.txt before the first page
Check every URL against robots.txt with your checker's own user-agent token before you request it. A link to a disallowed URL is not broken, so count it separately. robots.txt also tells you where the sitemaps are:
import robotsParser from 'robots-parser';
async function loadRobots(site) {
const robotsUrl = new URL('/robots.txt', site).href;
const res = await fetch(robotsUrl, { headers: { 'user-agent': UA } });
return robotsParser(robotsUrl, res.ok ? await res.text() : '');
}
const robots = await loadRobots('https://crawler-test.com/');
robots.isAllowed('https://crawler-test.com/', 'MyLinkChecker'); // true
robots.getSitemaps(); // 29 sitemap URLs on crawler-test.com
Don't guess /sitemap.xml: on many sites it does not exist, and the real sitemap has another name listed in robots.txt.
6. Sitemaps: indexes, .gz files and slow servers
Pages that no link points to are still in the sitemap, and comparing the two finds real problems: sitemap URLs that return errors, redirect or are set to noindex, and indexable pages missing from the sitemap. Three details:
- A sitemap index lists other sitemaps, not pages.
- A
.gzsitemap arrives as gzip bytes.fetchonly decompressesContent-Encoding, not a gzipped file, so check the first two bytes. - Some sitemaps are slow or missing. Give them a total time budget.
import { gunzipSync } from 'node:zlib';
async function readSitemap(url, urls = new Set(), depth = 0) {
const res = await fetch(url, { headers: { 'user-agent': UA } });
if (!res.ok) return urls;
let body = Buffer.from(await res.arrayBuffer());
if (body[0] === 0x1f && body[1] === 0x8b) body = gunzipSync(body);
const xml = body.toString('utf8');
const locs = [...xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)].map((m) => m[1].replaceAll('&', '&'));
if (/<sitemapindex/i.test(xml) && depth < 2) {
for (const loc of locs) await readSitemap(loc, urls, depth + 1);
} else {
for (const loc of locs) urls.add(loc);
}
return urls;
}
On crawler-test.com two of the sitemaps listed in robots.txt return 404 themselves, and a few are slow on purpose. Reading all of them took 30 seconds.
7. The page title that says "Close"
$('title') matches every <title> element, including the ones inside inline SVG icons, where they serve as accessible labels. On a page without a real title, a naive check reports "Close" or "Menu" instead of flagging the missing title. Skip titles inside <svg>:
function pageChecks(html) {
const $ = cheerio.load(html);
const title = $('title')
.filter((_, el) => $(el).closest('svg').length === 0)
.first().text().trim();
return {
title,
description: $('meta[name="description"]').attr('content')?.trim() ?? '',
h1Count: $('h1').length,
canonical: $('link[rel="canonical"]').attr('href') ?? null,
noindex: /noindex/i.test($('meta[name="robots"]').attr('content') ?? ''),
imagesWithoutAlt: $('img:not([alt])').length,
};
}
Remember the X-Robots-Tag response header too: it can set noindex without any meta tag.
8. Duplicates need the whole site
A duplicate title or meta description is not visible on any single page. Collect the checks for every page, then group by value after the crawl:
function duplicates(pages, field) {
const groups = new Map();
for (const p of pages) {
if (!p[field]) continue;
groups.set(p[field], [...(groups.get(p[field]) ?? []), p.url]);
}
return [...groups].filter(([, urls]) => urls.length > 1);
}
On crawler-test.com four pages share the title "Duplicate title".
9. Four copies of the same site
http:// and https://, with and without www: all four should end on one address. When they don't, search engines see four copies of every page:
async function hostVariants(host) {
const variants = ['https://', 'http://', 'https://www.', 'http://www.'].map((p) => `${p}${host}/`);
const finals = await Promise.all(variants.map((u) =>
fetch(u, { headers: { 'user-agent': UA }, signal: AbortSignal.timeout(10_000) })
.then((r) => new URL(r.url).origin)
.catch(() => null)));
return { finals, singleHost: new Set(finals.filter(Boolean)).size === 1 };
}
python.org sends all four to https://www.python.org. crawler-test.com answers 200 on all four without a redirect, which is one of its deliberate errors.
A full run
Here is everything above put together in one crawler, run on my machine against crawler-test.com with this input:
{
"startUrls": ["https://crawler-test.com/"],
"maxPagesPerSite": 60
}
It audited 60 pages and checked every link on them in 90 seconds, 30 of which went to the sitemaps. Part of the summary:
{
"healthScore": 95,
"pagesAudited": 60,
"issueTotals": { "errors": 3, "warnings": 91, "notices": 491 },
"brokenLinksCount": 65,
"hosts": { "httpRedirectsToHttps": false, "singleHost": false },
"aiSearch": { "llmsTxt": false }
}
The 65 broken links are all on the home page, which links to a test page for almost every HTTP status code. 58 of them return a 4xx or 5xx status. The other 7 fail differently: a DNS failure, a redirect loop, a redirect chain that never ends, a reset connection, a failed request and two timeouts. A few rows of the broken links CSV, with the site column left out:
page,broken_url,type,status_code,error,anchor_text
https://crawler-test.com/,https://invalid.crawler-test.com/,external,,DNS_FAILURE,Invalid Subdomain
https://crawler-test.com/,https://crawler-test.com/robots_protocol/non_200_with_noindex,internal,404,,Non-200 with Noindex
https://crawler-test.com/,https://crawler-test.com/redirects/infinite_redirect,internal,303,REDIRECT_LOOP,Infinite Redirect
https://crawler-test.com/,https://crawler-test.com/redirects/redirect_to_404,internal,404,,Redirect To 404 Http Status
https://crawler-test.com/,https://crawler-test.com/status_codes/status_100,internal,,CONNECTION_RESET,100 HTTP Status
The most common issues by number of pages:
| Issue | Pages |
|---|---|
| No mobile viewport meta tag | 55 |
| Title longer than 60 characters | 12 |
| Missing meta description | 8 |
| Title shared with another page | 6 |
Set to noindex
|
4 |
| Missing or empty title | 2 |
The health score is the share of pages without errors: 57 of 60. Warnings and notices, such as a missing lang attribute or Open Graph tags, don't lower it.
Things that kept the run fast and polite
- Check each URL once. The crawler-test.com home page alone has 408 internal links, and most pages link to the same navigation. Cache results by URL and every page after the first costs a few requests.
- Limit concurrency per host, not overall. Two to four requests at a time to one site, while external links to other hosts are checked in parallel.
- Cap the work. A page limit, a limit on link checks per site and a time limit keep a run on a huge site from going on for hours.
- No browser. Plain HTTP and cheerio need about 1 GB of memory. Earlier, in the cloud, 30 pages of python.org took 23 seconds and about $0.0026 of compute.
All of these checks are in the SEO Audit & Broken Link Checker linked at the top, with the HTML report, the CSV and the 0–100 score.
Top comments (3)
Solid list, and the 403-vs-broken distinction alone removes most false alarms. Three things not covered here that tend to show up on real sites:
<base href>: relative links resolve against it, not against the page URL. Resolving withnew URL(href, pageUrl)without checking for a base element reports 404s that don't exist in a browser./docs#installreturns 200 even ifid="install"was removed. If anchors matter, check that the target id (or a legacy<a name>) exists in the fetched HTML. And strip the fragment before using the URL as a cache key, otherwise the same page is fetched once per anchor.On 429s, honouring
Retry-After(seconds or an HTTP date) before re-checking keeps a rate-limited link from being reported as broken.Thanks, good additions. Checked them against the actor that came out of this post:
<base href>: handled. Links resolve against the first base element when the page has one.Follow-up: all three are in now, version 0.1.7 of apify.com/makerio/seo-audit-broken...
A run on python.org turned up three real ones, for example /about/apps/#database-access points to an id that is no longer on the page. Thanks for the list.