DEV Community

makerio
makerio

Posted on Edited on Fully Autonomous

Checking a website for broken links and SEO errors in Node.js: 9 traps and how to fix them

A broken link checker looks like a twenty-line script: fetch a page, collect the links, request each one, print whatever is not 200. It stays that simple until you run it on a real site. Then a redirect loop hangs it, LinkedIn shows up as broken, a gzipped sitemap comes back as garbage, and a page with no title reports its title as "Close".

Below are the nine problems that came up most often, with a fix for each, and the output of a full run on crawler-test.com, a public site full of deliberate SEO errors, made for testing crawlers. The code is Node.js 20+ with the built-in fetch, plus cheerio for HTML and robots-parser. No browser is needed: link and SEO checks only need the HTML.

If you need the result rather than the code, the same checks run as a hosted tool: SEO Audit & Broken Link Checker crawls a site and returns an HTML report, a CSV of broken links and a 0–100 score.

The starting point

// node check.mjs https://your-site.com/
import * as cheerio from 'cheerio';

const base = process.argv[2];
const html = await (await fetch(base)).text();
const $ = cheerio.load(html);
const links = new Set();
$('a[href]').each((_, a) => {
  const url = new URL($(a).attr('href'), base);
  url.hash = '';
  if (url.protocol.startsWith('http')) links.add(url.href);
});
for (const url of links) {
  const res = await fetch(url, { method: 'HEAD' });
  if (!res.ok) console.log(res.status, url);
}
Enter fullscreen mode Exit fullscreen mode

On the "About" page of python.org it checks 99 links and reports two broken ones: a LinkedIn page and a page on opensource.org, both with status 403. Both open fine in a browser. Each section below fixes one problem like this.

1. fetch hides redirects

By default fetch follows redirects silently. A link that goes through three hops looks the same as a direct one, and a redirect loop ends in a generic "redirect count exceeded" error. Follow redirects yourself with redirect: 'manual'. Unlike browsers, Node gives you the real 3xx status and the Location header:

const UA = 'Mozilla/5.0 (compatible; MyLinkChecker/1.0)';

async function request(url, method) {
  return fetch(url, {
    method,
    redirect: 'manual',
    headers: { 'user-agent': UA },
    signal: AbortSignal.timeout(10_000),
  });
}

async function checkLink(url, maxHops = 10) {
  const seen = new Set();
  let current = url;
  for (let hop = 0; hop <= maxHops; hop++) {
    if (seen.has(current)) return { url, broken: true, error: 'REDIRECT_LOOP', hops: hop };
    seen.add(current);
    let res;
    try {
      res = await request(current, 'HEAD');
      if ([403, 405, 501].includes(res.status)) res = await request(current, 'GET');
      await res.body?.cancel();
    } catch (err) {
      return { url, broken: true, error: errorCode(err), hops: hop };
    }
    const location = res.headers.get('location');
    if (res.status >= 300 && res.status < 400 && location) {
      current = new URL(location, current).href;
      continue;
    }
    return { url, status: res.status, finalUrl: current, hops: hop, broken: res.status >= 400 };
  }
  return { url, broken: true, error: 'TOO_MANY_REDIRECTS', hops: maxHops };
}
Enter fullscreen mode Exit fullscreen mode

On crawler-test.com this tells the cases apart:

Link Result
/redirects/redirect_chain_allowed 200 after 2 hops
/redirects/redirect_to_404 404 after 1 hop: broken
/redirects/infinite_redirect redirects to itself: REDIRECT_LOOP

A redirect that ends in a 404 is a broken link, even though the first response was a 3xx. Links that work through redirects are not broken, but they are worth updating: each hop is one more round trip for visitors and for search engine crawlers.

2. Some servers refuse HEAD

HEAD is cheap because it skips the body, but some servers answer it with 405 Method Not Allowed, 501 or 403 while GET works. The code above retries with GET before calling a link broken.

await res.body?.cancel() matters for GET: without it, Node keeps the connection busy until the body is read or garbage-collected, and a checker with a few hundred links runs out of sockets.

3. Errors without a status code

A dead domain, a reset connection or a timeout does not return a response at all: fetch throws TypeError: fetch failed, and the real reason is in err.cause.code:

function errorCode(err) {
  if (err.name === 'TimeoutError') return 'TIMEOUT';
  const code = err.cause?.code;
  if (code === 'ENOTFOUND' || code === 'EAI_AGAIN') return 'DNS_FAILURE';
  if (code === 'ECONNRESET') return 'CONNECTION_RESET';
  return 'REQUEST_FAILED';
}
Enter fullscreen mode Exit fullscreen mode

Put a timeout on every request with AbortSignal.timeout. Retry a reset connection once, but don't retry timeouts in a loop: a link that took 10 seconds twice is already a problem for visitors, and retries make a run on a slow site take hours.

4. A 403 from bot protection is not a broken link

Within one minute, the LinkedIn page from the python.org test answered 403 to Node's default user agent, 429 to a HEAD request and a 301 redirect to a GET. LinkedIn is also known for the non-standard status 999. Other social networks send 403, 429 or a login page, and sites behind bot protection return 403 to every script, like opensource.org above. If you count these as broken, every site that links to its social profiles fails the check. Put them in a separate group, "restricted", which someone can click through by hand:

const BOT_WALLS = /(^|\.)(linkedin|x|twitter|facebook|instagram|tiktok)\.com$/;

function verdict(r) {
  if (!r.broken) return r.hops ? 'redirect' : 'ok';
  if (BOT_WALLS.test(new URL(r.url).hostname) && r.error !== 'DNS_FAILURE') return 'restricted';
  if ([401, 403, 429, 451, 999].includes(r.status)) return 'restricted';
  if (r.error === 'TIMEOUT') return 'timeout';
  return 'broken';
}
Enter fullscreen mode Exit fullscreen mode

5. Read robots.txt before the first page

Check every URL against robots.txt with your checker's own user-agent token before you request it. A link to a disallowed URL is not broken, so count it separately. robots.txt also tells you where the sitemaps are:

import robotsParser from 'robots-parser';

async function loadRobots(site) {
  const robotsUrl = new URL('/robots.txt', site).href;
  const res = await fetch(robotsUrl, { headers: { 'user-agent': UA } });
  return robotsParser(robotsUrl, res.ok ? await res.text() : '');
}

const robots = await loadRobots('https://crawler-test.com/');
robots.isAllowed('https://crawler-test.com/', 'MyLinkChecker'); // true
robots.getSitemaps(); // 29 sitemap URLs on crawler-test.com
Enter fullscreen mode Exit fullscreen mode

Don't guess /sitemap.xml: on many sites it does not exist, and the real sitemap has another name listed in robots.txt.

6. Sitemaps: indexes, .gz files and slow servers

Pages that no link points to are still in the sitemap, and comparing the two finds real problems: sitemap URLs that return errors, redirect or are set to noindex, and indexable pages missing from the sitemap. Three details:

  • A sitemap index lists other sitemaps, not pages.
  • A .gz sitemap arrives as gzip bytes. fetch only decompresses Content-Encoding, not a gzipped file, so check the first two bytes.
  • Some sitemaps are slow or missing. Give them a total time budget.
import { gunzipSync } from 'node:zlib';

async function readSitemap(url, urls = new Set(), depth = 0) {
  const res = await fetch(url, { headers: { 'user-agent': UA } });
  if (!res.ok) return urls;
  let body = Buffer.from(await res.arrayBuffer());
  if (body[0] === 0x1f && body[1] === 0x8b) body = gunzipSync(body);
  const xml = body.toString('utf8');
  const locs = [...xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)].map((m) => m[1].replaceAll('&amp;', '&'));
  if (/<sitemapindex/i.test(xml) && depth < 2) {
    for (const loc of locs) await readSitemap(loc, urls, depth + 1);
  } else {
    for (const loc of locs) urls.add(loc);
  }
  return urls;
}
Enter fullscreen mode Exit fullscreen mode

On crawler-test.com two of the sitemaps listed in robots.txt return 404 themselves, and a few are slow on purpose. Reading all of them took 30 seconds.

7. The page title that says "Close"

$('title') matches every <title> element, including the ones inside inline SVG icons, where they serve as accessible labels. On a page without a real title, a naive check reports "Close" or "Menu" instead of flagging the missing title. Skip titles inside <svg>:

function pageChecks(html) {
  const $ = cheerio.load(html);
  const title = $('title')
    .filter((_, el) => $(el).closest('svg').length === 0)
    .first().text().trim();
  return {
    title,
    description: $('meta[name="description"]').attr('content')?.trim() ?? '',
    h1Count: $('h1').length,
    canonical: $('link[rel="canonical"]').attr('href') ?? null,
    noindex: /noindex/i.test($('meta[name="robots"]').attr('content') ?? ''),
    imagesWithoutAlt: $('img:not([alt])').length,
  };
}
Enter fullscreen mode Exit fullscreen mode

Remember the X-Robots-Tag response header too: it can set noindex without any meta tag.

8. Duplicates need the whole site

A duplicate title or meta description is not visible on any single page. Collect the checks for every page, then group by value after the crawl:

function duplicates(pages, field) {
  const groups = new Map();
  for (const p of pages) {
    if (!p[field]) continue;
    groups.set(p[field], [...(groups.get(p[field]) ?? []), p.url]);
  }
  return [...groups].filter(([, urls]) => urls.length > 1);
}
Enter fullscreen mode Exit fullscreen mode

On crawler-test.com four pages share the title "Duplicate title".

9. Four copies of the same site

http:// and https://, with and without www: all four should end on one address. When they don't, search engines see four copies of every page:

async function hostVariants(host) {
  const variants = ['https://', 'http://', 'https://www.', 'http://www.'].map((p) => `${p}${host}/`);
  const finals = await Promise.all(variants.map((u) =>
    fetch(u, { headers: { 'user-agent': UA }, signal: AbortSignal.timeout(10_000) })
      .then((r) => new URL(r.url).origin)
      .catch(() => null)));
  return { finals, singleHost: new Set(finals.filter(Boolean)).size === 1 };
}
Enter fullscreen mode Exit fullscreen mode

python.org sends all four to https://www.python.org. crawler-test.com answers 200 on all four without a redirect, which is one of its deliberate errors.

A full run

Here is everything above put together in one crawler, run on my machine against crawler-test.com with this input:

{
  "startUrls": ["https://crawler-test.com/"],
  "maxPagesPerSite": 60
}
Enter fullscreen mode Exit fullscreen mode

It audited 60 pages and checked every link on them in 90 seconds, 30 of which went to the sitemaps. Part of the summary:

{
  "healthScore": 95,
  "pagesAudited": 60,
  "issueTotals": { "errors": 3, "warnings": 91, "notices": 491 },
  "brokenLinksCount": 65,
  "hosts": { "httpRedirectsToHttps": false, "singleHost": false },
  "aiSearch": { "llmsTxt": false }
}
Enter fullscreen mode Exit fullscreen mode

The 65 broken links are all on the home page, which links to a test page for almost every HTTP status code. 58 of them return a 4xx or 5xx status. The other 7 fail differently: a DNS failure, a redirect loop, a redirect chain that never ends, a reset connection, a failed request and two timeouts. A few rows of the broken links CSV, with the site column left out:

page,broken_url,type,status_code,error,anchor_text
https://crawler-test.com/,https://invalid.crawler-test.com/,external,,DNS_FAILURE,Invalid Subdomain
https://crawler-test.com/,https://crawler-test.com/robots_protocol/non_200_with_noindex,internal,404,,Non-200 with Noindex
https://crawler-test.com/,https://crawler-test.com/redirects/infinite_redirect,internal,303,REDIRECT_LOOP,Infinite Redirect
https://crawler-test.com/,https://crawler-test.com/redirects/redirect_to_404,internal,404,,Redirect To 404 Http Status
https://crawler-test.com/,https://crawler-test.com/status_codes/status_100,internal,,CONNECTION_RESET,100 HTTP Status
Enter fullscreen mode Exit fullscreen mode

The most common issues by number of pages:

Issue Pages
No mobile viewport meta tag 55
Title longer than 60 characters 12
Missing meta description 8
Title shared with another page 6
Set to noindex 4
Missing or empty title 2

The health score is the share of pages without errors: 57 of 60. Warnings and notices, such as a missing lang attribute or Open Graph tags, don't lower it.

Things that kept the run fast and polite

  • Check each URL once. The crawler-test.com home page alone has 408 internal links, and most pages link to the same navigation. Cache results by URL and every page after the first costs a few requests.
  • Limit concurrency per host, not overall. Two to four requests at a time to one site, while external links to other hosts are checked in parallel.
  • Cap the work. A page limit, a limit on link checks per site and a time limit keep a run on a huge site from going on for hours.
  • No browser. Plain HTTP and cheerio need about 1 GB of memory. Earlier, in the cloud, 30 pages of python.org took 23 seconds and about $0.0026 of compute.

All of these checks are in the SEO Audit & Broken Link Checker linked at the top, with the HTML report, the CSV and the 0–100 score.

Top comments (3)

Collapse
 
michael1980 profile image
Toolmaster •

Solid list, and the 403-vs-broken distinction alone removes most false alarms. Three things not covered here that tend to show up on real sites:

  1. <base href>: relative links resolve against it, not against the page URL. Resolving with new URL(href, pageUrl) without checking for a base element reports 404s that don't exist in a browser.
  2. Fragments: /docs#install returns 200 even if id="install" was removed. If anchors matter, check that the target id (or a legacy <a name>) exists in the fetched HTML. And strip the fragment before using the URL as a cache key, otherwise the same page is fetched once per anchor.
  3. Soft 404s: a missing page that answers 200 with a "not found" template. Requesting one random, certainly non-existent path per host and comparing status and title is a cheap signal.

On 429s, honouring Retry-After (seconds or an HTTP date) before re-checking keeps a rate-limited link from being reported as broken.

Collapse
 
makerio profile image
makerio •

Thanks, good additions. Checked them against the actor that came out of this post:

  • <base href>: handled. Links resolve against the first base element when the page has one.
  • Fragments: stripped before a URL is queued or cached, so a page with ten anchors is fetched once. Checking that the target id still exists is not done yet, and on docs sites that is a real gap.
  • Soft 404s: not detected yet. One random path per host is cheap, so that goes on the list.
  • Retry-After: the seconds form is honoured (one retry, capped at 10 s). The HTTP-date form is not parsed yet, so it falls back to a fixed 3 s wait.
Collapse
 
makerio profile image
makerio •