Internal linking as a data structure

Most internal linking work happens by hand: an editor remembers a related page and drops a link
at the bottom of a post. That works until the site passes a few hundred URLs, at which point
nobody remembers what links to what, and pages start going dark without anyone noticing.

The fix is to stop treating internal links as content and start treating them as a graph. Every
URL is a node. Every <a href> pointing at another URL on the same domain is a directed edge.
Once the site is represented that way, orphan pages, link depth and missing connections become
computable properties instead of things an editor has to remember.

Building the node list from the sitemap

The sitemap is the fastest way to get every URL the site claims exists, but it does not tell you
what links to what. Parse it first to get the full node set, then crawl the HTML to get the edges.

import { XMLParser } from 'fast-xml-parser';

async function loadSitemapUrls(sitemapUrl) {
  const xml = await fetch(sitemapUrl).then(r => r.text());
  const parser = new XMLParser();
  const doc = parser.parse(xml);
  const entries = doc.urlset?.url ?? [];
  const list = Array.isArray(entries) ? entries : [entries];
  return list.map(e => new URL(e.loc).pathname);
}

If the sitemap is an index of sitemaps, recurse into sitemapindex.sitemap the same way before
returning the flat list.

Building the edge list from the HTML

For each URL, fetch the rendered HTML (or the pre-render output, if the site is a SPA) and extract
every internal anchor. This is where most homegrown link audits stop, at a flat list of "page X
links to page Y". That list is already an adjacency table; the next steps just query it properly.

import * as cheerio from 'cheerio';

async function extractLinks(pathname, baseUrl) {
  const html = await fetch(new URL(pathname, baseUrl)).then(r => r.text());
  const $ = cheerio.load(html);
  const targets = new Set();
  $('a[href]').each((_, el) => {
    const href = $(el).attr('href');
    try {
      const u = new URL(href, baseUrl);
      if (u.origin === baseUrl && !targets.has(u.pathname)) {
        targets.add(u.pathname);
      }
    } catch { /* mailto:, tel:, javascript: ignored */ }
  });
  return [...targets];
}

async function buildGraph(urls, baseUrl) {
  const adjacency = new Map(urls.map(u => [u, []]));
  for (const url of urls) {
    const links = await extractLinks(url, baseUrl);
    adjacency.set(url, links.filter(l => adjacency.has(l)));
  }
  return adjacency;
}

The filter on the last line matters: keep only edges that land on a node already in the sitemap.
Links to unlisted URLs are a separate problem, usually a stale sitemap or a page that was deleted
without removing its incoming links.

Depth: a breadth-first search, nothing more

"How many clicks from the home page" is the standard proxy for how much authority a page is
likely to receive, and it is a textbook BFS.

function computeDepth(adjacency, root = '/') {
  const depth = new Map([[root, 0]]);
  const queue = [root];
  while (queue.length) {
    const current = queue.shift();
    for (const next of adjacency.get(current) ?? []) {
      if (!depth.has(next)) {
        depth.set(next, depth.get(current) + 1);
        queue.push(next);
      }
    }
  }
  return depth;
}

Any node from the sitemap that never appears in depth after this runs is unreachable by internal
links alone, no matter what the sitemap says about it. That is a stronger signal than a manual page
review will ever give: the page exists, but nothing on the site walks a visitor, or a crawler, to
it.

Orphans and isolated clusters

An orphan page is any node absent from depth. A subtler failure is a connected component that
never touches the root at all: a cluster of pages linking to each other whose only inbound link
came from a page that is itself unreachable.

function findOrphans(allUrls, depth) {
  return allUrls.filter(u => !depth.has(u));
}

Category pages and old campaign landing pages tend to end up here after a redesign quietly drops
the navigation block that used to reach them.

Generating links instead of placing them

Once the graph exists, the interesting move is to stop writing internal links by hand and generate
the missing ones at build time, from data rather than editorial memory: match each page's tags or
category against a candidate pool, cap how many new links a page can receive per build, and never
touch a page that already has structural exits from its template. The Nooralto
approach on client sites runs this as a build step, so the link set is versioned alongside the
content instead of living in an editor's head.

What to check on a schedule

Depth and orphan detection are cheap enough to run on every deploy, not once a year. A page that
used to sit three clicks from the home page and now sits at six, because a template change removed
a related-posts block, is a regression like any other, and it should fail a build the same way a
broken test does.

Written by the team at Nooralto, a web and SEO studio working out of Agadir and Paris.

Comments

[Launch Gate]

Treating links as a graph is the right model. Orphans and depth fall out of it almost for free.

One thing I'd change: the filter at the end of buildGraph throws away the most useful edges, the links that point to paths not in the sitemap. Those are usually broken links (typos, deleted pages), old URLs that now redirect, or real pages someone forgot to add to the sitemap. I'd keep them in a separate list and send a HEAD request to each one. The 404s and 301s from that list are often the first fixes worth making.

Also worth normalizing before building nodes: trailing slash, lowercase, and stripping ?utm_* and #fragments. Otherwise /pricing and /pricing/ show up as two nodes, and one of them looks like an orphan.