DEV Community: Nooralto1

A build step that refuses to ship broken links

Nooralto1 — Fri, 25 Sep 2026 00:12:19 +0000

A build step that refuses to ship broken links

Most teams check for broken links the way they check smoke detector batteries: once a quarter, with a tool that crawls the live site and spits out a spreadsheet nobody reads before the next release. By the time the audit runs, the broken link has been live for weeks and possibly already flagged by a user who did not bother reporting it.

The fix is to move the check earlier and make it a gate instead of a report. Crawl the generated output before it deploys, fail the build on errors that are the site's fault, and warn on the ones that are not.

Crawl the build output, not the live site

Checking the deployed site catches problems after they ship. Checking the static output the build just produced catches them before, and it is faster too: no network latency, no CDN cache to fight, just files on disk.

// scripts/crawl-links.mjs
import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs';
import path from 'node:path';

function collectHtmlFiles(dir, files = []) {
  for (const entry of readdirSync(dir)) {
    const full = path.join(dir, entry);
    if (statSync(full).isDirectory()) {
      collectHtmlFiles(full, files);
    } else if (entry.endsWith('.html')) {
      files.push(full);
    }
  }
  return files;
}

function extractLinksAndIds(html) {
  const links = [...html.matchAll(/href="([^"]+)"/g)].map((m) => m[1]);
  const ids = new Set([...html.matchAll(/\sid="([^"]+)"/g)].map((m) => m[1]));
  return { links, ids };
}

A regex is enough here. The build output is markup this same pipeline generated, not arbitrary HTML from the web, so the shape of href and id attributes is known and consistent.

Internal links are the site's fault, external links are not

The distinction that matters is who can fix the problem. An internal link pointing at a page that does not exist in the build is a bug in the site: someone renamed a route, deleted a page, or typo'd a path. That should fail the build. An external link returning a 404 might be a bug on someone else's server, or a site that restructured its URLs months ago without telling anyone. Worth knowing, but not a reason to block a deploy that has nothing to do with it.

function classifyLink(href, currentFile, distRoot) {
  if (href.startsWith('#')) return { kind: 'anchor', target: currentFile, fragment: href.slice(1) };
  if (/^https?:\/\//.test(href)) return { kind: 'external', url: href };
  if (href.startsWith('mailto:') || href.startsWith('tel:')) return { kind: 'skip' };

  const [pathPart, fragment] = href.split('#');
  const resolved = path.join(path.dirname(currentFile), pathPart || '');
  const candidate = resolved.endsWith('.html') ? resolved : path.join(resolved, 'index.html');
  return { kind: 'internal', target: candidate, fragment };
}

Anchors get checked against the target page's own id set, whether that page is the current one or a different internal page. A link to /pricing#faq is only correct if /pricing/index.html actually contains id="faq".

External checks: cache them, do not hammer someone else's server

Re-fetching every external link on every build is slow and rude. A site with two hundred outbound links does not need two hundred fresh requests for a commit that only fixes a typo in the footer. Cache the result, keyed by URL, and only re-check entries older than a set window.

import { existsSync, readFileSync, writeFileSync } from 'node:fs';

const CACHE_PATH = '.link-cache.json';
const CACHE_TTL_MS = 7 * 24 * 60 * 60 * 1000; // one week

function loadCache() {
  return existsSync(CACHE_PATH) ? JSON.parse(readFileSync(CACHE_PATH, 'utf8')) : {};
}

async function checkExternal(url, cache) {
  const cached = cache[url];
  if (cached && Date.now() - cached.checkedAt < CACHE_TTL_MS) {
    return cached.ok;
  }
  try {
    const res = await fetch(url, { method: 'HEAD', signal: AbortSignal.timeout(8000) });
    const ok = res.ok || res.status === 405; // some servers reject HEAD, treat 405 as inconclusive-pass
    cache[url] = { ok, checkedAt: Date.now() };
    return ok;
  } catch {
    cache[url] = { ok: false, checkedAt: Date.now() };
    return false;
  }
}

A HEAD request is enough to know whether a link resolves, and it costs the remote server far less than a full GET. Persisting the cache between runs, as a committed file or a CI cache artifact, means a flaky link gets flagged once, not on every commit for a week.

Fail on the right category

The crawler walks every page, resolves every link, and separates the two buckets before deciding whether the build lives or dies.

let brokenInternal = 0;
let brokenExternal = 0;

// ... after walking all files and resolving every link:
if (brokenInternal > 0) {
  console.error(`${brokenInternal} broken internal link(s). Failing build.`);
  process.exit(1);
}
if (brokenExternal > 0) {
  console.warn(`${brokenExternal} external link(s) did not resolve. Review before next release.`);
}
process.exit(0);

Wired into CI, this runs after the static build and before the deploy step, so a broken internal link never reaches production in the first place:

# .github/workflows/deploy.yml
- name: Build site
  run: npm run build
- name: Check links
  run: node scripts/crawl-links.mjs dist
- name: Deploy
  run: npm run deploy

The team building this kind of pipeline gate for client sites, including the redirect and link checks that go with a redesign, writes about the broader approach at nooralto.com.

Treating link rot as a build-time concern rather than a quarterly chore changes who finds the problem. Instead of a user hitting a dead page, or an audit six weeks late, the commit that introduced the break never merges.

Written by the team at Nooralto, a web and SEO studio working out of Agadir and Paris.

Built by Nooralto — Nooralto.

Structured data that survives a redesign

Nooralto1 — Fri, 25 Sep 2026 00:06:36 +0000

Structured data that survives a redesign

Structured data breaks in a specific, boring way. Someone hand-writes a <script type="application/ld+json"> block in the page template, ships it, and it works. Six months later the site gets a redesign: the breadcrumb component changes, a category gets renamed, the templating engine moves from server includes to a component framework. Nobody touches the JSON-LD block, because nobody remembers it exists. It keeps rendering, keeps validating as syntactically correct JSON, and keeps describing a page that no longer matches what a visitor sees.

Search Console reports the drift weeks later, if at all, as a drop in a rich result rather than an error. By then it is a mystery to debug instead of a diff to review.

The block is duplicated data, not markup

The underlying problem is that the JSON-LD is a second, hand-maintained copy of information the page already has. The breadcrumb trail exists once, to render the visible nav. It exists again, retyped, inside the schema block. Two copies drift the moment one of them changes and the other does not get the memo.

The fix is to stop writing the block by hand and generate it from the same data the page uses to render. One function, one input, two outputs: the visible breadcrumb nav and the JSON-LD, both built from the same array.

// schema/breadcrumb.js
function buildBreadcrumbSchema(breadcrumbs, baseUrl) {
  return {
    '@context': 'https://schema.org',
    '@type': 'BreadcrumbList',
    itemListElement: breadcrumbs.map((crumb, index) => ({
      '@type': 'ListItem',
      position: index + 1,
      name: crumb.label,
      item: new URL(crumb.path, baseUrl).toString(),
    })),
  };
}

module.exports = { buildBreadcrumbSchema };

breadcrumbs here is the exact array the nav component already receives as a prop. Rename a category and both outputs update from the same edit. There is no second place to remember.

Validate the shape before it ships

Generating the block from real data removes typos, but it does not stop someone from passing an empty array, or a crumb with no label, or a path that resolves to a relative URL instead of an absolute one. BreadcrumbList expects itemListElement as a non-empty list of ListItem entries, each with position, name, and item. A build step that checks that shape catches a broken page before it deploys instead of after Search Console notices.

// schema/validate.js
function assertValidBreadcrumbSchema(schema) {
  if (schema['@type'] !== 'BreadcrumbList') {
    throw new Error('expected @type BreadcrumbList');
  }
  if (!Array.isArray(schema.itemListElement) || schema.itemListElement.length === 0) {
    throw new Error('itemListElement must be a non-empty array');
  }
  schema.itemListElement.forEach((item, i) => {
    if (item['@type'] !== 'ListItem') throw new Error(`item ${i}: wrong @type`);
    if (typeof item.position !== 'number') throw new Error(`item ${i}: position must be a number`);
    if (typeof item.name !== 'string' || item.name.length === 0) {
      throw new Error(`item ${i}: missing name`);
    }
    if (typeof item.item !== 'string' || !/^https?:\/\//.test(item.item)) {
      throw new Error(`item ${i}: item must be an absolute URL`);
    }
  });
}

module.exports = { assertValidBreadcrumbSchema };

Call it right after buildBreadcrumbSchema, in the same build script that writes the page out. A page with a broken schema should fail the build, not reach staging.

Test the rendered output, not the generator

Unit-testing the generator function proves the function works. It does not prove the schema actually lands on the page, survives the templating layer, and matches what a crawler would parse out of the final HTML. That needs a test against the rendered output itself.

// test/breadcrumb-schema.test.js
const { test } = require('node:test');
const assert = require('node:assert/strict');
const { renderPage } = require('../render');

test('category page ships a valid BreadcrumbList', () => {
  const html = renderPage({
    type: 'category',
    slug: 'hiking-boots',
    breadcrumbs: [
      { label: 'Home', path: '/' },
      { label: 'Footwear', path: '/footwear' },
      { label: 'Hiking boots', path: '/footwear/hiking-boots' },
    ],
  });

  const match = html.match(/<script type="application\/ld\+json">([\s\S]*?)<\/script>/);
  assert.ok(match, 'no JSON-LD script tag found in rendered HTML');

  const schema = JSON.parse(match[1]);
  assert.equal(schema['@type'], 'BreadcrumbList');
  assert.equal(schema.itemListElement.length, 3);
  assert.equal(schema.itemListElement[2].name, 'Hiking boots');
  for (const item of schema.itemListElement) {
    assert.equal(typeof item.position, 'number');
    assert.ok(item.name.length > 0);
    assert.match(item.item, /^https:\/\//);
  }
});

This is the test that actually catches a redesign breaking things. If a future template swap drops the <script> tag entirely, or renames the prop the generator reads from, this fails on the next commit instead of on the next Search Console crawl.

Run one of these per page type that carries a schema block: product, article, FAQ, organization. Each type has its own required fields, so each gets its own assertions rather than one generic check.

Watch it after deploy too

A build-time test only proves the schema was correct at build time. A CDN caching an old bundle, a canary rollout serving two versions, or a third-party script mutating the DOM after load can still put stale or missing structured data in front of real crawlers. A lightweight periodic check, fetching a handful of key pages in production and re-running the same validator against whatever JSON-LD comes back, closes that gap without needing a full monitoring platform.

We wire this kind of build-time validation into new builds at Nooralto, so the schema block stops being a thing someone remembers to update by hand.

Structured data earns its keep when it is generated, not authored. Treat it as a projection of the page's own data, test that projection like any other output, and a redesign stops being a threat to it.

Written by the team at Nooralto, a web and SEO studio working out of Agadir and Paris.

Built by Nooralto — Nooralto.

Why your pages are crawled but not indexed

Nooralto1 — Fri, 25 Sep 2026 00:01:18 +0000

Why your pages are crawled but not indexed

"Crawled, currently not indexed" in Search Console gets treated as one problem with one fix,
usually "publish better content." The status covers several distinct failure modes, and content
quality is only one of them. The other three are almost always visible in the server response
itself, if anyone actually diffs what Googlebot receives against what a browser receives.

What "crawled but not indexed" actually means

Google fetched the URL, read the response, and made a deliberate decision not to add it to the
index. That is different from "discovered, not crawled", which is a budget or robots issue, and
different from a canonical replacing the URL with another one Google chose to index instead. The
Search Console UI collapses these into similar-looking rows. The API is more precise, and it is the
only practical way to check hundreds of URLs without clicking through the UI one at a time.

Querying status through the API

The URL Inspection API exposes the same data the UI shows for a single URL, built for automation.
Each call needs the property's verified siteUrl and the full inspectionUrl, sent to
urlInspection.index.inspect, which maps to POST /v1/urlInspection/index:inspect on the Search
Console API.

const { google } = require('googleapis');

async function inspectUrl(auth, siteUrl, inspectionUrl) {
  const searchconsole = google.searchconsole({ version: 'v1', auth });
  const res = await searchconsole.urlInspection.index.inspect({
    requestBody: { siteUrl, inspectionUrl },
  });
  const result = res.data.inspectionResult.indexStatusResult;
  return {
    url: inspectionUrl,
    verdict: result.verdict,               // PASS, PARTIAL, FAIL, NEUTRAL
    coverageState: result.coverageState,    // e.g. "Crawled - currently not indexed"
    lastCrawlTime: result.lastCrawlTime,
    googleCanonical: result.googleCanonical,
    userCanonical: result.userCanonical,
  };
}

The quota on this endpoint is small, so a full-site sweep needs to run as a slow batch rather than
a loop firing as fast as the client allows. Space the calls, and cache the results, since a page's
status does not move minute to minute.

The two canonical fields are the first thing worth comparing. When googleCanonical and
userCanonical disagree, Google decided the site's own canonical tag was wrong and picked a
different URL to index in its place. That alone explains a large share of "crawled, not indexed"
rows, and it needs no further diagnosis beyond fixing the tag.

When the tag is correct but the server still lies

If the canonical fields agree and the page still is not indexed, the next check is whether
Googlebot and an ordinary visitor are looking at the same document. Some setups serve different
HTML by user agent, deliberately or by accident, through a caching layer or a bot-protection rule
that treats crawlers differently from everyone else.

#!/usr/bin/env bash
# Compare what Googlebot fetches against what a browser fetches
URL="$1"

curl -s -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -o googlebot.html -w "googlebot: %{http_code}\n" "$URL"

curl -s -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
  -o browser.html -w "browser:   %{http_code}\n" "$URL"

diff <(grep -o '<title>.*</title>' googlebot.html) \
     <(grep -o '<title>.*</title>' browser.html)

A status code mismatch is easy to catch this way. A content mismatch hiding inside two identical
200 responses is the one that survives for months, because nothing in the logs looks like an
error.

The failure that produces no error at all

JavaScript-rendered content is the quieter version of the same problem. Googlebot fetches the
initial HTML, queues the page for rendering, executes the scripts, and only then sees the final
DOM. If a script throws, if a third-party dependency times out, or if the content depends on a
client-side fetch that the render sandbox blocks, the rendered page ends up thinner than the one a
visitor sees, with an HTTP 200 at every step along the way. There is no error to alert on. The
only way to catch it is to render the page the way the crawler does and diff the result against the
server-side output, the same comparison as above, run against the rendered DOM rather than the raw
response.

Redirect chains as a slow leak

A URL that resolves through two or three redirects before reaching its final destination usually
still gets indexed, but it competes for crawl budget against everything else on the domain, and
each additional hop is another place for a canonical or a status code to quietly diverge from what
was intended. Checking chain length is a short loop built on curl -I --location-trusted with
manual redirect handling, and it is worth running across the whole URL list on the same cadence as
the nooralto.com technical audits that catch
this alongside the canonical and rendering checks above.

Reading the four checks together

None of these checks needs the content rewritten first. Canonical agreement, user-agent parity,
rendered-DOM parity and redirect chain length are structural facts about the response, and they
answer whether Google is capable of indexing the page before the conversation moves to whether the
page deserves to be. Running the content-quality argument before ruling these out wastes a sprint
on the wrong hypothesis.

Written by the team at Nooralto, a web and SEO studio working out of Agadir and Paris.

Internal linking as a data structure

Nooralto1 — Fri, 25 Sep 2026 00:01:05 +0000

Internal linking as a data structure

Most internal linking work happens by hand: an editor remembers a related page and drops a link
at the bottom of a post. That works until the site passes a few hundred URLs, at which point
nobody remembers what links to what, and pages start going dark without anyone noticing.

The fix is to stop treating internal links as content and start treating them as a graph. Every
URL is a node. Every <a href> pointing at another URL on the same domain is a directed edge.
Once the site is represented that way, orphan pages, link depth and missing connections become
computable properties instead of things an editor has to remember.

Building the node list from the sitemap

The sitemap is the fastest way to get every URL the site claims exists, but it does not tell you
what links to what. Parse it first to get the full node set, then crawl the HTML to get the edges.

import { XMLParser } from 'fast-xml-parser';

async function loadSitemapUrls(sitemapUrl) {
  const xml = await fetch(sitemapUrl).then(r => r.text());
  const parser = new XMLParser();
  const doc = parser.parse(xml);
  const entries = doc.urlset?.url ?? [];
  const list = Array.isArray(entries) ? entries : [entries];
  return list.map(e => new URL(e.loc).pathname);
}

If the sitemap is an index of sitemaps, recurse into sitemapindex.sitemap the same way before
returning the flat list.

Building the edge list from the HTML

For each URL, fetch the rendered HTML (or the pre-render output, if the site is a SPA) and extract
every internal anchor. This is where most homegrown link audits stop, at a flat list of "page X
links to page Y". That list is already an adjacency table; the next steps just query it properly.

import * as cheerio from 'cheerio';

async function extractLinks(pathname, baseUrl) {
  const html = await fetch(new URL(pathname, baseUrl)).then(r => r.text());
  const $ = cheerio.load(html);
  const targets = new Set();
  $('a[href]').each((_, el) => {
    const href = $(el).attr('href');
    try {
      const u = new URL(href, baseUrl);
      if (u.origin === baseUrl && !targets.has(u.pathname)) {
        targets.add(u.pathname);
      }
    } catch { /* mailto:, tel:, javascript: ignored */ }
  });
  return [...targets];
}

async function buildGraph(urls, baseUrl) {
  const adjacency = new Map(urls.map(u => [u, []]));
  for (const url of urls) {
    const links = await extractLinks(url, baseUrl);
    adjacency.set(url, links.filter(l => adjacency.has(l)));
  }
  return adjacency;
}

The filter on the last line matters: keep only edges that land on a node already in the sitemap.
Links to unlisted URLs are a separate problem, usually a stale sitemap or a page that was deleted
without removing its incoming links.

Depth: a breadth-first search, nothing more

"How many clicks from the home page" is the standard proxy for how much authority a page is
likely to receive, and it is a textbook BFS.

function computeDepth(adjacency, root = '/') {
  const depth = new Map([[root, 0]]);
  const queue = [root];
  while (queue.length) {
    const current = queue.shift();
    for (const next of adjacency.get(current) ?? []) {
      if (!depth.has(next)) {
        depth.set(next, depth.get(current) + 1);
        queue.push(next);
      }
    }
  }
  return depth;
}

Any node from the sitemap that never appears in depth after this runs is unreachable by internal
links alone, no matter what the sitemap says about it. That is a stronger signal than a manual page
review will ever give: the page exists, but nothing on the site walks a visitor, or a crawler, to
it.

Orphans and isolated clusters

An orphan page is any node absent from depth. A subtler failure is a connected component that
never touches the root at all: a cluster of pages linking to each other whose only inbound link
came from a page that is itself unreachable.

function findOrphans(allUrls, depth) {
  return allUrls.filter(u => !depth.has(u));
}

Category pages and old campaign landing pages tend to end up here after a redesign quietly drops
the navigation block that used to reach them.

Generating links instead of placing them

Once the graph exists, the interesting move is to stop writing internal links by hand and generate
the missing ones at build time, from data rather than editorial memory: match each page's tags or
category against a candidate pool, cap how many new links a page can receive per build, and never
touch a page that already has structural exits from its template. The Nooralto
approach on client sites runs this as a build step, so the link set is versioned alongside the
content instead of living in an editor's head.

What to check on a schedule

Depth and orphan detection are cheap enough to run on every deploy, not once a year. A page that
used to sit three clicks from the home page and now sits at six, because a template change removed
a related-posts block, is a regression like any other, and it should fail a build the same way a
broken test does.

Written by the team at Nooralto, a web and SEO studio working out of Agadir and Paris.