Two checks that stop a bot from announcing a 404

Background

The most embarrassing thing an update announcement can do is link to a 404. A typo in a slug, or a URL that has not been deployed yet, and the link is dead.

When a bot posts those announcements, it does not get embarrassed. It just keeps producing broken links on schedule.

So the pipeline runs two independent checks before anything is posted. The important part is not either check on its own — it is that they answer two different questions, and are deliberately not merged.

How it works

The pipeline runs the two checks at two different moments, as sketched below.

[1] at generation time  -- does this URL exist?
      the site's sitemap.xml is the list of published URLs.
      a URL that is not in it never becomes an announcement.

[2] just before posting -- can I reach it right now?
      hit the URL over HTTP and check for 2xx/3xx,
      while distinguishing "broken" from "temporarily unreachable".

Checking only over HTTP is fragile right after a deploy: the page is correct, the propagation just has not finished. Checking only the sitemap cannot tell you that something has since gone missing. Each check covers the other's blind spot, so both run, for different reasons.

Implementation

The sitemap is the source of truth for existence

Every site emits sitemap.xml at build time — that is, by construction, the list of URLs that will be published. Matching against it needs no dependency and no XML parser:

const fs = require("fs");

function loadSitemapUrls(path) {
  const xml = fs.readFileSync(path, "utf8");
  const set = new Set();
  for (const m of xml.matchAll(/<loc>([^<]+)<\/loc>/g)) {
    set.add(m[1].trim().replace(/\/$/, "")); // normalize the trailing slash
  }
  return set;
}

function assertInSitemap(url, urls) {
  const key = url.trim().replace(/\/$/, "");
  if (!urls.has(key)) throw new Error("URL not in sitemap (may be unpublished): " + url);
  return url;
}

const urls = loadSitemapUrls("public/sitemap.xml");
assertInSitemap("https://example.net/articles/foo", urls); // throws if absent

Normalizing the trailing slash on both sides is what makes the comparison usable; a sitemap that writes /foo/ and a generator that writes /foo will otherwise disagree about every URL.

Put assertInSitemap at the entrance of the announcement generator and a post pointing at a non-existent URL becomes structurally impossible to build.

Reachability distinguishes "broken" from "unknown"

The pre-post check tries HEAD first, falls back to GET, and sorts the outcome into three buckets. Collapsing that into "anything but 200 is bad" throws away correct URLs during deploy propagation.

async function checkUrl(url, { timeoutMs = 8000, client = axios } = {}) {
  if (!/^https?:\/\//i.test(url))
    return { ok: false, definitive: true, reason: "invalid-url" };

  const opts = { timeout: timeoutMs, maxRedirects: 5, validateStatus: () => true };
  const evaluate = (status) => ({
    ok: status >= 200 && status < 400,
    status,
    definitive: status >= 400 && status < 600, // <- unambiguously broken
    reason: status < 400 ? "ok" : `http-${status}`,
  });

  try {
    const head = await client.head(url, opts);
    // codes that commonly mean "HEAD not supported" -- re-check with GET
    if ([403, 405, 501].includes(head.status)) return evaluate((await client.get(url, opts)).status);
    return evaluate(head.status);
  } catch (headErr) {
    try {
      return evaluate((await client.get(url, opts)).status); // HEAD failed -> one GET
    } catch (getErr) {
      // status unknown = possibly transient. never exclude permanently
      return { ok: false, definitive: false, reason: getErr.code || "network-error" };
    }
  }
}

How the result is used is the whole point:

result	meaning	action
ok: true	reachable, 2xx/3xx	safe to announce
definitive: true	4xx/5xx, clearly broken	exclude permanently
definitive: false	timeout or unknown network state	leave in the queue, retry next run

Keeping definitive separate is what lets genuinely dead URLs be dropped forever while propagation lag and brief outages get picked up on the next scheduled run.

The two paths are visible in the logs, which is exactly what you want when reviewing a run:

Skip posting (broken link http-404): https://example.net/articles/typo
Defer posting (url unreachable now: ETIMEDOUT): https://example.net/articles/new-post
Done. Posted 3/5 due tweets (skipped 2 for unreachable/broken links).

The first line is a permanent exclusion. The second is not — that item is still queued and will be tried again.

Gotchas

• Trailing slashes. If the sitemap and the generator disagree about /, every comparison fails. Normalize both before comparing.
• Do not let deploy lag leak into the sitemap check. Existence is answered by the build output; "is it up right now" is answered by HTTP. Keeping the two questions apart is what makes both of them stable.
• HEAD is not universally supported. A CDN (the cache layer sitting in front of an origin) or a WAF (a web application firewall that filters requests) may answer 403, 405, or 501 to a perfectly good URL. Always keep a GET fallback.
• Never exclude permanently on a timeout. Dropping a correct URL because of a transient failure means it is never announced again. That is precisely the case definitive: false exists to prevent.

There is one more trap in the same family, one directory over. A separate job compares the number of URLs in the live sitemap against what Search Console reports. Sitemaps can be index files pointing at child sitemaps, and if a child fetch fails, returning the partial total would silently show up as "Search Console is behind." So the counter returns nothing at all when any child cannot be read:

const children = locsOf(xml).slice(0, MAX_CHILDREN);
let total = 0;
for (const c of children) {
  const childXml = await getText(c, fetchImpl); // on failure -> catch (never return a partial count)
  total += locsOf(childXml).length;
}

Same principle as definitive: false: an incomplete answer must not be presented as a complete one.

The result

One of the sites whose update announcements run through both checks: https://manga.autoarticles.net

Wrap-up

Broken links in automated announcements mostly disappear once you do three things:

1. Guarantee existence with the sitemap, at generation time.
2. Confirm reachability over HTTP, immediately before posting.
3. Never merge the two.

On the reachability side, the split between "broken" and "unknown" is what makes the whole thing tolerant of deploy lag. It comes to well under a hundred lines in total, and after it, an embarrassing 404 announcement stops being something that can happen.

---

This article is about my own side project. It was written with AI assistance.