Let the sitemap decide which URLs your bot is allowed to announce

The problem

I run about 20 small content sites that update themselves daily, and a bot announces each day's new articles on X. Every so often the announcement pointed at a URL that returned 404.

The cause was easy to find. The bot built announcement URLs from my own bookkeeping. The daily job writes "today I added this slug" (a slug being the short string that identifies an article) into state.json, and the bot glued that slug onto the site's base URL.

That is a lie in disguise. The ledger proves that the job intended to add a page. It does not prove the page is live. If the build failed halfway, or the deploy did not finish, the ledger row is still there. An announcement must not trust the ledger.

So I changed the rule: the bot may only announce URLs that appear as a <loc> in the site's own sitemap.xml.

The design

sitemap.xml is the site declaring, publicly, which pages exist. It is meant for search engines, but as a list of live pages it is far more trustworthy than my ledger. It is a build artifact, so a failed build leaves it stale, and an undeployed build never reaches production at all.

The pipeline became:

1. Take today's slugs from the ledger (state.json).
2. Fetch https://<site>/sitemap.xml and parse it into a list of <loc> values.
3. Find a <loc> containing the slug. Only those become announcement candidates.
4. If a site resolves zero candidates, report the count and drop the site.

Step 3 failing means "in the ledger, not published." Do not announce it.

The core of it

Resolving a slug to a real URL is a plain substring match. locs below is the array of URLs pulled from the sitemap:

_resolveSlugUrl(slug, locs) {
  if (!slug) return null;
  const matches = locs.filter(u => u.includes(slug));
  if (matches.length === 0) return null;
  // prefer the article itself over /tags/ and other index pages
  const preferred = matches.find(u => !/\/tags?\//i.test(u));
  return preferred || matches[0];
}

Returning null on no match is the whole point. The tempting "helpful" version — fall back to building the URL from the base path — puts you right back where you started.

The /tags/ branch came out of real runs. Most of these sites publish /tags/<same word as the slug>, so the tag index matches before the article does. Tag pages return 200, so a pure liveness check cannot reject them. They exist; they are just not what I meant to announce.

Where it broke

One site resolved zero candidates every single day while publishing normally. Its article URLs share no characters with the ledger index. The ledger holds CVE (Common Vulnerabilities and Exposures — the globally unique identifier assigned to a single known security flaw) numbers, while the public URLs look like /article/post_<epoch_ms>_<random>. Measured on 2026-08-20: of 1338 <loc> entries, 0 contained a CVE ID.

The mechanism was behaving exactly as specified. It was also silently discarding a site that published every day.

The fix had two parts. First, count what you drop: sites that ended with zero candidates now show up in the report as {key:"ai-news", added:1, tried:3} — "one article added, three strategies tried, nothing resolved." That line is what made the problem visible at all. Second, add a fallback that reads <lastmod> (each page's last-modified timestamp) and treats same-day entries as today's publications — with a guard:

const dated = entries.filter((e) => e.lastmod && lastmodJstDate(e.lastmod) === logDate);
if (dated.length === 0) return [];
/* If *every* page carries today's lastmod, the site regenerated itself.
 * That is not the same as "these pages were written today." */
if (dated.length / entries.length > 0.5) return [];

One site regenerates all 26 of its pages daily, so every lastmod is today's date. Calling those "new articles" would be a lie to readers, hence the ratio guard. The CVE site, by contrast, had exactly one same-day lastmod, matching the ledger's "1 added."

There was also a timezone bug worth naming. <lastmod> is UTC (2026-09-04T21:23:46.517Z), my calendar day is JST, and I was comparing slice(0, 10) of both. Articles published between 00:00 and 09:00 JST never matched. Announcements silently vanished on nights the job ran late — a wonderfully hard bug to reproduce on purpose.

Liveness, in three states

The final check does an actual HTTP request, and its verdict is deliberately not a boolean:

const evaluate = (status) => ({
  ok: status >= 200 && status < 400,
  status,
  definitive: status >= 400 && status < 600,   // genuinely broken
  reason: status >= 200 && status < 400 ? 'ok' : `http-${status}`
});

definitive is set only when a numeric 4xx/5xx came back. A timeout or DNS failure leaves it false. "Broken" and "could not determine" must not collapse into one value, or a transient network blip permanently evicts a perfectly good page — quietly, which is the worst part.

One of the live sites: https://manga.autoarticles.net

Takeaway

Derive outward-facing correctness from the published artifact, never from your own records. Your records prove intent; the sitemap proves publication. And whenever you add a filter, count what it removes and print that count — a rejection nobody counts is indistinguishable from a rejection that never happened.

---

This article is about my own side project. It was written with AI assistance.