Our IndexNow submitter reads the live sitemap rather than importing it, and the one engine we care about ignores the whole protocol

Notifio is a desktop app that watches rental search pages, and the site that sells it is 38 URLs. You can count them yourself in the sitemap: ten top-level pages, fifteen per-site alert pages, five comparison pages, five guides and three audience pages such as the one for students. Nothing in that list is hand-maintained. The sitemap is derived from the same typed catalogues the pages render from, so adding a site is one edit and it cannot be half-added.

Having a sitemap means waiting to be crawled. IndexNow is the protocol for not waiting: you POST a list of changed URLs and a key, and participating engines come and look. There is a sentence at the top of the module that decides how much to invest in it:

/**
 * IndexNow: telling search engines a URL changed instead of waiting to be
 * crawled again.
 *
 * Google does not participate, so this is strictly additive to the Search
 * Console sitemap rather than a replacement for it. What it does buy is Bing
 * (and therefore Copilot and DuckDuckGo), Yandex, Seznam and Naver picking up
 * an edit in minutes instead of whenever their crawler next decides to look.
 */

The engine that sends most of the traffic ignores all of this. That is the honest framing, and it is also why the whole integration is one dependency-free module plus one script, rather than anything that runs automatically on deploy.

The URL list comes off the live site

/**
 * Pulls the `<loc>` values out of a sitemap.
 *
 * The app's sitemap is generated from the catalogues in `lib/alerts`,
 * `lib/compare` and `lib/articles` through `app/sitemap.ts`, and that module
 * cannot be imported from a plain Node script because of its path aliases.
 * Reading the rendered XML keeps one source of truth for "every URL on the
 * site" instead of a second hand-maintained list that would drift the first
 * time a page was added.
 */
export function parseSitemapUrls(xml: string): string[] {
  const urls: string[] = [];
  for (const match of xml.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)) {
    urls.push(decodeXmlEntities(match[1]));
  }
  return urls;
}

A regex over XML, which is normally a thing to apologise for. Here the XML is generated by Next's own sitemap route from a typed array, the only element being read is <loc>, and the alternative is a second list of every URL on the site maintained by hand. The entity decoder goes to the trouble of replacing & last, so an encoded < does not turn into < on the way through.

The reason for reading the deployed XML rather than importing the module is mundane: app/sitemap.ts imports through @/lib/... aliases that a bare Node script cannot resolve. The consequence is better than the reason. You can only submit URLs that are already live, which is exactly the set of URLs you should be announcing.

The guard that fires most often

Here is the command you would type first, in a fresh checkout, with the repo's own .env loaded:

$ pnpm indexnow --sitemap --dry-run
✗ http://localhost:3001 is not a publicly reachable origin, so the engines could never fetch the key file.
  Pass the live origin explicitly, for example: pnpm indexnow --site https://notifio.app ...

NEXT_PUBLIC_APP_URL is localhost in development, so the default origin for the submission command is the one origin for which a submission can never work. Without that check the mistake arrives as a failed fetch on a key file, several steps later, and reads like a network problem rather than a pointing problem.

export function isPublicHost(host: string): boolean {
  const hostname = host.replace(/:\d+$/, "").toLowerCase();

  if (hostname === "localhost" || hostname.endsWith(".localhost")) return false;
  if (hostname === "::1" || hostname === "[::1]") return false;
  if (hostname.endsWith(".local") || hostname.endsWith(".internal")) return false;

  // Private and loopback IPv4 ranges, per RFC 1918 and RFC 5735.
  const ipv4 = hostname.match(/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/);
  if (ipv4) { /* trimmed here: 127, 10, 0, 192.168, 172.16-31 and 169.254 */ }

  // A bare name with no dot cannot be a registrable domain.
  return hostname.includes(".");
}

With the origin spelled out, the same command does what you wanted:

$ pnpm indexnow --site https://notifio.app --sitemap --dry-run
• Loaded 38 URLs from https://notifio.app/sitemap.xml
• 38 URLs on notifio.app to shared (https://api.indexnow.org/indexnow)
    https://notifio.app/
    https://notifio.app/pricing
    ...
• Dry run: would POST 1 batch. Nothing sent.

One batch, because the protocol's cap is 10,000 URLs per request and we have 38. The batching code exists anyway, with a test that it splits at the cap without losing or reordering anything, because the cheapest time to write that is before you have 10,001 URLs.

One stray URL voids the whole request

const { onHost, offHost } = partitionByHost(resolved, host);
if (offHost.length > 0) {
  // A single foreign URL makes the engines reject the entire batch, so this is
  // fatal rather than a warning: submitting the remainder silently would hide a
  // mistyped hostname behind a successful-looking run.
  die(`These URLs are not on ${host}:\n  ${offHost.join("\n  ")}`);
}

The engines reject the entire submission if any single URL does not belong to the host, which means filtering has to happen before the POST, not after a 422. The choice there is whether a mistyped hostname becomes a dropped URL or a stopped run. It stops the run: a submission that quietly covered 37 of 38 pages and exited 0 is worse than one that refuses to go.

Subdomains count as different hosts, and there is a test that says so in those words, because that is not obvious from the spec and it is exactly the sort of thing that would be "fixed" by someone comparing registrable domains instead.

The status codes lie in both directions

/**
 * Translates an IndexNow response code into what it says about the submission.
 *
 * The codes are not self-explanatory and two of them are actively misleading:
 * 202 looks like a failure but is the normal answer while a key is still being
 * verified, and 200 says nothing about whether the key file was ever fetched.
 */
export function describeStatus(status: number): StatusMeaning {
  switch (status) {
    case 200:
      return { ok: true, retryable: false, message: "URLs submitted" };
    case 202:
      return {
        ok: true, retryable: false,
        message: "accepted, key validation still pending (normal on a new key)",
      };
    case 403:
      return {
        ok: false, retryable: false,
        message: "forbidden: the key file was not found or did not match",
      };
    case 422:
      return {
        ok: false, retryable: false,
        message: "unprocessable: URLs do not belong to the host, or the key does not match",
      };
    case 429:
      return { ok: false, retryable: true, message: "rate limited: too many requests" };

A 200 here means "we have taken your list", not "we have verified you own this host". The failure it hides shows up as a 403 or a 422 on some later run, or as nothing at all. Which is why the submitter checks the key file itself, over the public internet, before every run:

$ pnpm indexnow --site https://notifio.app --verify
✓ https://notifio.app/<key>.txt matches

That check reads the hosted file and compares its contents to the key being submitted, and when it disagrees it says what it found rather than that it failed. A key mismatch is the single most common way this integration breaks, and "the file at this URL says something else" is a far better error than a 403 on a submission.

The key file is committed into public/ rather than generated, because the engines fetch it unauthenticated anyway, so there is no secret to protect, and a file in public/ cannot be forgotten by a deploy that skipped a step.

31 tests, no framework, and no retry loop

The module is tested with node --test and nothing else: 31 cases across 9 groups, no test framework, no mocks beyond a fetchImpl parameter that defaults to fetch. A few of the names are the documentation I actually reach for:

treats 202 as success, since it only means the key is still validating
marks key and host problems as not worth retrying
separates foreign URLs, which would void the whole batch
treats a subdomain as a different host, because the engines do
rejects the local origins NEXT_PUBLIC_APP_URL holds in development
fails, and says what it found, when the key has drifted

And there is no retry loop anywhere in it:

/**
 * POSTs one batch to one endpoint.
 *
 * No retry loop here on purpose: the only retryable code is 429, and the right
 * answer to being rate limited by a best-effort ping is to come back later, not
 * to hammer it inside the same run.
 */

The last piece of advice in the script's own header is the one I have to keep taking:

Submitting the same URL repeatedly with no change to the page is the one documented way to get throttled, so prefer passing the handful of pages a deploy actually touched over running --sitemap out of habit.

Which makes the real usage, after a deploy that edited two pages, pnpm indexnow --site https://notifio.app /pricing /download. The --sitemap flag is for the day the catalogue grows, not for every Tuesday. Google will not have heard any of it either way, and the sitemap it does read is right there where it has always been.