A 200 from IndexNow does not mean it read your key

Notifio's marketing site is a Next.js app whose sitemap currently lists 38 URLs, most of them generated from typed catalogues rather than hand written: a page per rental site we monitor at notifio.app/alerts, a page per competitor at /compare, a guide per question at /guides. Generated pages get edited often, and an edit that nobody crawls is an edit that does not exist yet.

The sitemap tells search engines that a page exists. It does not tell them it changed a minute ago. IndexNow does, and it is about forty lines of protocol, which is exactly why it is easy to get subtly wrong.

Here is what I got wrong, and what the module looks like now.

The protocol, in full

One POST:

{
  "host": "notifio.app",
  "key": "b7989e6abc9208a182d57a642804601a",
  "urlList": ["https://notifio.app/alerts/kamernet"]
}

Plus one file hosted at the root, named after the key and containing the key: notifio.app/b7989e6abc9208a182d57a642804601a.txt. That is the whole thing. The key is public by design, because the engines fetch it unauthenticated to prove you control the host.

Before going further: Google does not participate. This is additive to your Search Console sitemap and replaces nothing. What it buys is Bing, and therefore Copilot and DuckDuckGo, plus Yandex, Seznam and Naver, picking up an edit in minutes instead of whenever their crawler next feels like it.

The status codes do not say what they look like they say

This is the one that cost me an evening. I submitted, got a 200, and assumed the integration worked. It had not been reading my key file at all.

export function describeStatus(status: number): StatusMeaning {
  switch (status) {
    case 200:
      return { ok: true, retryable: false, message: "URLs submitted" };
    case 202:
      return {
        ok: true,
        retryable: false,
        message: "accepted, key validation still pending (normal on a new key)",
      };
    case 400:
      return { ok: false, retryable: false, message: "bad request: invalid format" };
    case 403:
      return {
        ok: false,
        retryable: false,
        message: "forbidden: the key file was not found or did not match",
      };
    case 422:
      return {
        ok: false,
        retryable: false,
        message: "unprocessable: URLs do not belong to the host, or the key does not match",
      };
    case 429:
      return { ok: false, retryable: true, message: "rate limited: too many requests" };
    default:
      return {
        ok: status >= 200 && status < 300,
        retryable: status >= 500,
        message: `unexpected status ${status}`,
      };
  }
}

Two of those are actively misleading if you read them as ordinary HTTP:

• 200 means "we accepted your list". It says nothing about whether your key file was ever fetched. Key problems surface later, as a 403 or a 422.
• 202 looks like a soft failure and is the normal answer while a brand new key is still being verified. Treating it as an error means your first successful submission reads as broken.

Writing that switch out as a function with a message was not ceremony. It is the difference between a deploy script that prints 202 and one that prints accepted, key validation still pending (normal on a new key), which is the sentence I needed at 23:00.

One foreign URL rejects the entire batch

Every URL in a request must live on the same host as host. Not most of them. If one does not, the whole submission is refused, and you find out as a 422 that does not name the offender.

So the filtering has to happen before the request, and the strays are worth reporting rather than silently dropping, because a stray is almost always a typo:

export function partitionByHost(urls: string[], host: string): HostPartition {
  const onHost: string[] = [];
  const offHost: string[] = [];

  for (const url of urls) {
    let parsed: URL;
    try {
      parsed = new URL(url);
    } catch {
      offHost.push(url);
      continue;
    }
    (parsed.host === host ? onHost : offHost).push(url);
  }

  return { onHost, offHost };
}

A malformed URL goes in the same bucket as a foreign one. Both mean "this would have poisoned the batch", which is the only distinction the caller needs.

Check your own key file first

Given that a key mismatch is the most common way this breaks, and that it surfaces as an unexplained 403, the cheapest possible fix is to fetch your own key file before every run and compare it with the key you are about to send:

const body = (await response.text().catch(() => "")).trim();
if (body !== key) {
  const preview = body.length > 60 ? `${body.slice(0, 60)}...` : body || "(empty)";
  return { ok: false, url, message: `contains ${preview}, expected ${key}` };
}

"The file at this URL says (empty), expected b7989e..." is a bug report. A 403 from Bing is a mystery.

This catches the specific failure where you rotate the key in an environment variable, forget that the file in public/ is a separate artifact, and deploy a site whose key file and whose submissions disagree.

The default origin is the one origin that cannot work

NEXT_PUBLIC_APP_URL is http://localhost:3001 in local development. That makes the obvious default for a submission command a host no search engine can reach, and the failure looks like a confusing fetch error on the key file rather than "you are pointing at localhost".

export function isPublicHost(host: string): boolean {
  const hostname = host.replace(/:\d+$/, "").toLowerCase();

  if (hostname === "localhost" || hostname.endsWith(".localhost")) return false;
  if (hostname === "::1" || hostname === "[::1]") return false;
  if (hostname.endsWith(".local") || hostname.endsWith(".internal")) return false;

  const ipv4 = hostname.match(/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/);
  if (ipv4) {
    const [a, b] = ipv4.slice(1, 3).map(Number);
    if (a === 127 || a === 10 || a === 0) return false;
    if (a === 192 && b === 168) return false;
    if (a === 172 && b >= 16 && b <= 31) return false;
    if (a === 169 && b === 254) return false;
    return true;
  }

  return hostname.includes(".");
}

Twenty lines to turn a class of confusing failure into one clear refusal.

The URL list comes from the rendered sitemap

The obvious way to write pnpm indexnow --sitemap is to import the sitemap module and read its array. That does not work here, because app/sitemap.ts uses @/ path aliases and the submission script runs on bare Node.

The tempting workaround is a second list of "every URL on the site" maintained beside the first. That list drifts the first time somebody adds a page, and it drifts silently, because nothing reads it except the thing you are not watching.

So the script fetches the live /sitemap.xml and parses out the <loc> values:

export function parseSitemapUrls(xml: string): string[] {
  const urls: string[] = [];
  for (const match of xml.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)) {
    urls.push(decodeXmlEntities(match[1]));
  }
  return urls;
}

A regex over XML, which is normally a sin. It is fine here because the input is a sitemap we generate ourselves, the target element has no attributes and cannot nest, and the alternative is a parser dependency in a module that is deliberately dependency free so both the Next app and a plain .mjs script can import it.

Deriving the list from the deployed artifact also means the script tells the truth about production rather than about my branch.

No retry loop

The only retryable status is 429, and the entire point of rate limiting a best-effort ping is that you should come back later. So submitBatch does one request and returns what happened. Retrying inside the run would be arguing with the answer.

There is a related trap the docs mention and it is worth repeating: submitting the same unchanged URL over and over is the documented way to get throttled. The script defaults to the pages you name, and --sitemap exists for the rare deploy that genuinely touched everything.

Was it worth it?

For a site whose content is generated from catalogues and edited in batches, yes. The pages that benefit are the long tail: something like /alerts/rightmove or /guides/how-fast-do-rental-listings-go, which are not crawled daily and where a factual correction sitting uncrawled for a fortnight is a real cost.

It is one afternoon of work, most of it spent on error messages rather than on the request. That ratio feels about right for any integration whose failure mode is silence.

Notifio itself is a desktop app that watches rental search pages and tells you the second something new is posted: notifio.app, with the full site list at /alerts and the pricing at /pricing.