A 200 from IndexNow does not mean it read your key

I have now wired IndexNow into two products, and the interesting thing is that the second one looks nothing like the first.

The first was IndexNow, but only when the page actually changed: a fingerprint of every page's source data in Redis, and a route that submits only the URLs whose fingerprint moved. That post was about when to submit. This one is about everything that goes wrong once you have decided to.

Why the second one is a command, not a cron

Notifio has 38 URLs in its sitemap, and they change when I sit down and change them. There is no generated-content pipeline whose output can shift under me. So the whole integration is a CLI I run after a deploy that touched copy:

 * Usage (from server/):
 *   pnpm indexnow <url|path>...        Submit specific pages
 *   pnpm indexnow --sitemap            Submit every URL in /sitemap.xml
 *   pnpm indexnow --init               Create a key and its public key file
 *   pnpm indexnow --verify             Only check the hosted key file
 *
 * Submitting the same URL repeatedly with no change to the page is the one
 * documented way to get throttled, so prefer passing the handful of pages a
 * deploy actually touched over running --sitemap out of habit.

The lesson from the first build survives, it just lives in the usage text instead of in a Redis hash. --sitemap exists because it is genuinely right once, on the first run, when every page is new to the engines.

The response codes are worse than undocumented, they are misleading

This is the function I would most like to hand to anyone starting an IndexNow integration:

/**
 * Translates an IndexNow response code into what it says about the submission.
 *
 * The codes are not self-explanatory and two of them are actively misleading:
 * 202 looks like a failure but is the normal answer while a key is still being
 * verified, and 200 says nothing about whether the key file was ever fetched.
 */
export function describeStatus(status: number): StatusMeaning {
  switch (status) {
    case 200:
      return { ok: true, retryable: false, message: "URLs submitted" };
    case 202:
      return { ok: true, retryable: false,
        message: "accepted, key validation still pending (normal on a new key)" };
    case 400:
      return { ok: false, retryable: false, message: "bad request: invalid format" };
    case 403:
      return { ok: false, retryable: false,
        message: "forbidden: the key file was not found or did not match" };
    case 422:
      return { ok: false, retryable: false,
        message: "unprocessable: URLs do not belong to the host, or the key does not match" };
    case 429:
      return { ok: false, retryable: true, message: "rate limited: too many requests" };
    default:
      return { ok: status >= 200 && status < 300, retryable: status >= 500,
        message: `unexpected status ${status}` };
  }
}

Two of those deserve saying out loud.

200 means "we accepted your list". It does not mean your key was fetched, your ownership was proven, or anything at all about whether those URLs will be crawled. If you log 200 OK and walk away, you can run a broken integration for months and see nothing but green.

202 looks like something went wrong and is usually the healthiest answer you will get. It is what a new key returns while validation is pending. Treating it as a failure means your first ever submission reports an error, which is exactly when you are most likely to start "fixing" things that are not broken.

So ok and retryable are separate fields. 403 and 422 are failures you must not retry, because the identical request will fail identically until a human changes something. 429 is the only code where coming back later makes sense, and submitBatch deliberately has no retry loop:

/**
 * No retry loop here on purpose: the only retryable code is 429, and the right
 * answer to being rate limited by a best-effort ping is to come back later, not
 * to hammer it inside the same run.
 */

Check the key yourself, before every run

Because a 200 tells you nothing about the key, the script asks the question directly instead:

/**
 * Confirms the hosted key file exists and matches the key being submitted.
 *
 * Worth doing before every run: a key mismatch is the single most common way
 * this integration breaks, and it fails as a 403 on the submission itself,
 * which is far harder to read than "the file at this URL says something else".
 */
export async function checkKeyFile(origin, key, fetchImpl = fetch): Promise<KeyFileCheck> {
  // ...
  const body = (await response.text().catch(() => "")).trim();
  if (body !== key) {
    const preview = body.length > 60 ? `${body.slice(0, 60)}...` : body || "(empty)";
    return { ok: false, url, message: `contains ${preview}, expected ${key}` };
  }
  return { ok: true, url, message: "matches" };
}

contains abc..., expected def... is a diagnosis. 403 Forbidden is a puzzle. The two cost the same to implement.

One foreign URL voids the whole request

This is the protocol rule that is easiest to miss and most expensive to learn: every URL in a submission must be on the same host as the host field, and if one is not, the engine rejects all of them. Not the stray. All of them.

/**
 * Splits URLs by whether they belong to the submitting host.
 *
 * A single foreign URL makes an engine reject every URL in the request, so the
 * strays are separated out and reported rather than silently dropped: a typo in
 * a hostname is worth seeing.
 */
export function partitionByHost(urls: string[], host: string): HostPartition {
  const onHost: string[] = [];
  const offHost: string[] = [];
  for (const url of urls) {
    let parsed: URL;
    try { parsed = new URL(url); } catch { offHost.push(url); continue; }
    (parsed.host === host ? onHost : offHost).push(url);
  }
  return { onHost, offHost };
}

Note that this filters before submitting rather than interpreting a 422 afterwards, and that it returns the rejects instead of dropping them. Silently discarding a URL because it did not match the host is how you spend an afternoon wondering why one page never gets announced, when the real answer is that its hostname has a typo in it. A subdomain is a different host too, as far as the engines are concerned, and there is a test that pins that down so nobody "helpfully" relaxes it later.

The URL list comes from the rendered sitemap, not from the code that renders it

The obvious implementation of --sitemap is to import the app's sitemap.ts and read the array. I parse the served XML instead:

/**
 * The app's sitemap is generated from the catalogues in `lib/alerts`,
 * `lib/compare` and `lib/articles` through `app/sitemap.ts`, and that module
 * cannot be imported from a plain Node script because of its path aliases.
 * Reading the rendered XML keeps one source of truth for "every URL on the
 * site" instead of a second hand-maintained list that would drift the first
 * time a page was added.
 */
export function parseSitemapUrls(xml: string): string[] {
  const urls: string[] = [];
  for (const match of xml.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)) {
    urls.push(decodeXmlEntities(match[1]));
  }
  return urls;
}

It started as a workaround: the script runs on bare Node and cannot resolve @/ path aliases. It turned out to be the better design anyway. The thing I want to announce is the set of URLs the site actually serves, and the sitemap at notifio.app/sitemap.xml is that set by definition. A parallel list built from the same catalogues would be a second answer to the same question, and one of the two would eventually be wrong.

The entity decoder has the one ordering bug worth knowing about:

  .replace(/"/g, '"')
  .replace(/'/g, "'")
  // Ampersand last, so that an encoded "&amp;lt;" does not become "<".
  .replace(/&amp;/g, "&");

Decode & first and you have re-created an entity you then decode again.

The default origin is the one origin that can never work

NEXT_PUBLIC_APP_URL is http://localhost:3001 in development. That variable is also the natural default for "which site am I submitting for". So the friendly default is a guaranteed failure, and it fails as a confusing fetch error on the key file rather than as an explanation:

export function isPublicHost(host: string): boolean {
  const hostname = host.replace(/:\d+$/, "").toLowerCase();

  if (hostname === "localhost" || hostname.endsWith(".localhost")) return false;
  if (hostname === "::1" || hostname === "[::1]") return false;
  if (hostname.endsWith(".local") || hostname.endsWith(".internal")) return false;

  const ipv4 = hostname.match(/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/);
  if (ipv4) {
    const [a, b] = ipv4.slice(1, 3).map(Number);
    if (a === 127 || a === 10 || a === 0) return false;
    if (a === 192 && b === 168) return false;
    if (a === 172 && b >= 16 && b <= 31) return false;
    if (a === 169 && b === 254) return false;
    return true;
  }

  // A bare name with no dot cannot be a registrable domain.
  return hostname.includes(".");
}

Any tool whose default value comes from an environment variable that is different in development should ask whether the development value is a valid input at all. If it is not, say so in a sentence the user can act on.

Where I changed my mind

In the first post I recommended keyLocation over a key-named file at the root, on the grounds that a key in a filename means rotation is a rename plus a code edit. Notifio does the opposite, and the comment says why:

/**
 * The file has to sit at the root and be named for the key itself. The protocol
 * does allow an arbitrary location via a `keyLocation` field, but support for
 * that is less uniform across engines than the root convention, and the key is
 * public by design (the engines fetch it unauthenticated), so there is nothing
 * to gain by hiding it somewhere else.
 */
export function keyFileUrl(origin: string, key: string): string {
  return `${origin.replace(/\/$/, "")}/${key}.txt`;
}

Both are defensible. The root convention is the path every engine definitely supports, and --init generates a key and writes its public file in one step, so the rotation cost I was worried about is one command. If I were submitting to one engine I would use keyLocation. Fanning out to all of them, I would rather be boring.

The shape that made it testable

Every piece above is a pure function, and the two that talk to the network take their fetch as a parameter:

export async function submitBatch(
  endpoint: string,
  payload: IndexNowPayload,
  fetchImpl: typeof fetch = fetch,
): Promise<SubmitResult>

The result is a test file that covers key validation, host filtering, batching at the 10,000 URL protocol cap, entity decoding and every status code, with no network and no mocking framework. The batching function throws on a size below one rather than looping forever, which is the kind of thing you only write a test for when you have already written the infinite loop once.

Go and look at the pages this is announcing

The URLs in that sitemap are mostly real pages with a real job. notifio.app/alerts is one page per rental portal we monitor, notifio.app/compare is the honest comparison set including the tools I would use instead of ours in some cases, and notifio.app/guides is the writing that is not about the product at all. If any of them were edited today, they were announced with the command above.