Two paths announce our pages to Bing, and one Redis hash keeps them from announcing twice

Nakodo has 76 public pages. You can count them yourself in the sitemap: the home page, the pricing and method pages, and then the content families, like the guides hub and the per-niche pages under /youtubers.

Google finds changes on those pages by reading the sitemap when it feels like it. Bing, Yandex, Seznam and Naver will take a push instead: one POST to api.indexnow.org/indexnow with a host, a key and a list of URLs, and every engine that participates in IndexNow gets it. Google does not participate.

A push protocol sounds like three lines of code. The interesting part is not the POST. It is deciding what to send, from two places, without sending the same thing twice.

The key is deliberately public

An engine verifies a submission by fetching a key file from the site itself. So the key cannot be a secret, and ours is checked in:

export const INDEXNOW_KEY = "…";
export const KEY_PATH = `/${INDEXNOW_KEY}.txt`;

public/<key>.txt holds exactly that string and nothing else. One test asserts all three facts at once, because a wrong key file does not fail loudly, it turns every submission into a 403 forever:

test("the key file holds exactly the key, and crawlers can open it", () => {
  assert.match(INDEXNOW_KEY, /^[A-Za-z0-9-]{8,128}$/);
  assert.equal(readFileSync(`public${KEY_PATH}`, "utf8"), INDEXNOW_KEY);
  assert.ok(isPublicPath(KEY_PATH), `${KEY_PATH} would redirect the engines to /login`);
});

That third line is the one that has actually saved us. Everything in the app that is not on an allowlist gets redirected to /login by the proxy. A key file behind a redirect is an unverifiable key file.

What changed, as a diff against a hash

The sitemap entries each carry the date their content last changed, not the build date. That date is the whole signal:

export function pendingUrls(pages: IndexNowPage[], submitted: Record<string, unknown>) {
  if (pages.length === 0) return { changed: [], removed: [] };
  const live = new Set(pages.map((p) => p.url));
  return {
    changed: pages.filter((p) => String(submitted[p.url] ?? "") !== p.lastModified).map((p) => p.url),
    removed: Object.keys(submitted).filter((url) => !live.has(url)),
  };
}

submitted is a Redis hash of URL to the date it had when we last announced it. New page: no entry, so it is changed. Edited page: date moved, so it is changed. Deleted page: still in the hash, no longer in the sitemap, so it is announced as removed and the engines recheck it and drop it.

The first branch is there because of what the function would otherwise do with an empty list. No pages means every key in the hash is missing from live, so every URL on the site would be announced as gone. An empty sitemap is a bug in the sitemap, not a site that deleted itself, so it announces nothing.

The String(...) is not decoration either. Upstash parses values it reads back as JSON where it can, so "2026-10-03" comes back as a string but a bare year would not. Comparing a number to a string marks a page changed on every run forever.

Two writers, one record

There are two ways to announce. An hourly cron, which runs inside the deployment and reads the sitemap straight from the code:

export function indexNowPages(): IndexNowPage[] {
  return sitemap().map((e) => ({ url: e.url, lastModified: String(e.lastModified ?? "") }));
}

And pnpm indexnow, for when I have just shipped a page and do not want to wait an hour. That one deliberately does not read the checkout, because my checkout is usually ahead of production:

export async function livePages(): Promise<IndexNowPage[]> {
  const url = absoluteUrl("/sitemap.xml");
  const response = await fetch(url, { cache: "no-store", signal: AbortSignal.timeout(10_000) });
  if (!response.ok) throw new Error(`${url} answered ${response.status}`);
  return parseSitemap(await response.text());
}

Announcing a page that is not deployed yet sends four search engines to a page that redirects to the login form. Reading the live sitemap.xml makes that impossible: if it is not in there, the CLI tells me it is not deployed and skips it.

So one path parses XML and the other maps an array of objects, and they both write to the same hash. If they ever disagreed about a date, each run would undo the other's record and all 76 pages would be announced on every single run. There is a test for exactly that, and it renders the sitemap with Next's own resolver so the comparison is against real output rather than my idea of it:

import { resolveSitemap } from "next/dist/build/webpack/loaders/metadata/resolve-route-data";

test("the served sitemap.xml reads back as the same pages and dates", () => {
  assert.deepEqual(parseSitemap(resolveSitemap(sitemap())), indexNowPages());
});

Reaching into next/dist/build/webpack/loaders/... is not something to do casually. The alternative was a test that asserts my regex matches my own fixture, which is a test of nothing. This one fails if Next ever changes how it serializes a lastModified.

The failure modes get names, not throws

export async function submitToIndexNow(urls: string[]) {
  if (urls.length === 0) throw new Error("No URLs to submit to IndexNow");
  if (urls.length > MAX_URLS_PER_REQUEST) { /* 10,000 per request */ }
  const response = await fetch(ENDPOINT, { method: "POST", /* host, key, keyLocation, urlList */ });
  return { ok: response.ok, status: response.status, body: await response.text() };
}

It resolves with the status instead of throwing, because the statuses mean different things to the person reading the output: 200 or 202 is accepted (202 while the key is still being checked), 403 is a key the engines could not verify, 422 is a URL that is not on this host, 429 is too many submissions.

And on any rejection the caller records nothing, so the next hour tries the same pages again. The record is written only after the engines have taken them:

const result = await submitToIndexNow([...changed, ...removed]);
if (!result.ok) return { error: `IndexNow answered ${result.status}`, detail: result.body, changed, removed };
await recordAnnounced(redis, pages, removed);

Two more guards sit in front of the whole thing: only the production deployment announces, because a preview deployment is not the public site, and with no Redis configured it does nothing at all rather than announce all 76 pages every hour. A missing cache should make a feature quiet, not loud.

Most hourly runs now submit zero URLs, which is the point. The ones that do anything are the ones right after I change an updated date in src/content, and those run while I am still looking at the page I just edited on the blog.