IndexNow, but only when the page actually changed

IndexNow is a small, unglamorous protocol that does one useful thing: you POST a list of URLs and the participating search engines (Bing, Yandex, Seznam and others) know your content changed immediately, instead of finding out when a crawler happens to come back weeks later.

One POST to api.indexnow.org fans out to all of them. It takes about twenty minutes to wire up.

Then you hit the part nobody writes about, which is when to submit. Get that wrong and you have built a spam cannon pointed at your own domain.

The rule that actually matters

Bing's guidance is to submit a URL when its content is added, updated or removed. Not on a schedule. Not on deploy. When the content changed.

Re-announcing unchanged pages is what earns you a 429, and eventually a reputation. A daily cron that submits your whole sitemap is the single most common IndexNow implementation and it is precisely the behaviour the protocol asks you not to exhibit.

So the real problem is not "how do I call this API". It is "how does my build know which pages genuinely changed?"

The obvious answer is wrong

Hash the rendered HTML and compare against last time.

This does not work, and it fails in a way that looks like it is working. Next.js stamps a fresh build id into every page on every deploy. So every page's HTML differs after every deploy, so every page looks changed, so you submit all 375 URLs every time you fix a typo in your footer.

You have built the exact spam pattern you were trying to avoid, with extra steps and a hashing function.

Fingerprint the data, not the markup

The fix is to hash the content the page is derived from, and nothing else. No build id, no markup, no framework output.

export function pageFingerprints(): PageFingerprint[] {
  const pages = [
    // The index changes when the set of guides, or the line each shows, changes.
    {
      path: '/conditions',
      hash: fingerprint(CONDITION_PAGES.map((p) => [p.slug, p.name, p.summary])),
    },
    ...CONDITION_PAGES.map((page) => ({
      path: `/conditions/${page.slug}`,
      hash: fingerprint(page),
    })),

    { path: '/answers', hash: fingerprint(ANSWER_PAGES.map((p) => p.question)) },
    ...ANSWER_PAGES.map((page) => ({
      path: `/${page.question}`,
      hash: fingerprint(answerContent(page.question)),
    })),
  ];

  return pages.map((page) => ({ url: absoluteUrl(page.path), hash: page.hash }));
}

Each entry declares what "changed" means for that page. An index page changes when the set of items on it changes. A detail page changes when its own record changes. Deploys are invisible. Copy edits are not.

The last line of that list is the one I like most. On Munchable, the pages at munchable.app/answers do not contain any hand-written verdicts: each one runs the production rules engine to produce its answer. So the fingerprint hashes the generated content, which means moving a threshold in the rules engine correctly marks every affected answer page as changed, without anyone touching a word of copy.

The same idea shows up on our recipe pages:

// Hashing the derived page rather than the recipe is deliberate: move a rule
// threshold and a recipe can stop suiting a condition without a word of its
// own copy changing, and that is exactly a change worth announcing.

Hash the output of your content pipeline, not its input. A recipe whose text is identical but which now suits a different set of conditions has genuinely changed for a reader, and that is the thing you are announcing.

Pages whose copy lives in JSX

Not every page has a data module behind it. Our homepage and licences page are JSX, and no fingerprinting scheme can see an edit to them.

Pretending otherwise would mean either submitting them constantly or never. So we are explicit:

/**
 * Pages whose copy lives in JSX rather than in a data module: the homepage and
 * /licenses. Nothing here can detect an edit to them, so they are announced once
 * (the first run, when they are new to the engines) and then left alone. Editing
 * their copy is the case the route's POST exists for.
 */
const STATIC_COPY = fingerprint('static-copy');

A constant. The page gets announced on the first run, when it is new to the engines, and never again automatically. When we actually rewrite the homepage, we hit the manual endpoint.

Naming the limitation and providing a manual override is a much better answer than a heuristic that is wrong in both directions.

The bug that cost me an hour

This one is worth the price of admission on its own.

/**
 * Prefixed, never bare hex. Upstash's client JSON-parses stored values on read,
 * so an all-digit hash would come back as a number and never compare equal to
 * the string it was written as, marking that page changed on every single run.
 */
function fingerprint(value: unknown): string {
  return `v1:${createHash('sha256').update(JSON.stringify(value) ?? 'null').digest('hex').slice(0, 16)}`;
}

We store fingerprints in a Redis hash on Upstash. Upstash's REST client helpfully JSON-parses values on read.

A truncated SHA-256 in hex is usually something like a3f9c2..., which JSON cannot parse as anything, so it comes back as a string. Fine. But roughly one hash in sixteen million is all digits. That one comes back as a number, !== the string you wrote, and that page is marked changed on every single run, forever.

It is a one-in-sixteen-million bug that is permanent once it lands, invisible in every test, and presents as "why does this one page keep getting submitted". The v1: prefix makes the value unparseable as JSON, which fixes it. The version prefix is also free invalidation if we ever change what goes into the hash.

The general lesson: any store that infers types on read will eventually infer the wrong one. If a value is a string, make it a string that cannot be mistaken for anything else.

Two protocol details

The key is not a secret. IndexNow proves you own the host by requiring the key to be publicly readable at a well known URL. That is the entire ownership check. Ours is checked into the repo, correctly.

Use keyLocation, not a key-named file. The protocol lets you either serve <key>.txt from your root, or serve the key anywhere and point at it with keyLocation on each submission. The first form hardcodes your key into a filename, so rotating it means renaming a committed file and editing the submitter to match. The second form keeps the key in exactly one place:

export const KEY_PATH = '/indexnow-key.txt';

export function indexNowKey(): string | null {
  const key = (process.env.INDEXNOW_KEY || DEFAULT_KEY).trim();
  return isValidKey(key) ? key : null;
}

/** Protocol: 8 to 128 characters, `a-z A-Z 0-9 -` only. */
export function isValidKey(key: string): boolean {
  return /^[A-Za-z0-9-]{8,128}$/.test(key);
}

The key file route and the submitter both read from that one function, so they cannot disagree, and the key can be rotated from a dashboard environment variable without a code change.

Batch anyway. The protocol ceiling is 10,000 URLs per request. We are at 375, which is nowhere near it, and the batching code exists anyway so that a future product-page sitemap cannot silently overflow.

One more thing: keep it in step with your sitemap

Your fingerprint list and your sitemap describe the same set of pages. They will drift the moment somebody adds a route and only remembers one of them.

So there is a test that fails if they ever disagree. Not a comment asking people to remember. A test.

The summary

• Submit on content change, never on a schedule, never on deploy.
• Fingerprint the data your page is derived from, never the rendered HTML.
• Hash the output of your content pipeline, so a logic change that alters what a page says counts as a change.
• Be explicit about the pages you cannot fingerprint, and give yourself a manual trigger for them.
• Prefix your hashes so a type-inferring store cannot turn one into a number.

You can see the pages this runs over at munchable.app/answers and munchable.app/conditions. If you would rather see what the engine behind them does with a real barcode, the free tier needs no card.