Telling search engines only what actually changed, from 426 pages

IndexNow is a small, slightly unglamorous protocol: you POST a list of URLs to one endpoint, and Bing, Yandex, Seznam and the other participants are told your content changed. No waiting for a crawler to wander back. One request, all of them.

The part nobody writes about is what you are supposed to put in the list. Bing's guidance is explicit: submit a URL when its content is added, updated or removed. Announcing pages that did not change is how you earn a 429 and, eventually, a reputation.

So the protocol is easy and the job is not. The job is answering "which of my pages actually changed?" every hour, cheaply, without fetching your own site.

Here is how that went for Munchable, which publishes 426 URLs: seven condition guides, 38 recipes, 373 ingredient answer pages and a handful of static ones.

The obvious implementation is broken

Hash the rendered HTML of each page, compare to last time, submit the ones that moved.

This does not work on a modern framework, and it fails in the worst direction. Next.js stamps a fresh build id into every page on every deploy. Hash the markup and every page looks changed after every deploy, so every run announces the entire site. You have automated the exact spam pattern the protocol asks you not to perform.

Fetching your own 426 pages every hour to compute those useless hashes is the other half of the joke.

Hash what the page says, not what it renders

Every one of our pages is generated from a data module. So the fingerprint is taken from the data, in the process, with no HTTP involved at all:

{ path: '/conditions', hash: fingerprint(CONDITION_PAGES.map((p) => [p.slug, p.name, p.summary])) }

A run is one pass over the content modules, one Redis read of what was last announced, and almost always no network call at all. The runs that do submit are the ones right after a deploy that changed something real.

Two decisions inside that turned out to matter more than the hashing.

Hash a projection, not the whole object. An answer page has a question, a verdict, an explanation and the condition guide it belongs to. Hashing the whole page object would include the entire guide, so fixing a typo in the third paragraph of one guide would re-announce every answer page under it. The fingerprint is a deliberate subset: the question, the verdict in both label positions, the explanation, the ingredient name, the guide's slug.

Hash the derived page, not the source data. A recipe page's fingerprint includes the set of conditions our engine clears that recipe for, not just the recipe. The payoff arrives on a day nobody edited any content: move a threshold in a rule set, and a recipe can stop suiting a condition without a single word of its own copy changing. That is a page whose meaning changed, and the next hourly run announces it without anyone remembering to.

The same trick does the heavy lifting on the answer pages, whose fingerprints include the engine's verdict. Change a rule so an ingredient flips from caution to avoid, and exactly those pages get announced. Not the whole site, not nothing, just the ones whose answer moved.

The bug that announced everything, every hour

For a while the diff said every page had changed on every run, and the cause is worth the price of admission.

The hashes were bare hex, sliced to 16 characters. Stored in Redis through a client that JSON-parses values on read. Every so often, a hash comes out as all digits. On the way back it is no longer the string "4839105827364519", it is the number 4839105827364519, which is not equal to the string it was written as.

// Prefixed, never bare hex.
return `v1:${createHash('sha256').update(JSON.stringify(value)).digest('hex').slice(0, 16)}`;

One prefix, and the value can never be mistaken for a number. The same class of bug is waiting in any store that infers types on read: an ID, a postcode, a version string. If a value is a string, make it unable to look like anything else.

The guard that matters more than the diff

// A run that produced no pages is a bug in the content modules, not a site
// that just deleted itself.
if (current.length === 0) return { changed: [], removed: [] };

The diff has a second output alongside "changed": URLs that used to exist and no longer do, submitted so the engines recheck and drop them. Which means a run where the content modules yield nothing, an import cycle, a bad deploy, a refactor midway, would compute "everything was removed" and politely ask every search engine to drop the entire site.

Any diff that can emit deletions needs the empty-input case handled explicitly. The blast radius is not symmetric with the additions.

Ownership, and the key that is not a secret

IndexNow proves you own the host by having you serve a key file from it. Ours is live and you can read it: munchable.app/indexnow-key.txt.

That is not a leak. The protocol requires the key to be publicly readable, because fetching it is exactly how an engine verifies that whoever submitted those URLs controls the host.

There are two ways to serve it. The common one is a static <key>.txt at the root, which bakes the key into a filename: rotating it means renaming a committed file and editing the submitter to match, and getting one of the two wrong gives you a 403 you will debug on a Tuesday. We use the protocol's named-location form instead. The file is served by a route that reads the key from one place, and every submission carries a keyLocation pointing at it, so the key exists exactly once and can be rotated from a dashboard environment variable without a deploy touching code.

A few more details that stop this being a footgun:

• Only the production deploy submits anything. Preview deploys are noindex and their sitemap is empty, so announcing their URLs would hand the engines a competing copy of the site.
• 202 counts as success. It is the protocol's "accepted, key validation pending", which is what a first submission from a brand new key file gets.
• Every response code gets a named meaning in the log, so a failure reads key_not_valid or url_or_key_mismatch rather than a number you have to go and look up.
• A failed submission does not record the fingerprints, so the next run retries. Recording on failure is how you silently stop announcing a page forever.
• A unit test fails if the fingerprint list and the sitemap ever describe different sets of pages. Two lists of every URL on the site is one list too many, and only a test can keep them honest.

The pages nothing can fingerprint

Our homepage and licences page have their copy in JSX rather than in a data module. Nothing can detect an edit to them, so they are announced once and then left alone, with a manual endpoint as the escape hatch for the day someone rewrites the hero.

That is a real limitation and it is written down as one. It is better than the alternative, which is a clever heuristic that re-announces your homepage every hour because a date or a counter moved somewhere in the tree.

Have a look

• The key file itself, publicly readable by design: munchable.app/indexnow-key.txt
• The URL set it keeps in step with: munchable.app/sitemap.xml
• Pages whose fingerprints include an engine verdict, so a rule change announces them: munchable.app/answers
• Recipe pages fingerprinted on the conditions they clear: munchable.app/recipes

If you already have a sitemap generated from code, the whole thing is an afternoon. The hour goes on deciding what counts as a change, which is the only part search engines actually judge you on.