Munchable publishes 426 URLs, and every one of them comes from a TypeScript module rather than a database. Seven condition guides, 38 recipes, 373 ingredient answer pages, and eight hand-written pages on top. You can count them yourself in the sitemap.
A sitemap tells a crawler what exists. It does not tell anybody that something changed, and a crawler that visited an answer page last month has no reason to come back today. IndexNow is the fix for that: one POST to a shared endpoint, which fans out to the participating engines, which the protocol's own site lists as Microsoft Bing, Naver, Seznam.cz, Yandex and Yep.
The protocol is about ten minutes of work. Deciding which URLs to put in the payload is the entire problem.
Submitting everything is the thing you must not do
Every piece of official guidance says the same thing. From the IndexNow FAQ, on rate limits:
Submit only when content has changed. Do not resubmit unchanged URLs.
And on repeat submissions:
Avoid submitting the same URL many times a day unless there are meaningful content changes.
The sanction is an HTTP 429, and the worse sanction is the one nobody sends you a status code for, which is being the host that pings 426 URLs every time somebody fixes a typo.
So the job needs an answer to "has this page changed since the last time I told anyone about it?" without fetching 426 pages and diffing them.
The fingerprint that does not work
The obvious move is to hash the rendered HTML. It is also useless, and the reason is worth knowing if you are on any modern framework:
// Hashing the rendered markup would be simpler and useless: Next stamps a
// fresh build id into every page on every deploy, so every page would look
// changed every time, which is the spam pattern this is here to avoid.
Every deploy produces a new build id, the build id appears in the payload of every page, and so a markup hash says "all 426 pages changed" after a deploy that moved one word of CSS. That is not a near miss. It is precisely the behaviour the protocol asks you not to have, implemented with a hash function so it looks rigorous.
Fingerprinting the content instead
Each page's fingerprint is a hash of the data that page renders, built in the same module that knows where the pages come from:
export function pageFingerprints(): PageFingerprint[] {
const pages: Array<{ path: string; hash: string }> = [
{ path: '/', hash: STATIC_COPY },
{ path: '/about', hash: STATIC_COPY },
{ path: '/licenses', hash: STATIC_COPY },
{ path: '/privacy', hash: fingerprint(PRIVACY_VERSION) },
{ path: '/terms', hash: fingerprint(TERMS_VERSION) },
{
path: '/conditions',
hash: fingerprint(CONDITION_PAGES.map((p) => [p.slug, p.name, p.summary])),
},
...CONDITION_PAGES.map((page) => ({
path: `/conditions/${page.slug}`,
hash: fingerprint(page),
})),
// ... recipes and answers
];
return pages.map((page) => ({ url: absoluteUrl(page.path), hash: page.hash }));
}
Three things in there are decisions rather than plumbing.
The legal pages hash a version number. The privacy policy and the terms already carry a version constant that gets bumped by hand when their wording materially changes, because that bump is what triggers a reconsent notice in the app. That constant is exactly the signal this job wants, so it does not need to look at the prose at all.
An index page hashes a projection, not its children. /conditions hashes each child's slug, name and summary, because those three fields are what the index page actually prints. Hashing the full condition objects would re-announce the index whenever any detail inside any guide changed, even though the index would look identical.
The answer pages hash what the page asserts, not the object behind it. This is the one that took a second attempt:
* Deliberately a projection rather than the whole `answerPage()` object, which
* embeds the full condition guide. Hashing that would re-announce all 25 answer
* pages for a condition because someone fixed a typo in the guide's third
* paragraph.
Each answer page embeds the guide for the condition it belongs to, so the naive hash coupled 25-odd pages to a paragraph none of them displays. The projection includes the question, the answer the page gives and its own prose, and nothing else.
The payoff of including the answer, and not just the text around it, is that the job notices things nobody remembers to tell it. If the app's engine starts giving a different answer for one of these questions, the next hourly run announces exactly the pages whose answer moved, because their fingerprint moved with it. No one has to maintain a list.
Two guards that cost nothing and save a lot
The first is a prefix on the hash, and it is the kind of bug you only find by watching a job do nothing visible for a week:
/**
* Prefixed, never bare hex. Upstash's client JSON-parses stored values on read,
* so an all-digit hash would come back as a number and never compare equal to
* the string it was written as, marking that page changed on every single run.
*/
function fingerprint(value: unknown): string {
return `v1:${createHash('sha256').update(JSON.stringify(value) ?? 'null').digest('hex').slice(0, 16)}`;
}
A 16-character hex string is all digits about one time in 1,845, which is (10/16) ** 16. Across 426 pages that works out at roughly a one in five chance that any given hashing scheme has at least one page whose stored fingerprint comes back from Redis as a number, fails a string comparison against the identical hash it was written as, and is announced on every run forever. Nothing in the logs would look wrong, because announcing a changed page is what the job is for. The v1: prefix makes every value unambiguously a string, and doubles as a version marker for when the scheme does change.
The second guard is three lines in the diff:
export function diffFingerprints(
current: readonly PageFingerprint[],
stored: Record<string, unknown>,
): UrlDiff {
// A run that produced no pages is a bug in the content modules, not a site
// that just deleted itself. Announcing 426 removals on the strength of it
// would ask every engine to drop the entire site.
if (current.length === 0) return { changed: [], removed: [] };
const live = new Set(current.map((page) => page.url));
return {
changed: current
.filter((page) => String(stored[page.url] ?? '') !== page.hash)
.map((page) => page.url),
removed: Object.keys(stored).filter((url) => !live.has(url)),
};
}
Removals are a real feature: a URL that used to exist and now does not should be submitted, so the engines recheck it and drop it. Which means the failure mode of an empty current array is "ask five search engines to delete your website", and the cost of ruling that out is one early return. Note also the String(stored[...] ?? '') on the comparison side, which is belt and braces against the Upstash coercion above.
Failure is a no-op, deliberately
const outcome = await submitUrls([...changed, ...removed]);
if (!outcome.ok) {
// Nothing is recorded on a failure, so the next run announces the same
// URLs again rather than quietly forgetting them.
Fingerprints are only written after the endpoint accepts them. A failed submission leaves the stored state untouched, so the next hourly run computes the same diff and tries again. The alternative, recording optimistically, loses an announcement permanently every time the network blips, and you would never know, because the job's happy path and its silent-loss path look identical in the logs.
202 counts as acceptance, incidentally. It is the protocol's "accepted, key validation pending", which is what the very first submission from a new key file gets.
The key is not a secret, and it still should not be a filename
Ownership is proved by hosting the key where the engines can read it. The protocol allows a <key>.txt at the site root, which is the form most tutorials show, and it bakes the key into a committed filename. Rotating it then means renaming a file and editing the submitter to match.
The alternative form puts the key at a path of your choosing and names that path in every submission:
export const KEY_PATH = '/indexnow-key.txt';
export function indexNowKey(): string | null {
const key = (process.env.INDEXNOW_KEY || DEFAULT_KEY).trim();
return isValidKey(key) ? key : null;
}
Both the key file route and the submitter read it from that one function, so they cannot disagree, and the route is deliberately dynamic rather than prerendered so that changing the environment variable takes effect without a redeploy. It is served to everybody, with a five minute cache, because the protocol requires exactly that: a publicly readable key at a known URL is how an engine checks that whoever submitted those URLs controls the host. The key is not a secret in any sense, which is why it is checked into the repository as the default.
One last thing worth copying: the job refuses to run outside production.
if (!IS_INDEXABLE) {
return NextResponse.json({ skipped: true, reason: 'not_production' }, { status: 200 });
}
Preview deploys are noindex and serve an empty sitemap. A preview that announced its own URLs would be handing five search engines a complete, crawlable, competing copy of the site, submitted with a valid key for a host they would be perfectly entitled to index.
Top comments (1)
Hashing what the page says instead of the markup is the right call. We ran into the same thing from the other side: a build stamp and a "last updated" line in the footer made every page look changed on every hourly build. Two things that kept our IndexNow payload small: normalize dates and counters out of the fingerprint (a listing page whose only change is "updated 10:05" shouldn't count), and cap each run so a template change spreads over several days instead of one burst. Curious whether you've seen Bing actually recrawl faster after the hash change, or mainly a drop in 429s?