One POST tells four search engines a page changed, and the URL list must not be typed by hand

A sitemap is a standing invitation. IndexNow is a phone call.

The protocol is small enough to explain in a paragraph. You put a random key in a text file at the root of your site, then POST a JSON body naming your host, that key, where the key file lives, and a list of URLs that changed. The endpoint fetches the key file to check you control the host, and fans the submission out to every participating engine.

Participating means Bing, Yandex, Seznam and Naver. Google does not participate, and that is the first thing worth saying out loud, because it sets what this is for: it sits alongside Search Console, and it never replaces a sitemap.

The key is public and that is fine

// Public by design. IndexNow proves ownership by fetching the key back from
// https://<host>/<key>.txt, so this must match the filename in /public.
export const INDEXNOW_KEY = 'ab5629895afc40d8af55f18ffe75c6e5';

The key is committed to the repository in plain text, and the same string is both the filename and the contents of a file in public/. You can go and read ours at cogniprep.app/ab5629895afc40d8af55f18ffe75c6e5.txt.

It looks like a secret and it is the opposite of one. The authorisation is not knowledge of the key, it is the ability to serve that file from the host you claim to own. Anybody can read the key; only someone who controls the domain can make the verification fetch succeed. So there is nothing to protect, and treating it as a secret would mean putting it in an environment variable while also publishing it at a well known URL, which is theatre.

The failure this design does prevent is real, though: without the fetch-back step, anyone could submit URLs on your behalf, and the interesting abuse is not submitting your pages, it is submitting a firehose of them to burn your crawl reputation.

The list has to be derived, and I have the receipts

import sitemap from '@/app/sitemap';

export function getIndexNowUrls(): string[] {
  return sitemap().map((entry) => entry.url);
}

Two things submit to IndexNow: an API route for deploy hooks, and a CLI script. Both used to carry their own hardcoded array of URLs.

They had drifted. Between them they were missing /tests, every /tests/<format> page, and /about.

The reason this kind of list rots faster than most is in the failure mode. A missing URL produces no error anywhere. The page builds, renders, ranks eventually, and is simply never announced. There is no log line, no 404, nothing in Search Console that says "this page was not submitted", because not submitting is the default state of the universe. The only way to find out is to read two files side by side and notice.

Importing app/sitemap.ts fixes it at the root rather than patching the two copies. It works because a sitemap in the App Router is an ordinary function that returns an array, so anything can call it. Adding a provider, a test type, a blog post or an employer page now picks itself up in both places with no extra wiring, because all three are already derived from the same registries the sitemap reads.

The rule I would generalise: if two places need the same list, neither of them owns it. Find the thing that already has to be right for the site to work, and read that.

Ours is currently 235 URLs.

The client returns the response rather than throwing

export async function submitToIndexNow(urls, baseUrl = getSiteBaseUrl()): Promise<IndexNowResponse> {
  const response = await fetch(INDEXNOW_ENDPOINT, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json; charset=utf-8' },
    body: JSON.stringify({
      host: new URL(baseUrl).hostname,
      key: INDEXNOW_KEY,
      keyLocation: `${baseUrl}/${INDEXNOW_KEY}.txt`,
      urlList: urls,
    }),
  });

  return { ok: response.ok, status: response.status, statusText: response.statusText, body: await response.text() };
}

No response.ok check, no thrown error on a non-2xx. The caller gets the status, the status text and the raw body.

That is deliberate, because IndexNow's error codes are specific and each one means a different thing you have to go and fix:

• 403 means the key is wrong, or the key file is not reachable at keyLocation. That is an infrastructure problem.
• 422 means one or more URLs do not belong to the host you claimed. That is a data problem, usually a base URL pointing somewhere it should not.
• 429 means you are submitting too often, which is a scheduling problem.

Collapsing all three into throw new Error('IndexNow submission failed') destroys exactly the information that tells you which of those three things to look at. The route surfaces the detail verbatim, and the CLI prints it.

The two things that do throw are the ones where the request is malformed before it is sent: an empty URL list, and a list over IndexNow's 10,000 URL limit. Those are programming errors rather than remote outcomes, and there is no useful response to return for them.

Two entry points, for a slightly absurd reason

POST /api/indexnow     # for deploy hooks
pnpm indexnow          # for a human at a terminal

The route exists for automation. The script exists because the route is unreachable from a terminal: the site sits behind a bot challenge, so curl gets an interstitial rather than the endpoint. The API route is perfectly good and simply cannot be called by the person most likely to want to call it.

Rather than punch a hole in the challenge for one endpoint, the CLI imports the same module and talks to IndexNow directly. Both paths share lib/seo/indexnow.ts, so there is one submission format and one URL list, and the two entry points differ only in who triggers them.

The route's auth is worth a line on its own:

// Uses the shared helper, which fails CLOSED: the previous inline check
// skipped verification entirely when CRON_SECRET was unset.
const unauthorized = verifyCronSecret(request);
if (unauthorized) return unauthorized;

The inline version it replaced treated a missing secret as "nothing to check". That is the wrong default for every auth check ever written, and it is an easy one to write by accident, because during local development it is indistinguishable from the code working.

The CLI has two small guards worth stealing

function resolveUrl(target: string, baseUrl: string): string {
  const url = new URL(target, baseUrl);
  if (url.hostname !== new URL(baseUrl).hostname) {
    throw new Error(`${target} is not on ${new URL(baseUrl).hostname}; IndexNow would reject it`);
  }
  return url.toString().replace(/\/$/, '');
}

This means pnpm indexnow /blogs/my-new-post and pnpm indexnow https://cogniprep.app/pricing both work, and a URL on somebody else's host fails locally with a sentence explaining why, instead of remotely with a 422 you then have to go and look up.

And:

Submits against the production host by default. Note that .env.local is
deliberately not loaded, because a localhost value in there would produce
a submission IndexNow rejects.

This is the one that took a real debugging session to understand. Almost every script in a Next.js repo wants .env.local. This one must not have it, because the variable it would pick up is the website URL, and the whole point of a local env file is that it points at localhost. Loading it means announcing http://localhost:3000/pricing to Bing.

--dry-run prints the list and submits nothing, which given all of the above is the flag you use first every time.

See it

Everything here is public:

• cogniprep.app/ab5629895afc40d8af55f18ffe75c6e5.txt is the key file. It contains the key, which is also its own filename. That circularity is the whole verification mechanism.
• cogniprep.app/sitemap.xml is the list that gets submitted, 235 <loc> entries, and it is the same function call in both cases.
• While you are looking at root-level files nobody visits: cogniprep.app/llms.txt and cogniprep.app/humans.txt are the other two.

Then check your own site. Open https://yourdomain/<your-key>.txt in a browser and confirm it actually serves. A key file that 404s means every submission you have made has been answered with a 403, and unless you logged the response body, nothing told you.