The 235 URLs we announce to Bing come from one function, and it is sitemap()

IndexNow is the protocol Bing, Yandex, Seznam and Naver use to be told a URL changed, instead of waiting for a crawler to notice. Google does not participate, so it sits alongside your Search Console sitemap rather than replacing it. The protocol itself is about as small as a protocol gets: you host a text file containing a key, then POST a JSON body with a list of URLs.

Which means the entire engineering problem is the list. And a list of URLs is the single easiest thing in a web codebase to get quietly wrong, because a URL missing from it does not break anything. The page still renders. It just never gets announced, and you find out months later when you audit coverage by hand.

Two copies, both incomplete

We had two callers: a POST /api/indexnow route for deploy hooks, and a pnpm indexnow command for running it by hand. Each kept its own hardcoded array of paths. Between them they were missing /tests, every /tests/<format> page, and /about.

Nobody wrote a bad line of code to cause that. Someone shipped a content cluster and updated the sitemap, because the sitemap is the file you think of. The other two lists are not in anyone's mental model of "publishing a page".

So the fix is to delete them and derive:

import sitemap from '@/app/sitemap';

/**
 * Every canonical URL, read straight out of the sitemap so the two can never
 * disagree. Adding a provider, test type, cheating guide, blog post or employer
 * page picks itself up here with no extra wiring.
 */
export function getIndexNowUrls(): string[] {
  return sitemap().map((entry) => entry.url);
}

That is the whole module's reason to exist. app/sitemap.ts is a plain function that returns an array of objects in Next.js, which means it is importable from anywhere, not only from the route that serialises it to XML. Any file in your app that needs "the list of our pages" should be calling it rather than maintaining a parallel copy.

The test that guards it is one line, and it is the only assertion in the file that really matters:

it('submits exactly the sitemap, so the two can never drift apart', () => {
  expect(getIndexNowUrls()).toEqual(sitemap().map((entry) => entry.url));
});

Plus one that names the specific clusters the old lists had already dropped, so the regression has a shape someone can recognise:

expect(urls).toContain(`${baseUrl}/tests`);
expect(urls).toContain(`${baseUrl}/about`);
expect(urls.some((url) => url.startsWith(`${baseUrl}/tests/`))).toBe(true);

And one that exists purely because of the protocol's economics: duplicates count against the daily quota, so emitting the same URL twice is a real cost, not a cosmetic issue.

expect(new Set(urls).size).toBe(urls.length);

See it: open cogniprep.app/sitemap.xml and count the <loc> elements. That number, 235 as I write this, is exactly what a pnpm indexnow run submits, because both come from the same call.

The sitemap has to earn that trust

Deriving from the sitemap is only an improvement if the sitemap itself is derived. Ours is built from the registries that already exist for other reasons, so a new page cannot be added without appearing in it:

...ALL_PROVIDERS.map((provider) => ({
  url: `${baseUrl}/games/${provider}`,
  lastModified: new Date('2026-08-09'),
  changeFrequency: 'weekly' as const,
  priority: 0.9,
})),

The same shape covers test formats, cheating guides, interview guides, assessment centre guides and blog posts. The one cluster with no registry is the employer pages, which are hand written articles in their own route folders, so the sitemap reads the directory instead:

const employerSlugs = readdirSync(employersDir, { withFileTypes: true })
  .filter((entry) => entry.isDirectory())
  .map((entry) => entry.name);

That works because the sitemap runs at build time, on the machine that has the source tree. Creating the folder is the act of creating the page, so there is no second step anyone can forget. If a page exists, it is announced.

See it: cogniprep.app/employers lists those pages, and every one of them has a <loc> in the sitemap above without anyone having typed the slug twice.

The submission

Once the list is honest, the POST is unremarkable:

const response = await fetch(INDEXNOW_ENDPOINT, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json; charset=utf-8' },
  body: JSON.stringify({
    host: new URL(baseUrl).hostname,
    key: INDEXNOW_KEY,
    keyLocation: `${baseUrl}/${INDEXNOW_KEY}.txt`,
    urlList: urls,
  }),
});

api.indexnow.org fans one submission out to every participating engine, so you do not need four requests. The key proves ownership: the engine fetches keyLocation and expects the file to contain the key and nothing else.

See it: cogniprep.app/ab5629895afc40d8af55f18ffe75c6e5.txt. It is a public file by design, which surprises people. It is not a credential for anything other than saying "these URLs belong to this host", and that claim is already verifiable.

Two details worth copying in the submit helper. It resolves rather than throws on a rejection:

return { ok: response.ok, status: response.status, statusText: response.statusText, body: await response.text() };

IndexNow answers 403 for a bad key and 422 for URLs that do not belong to the host, and both of those are worth surfacing verbatim to whoever ran the command. An exception loses the body.

And it refuses to send more than the protocol accepts:

const MAX_URLS_PER_REQUEST = 10_000;

We are two orders of magnitude below that. The check is there so the day we are not, it fails on our side with a sentence, rather than on theirs with a status code.

Why there is a CLI at all

The API route is secured with a cron secret, and the helper that checks it fails closed. An earlier inline version skipped verification entirely when the secret was unset, which quietly turned an authenticated endpoint into an open one that makes outbound requests on your behalf.

const unauthorized = verifyCronSecret(request);
if (unauthorized) return unauthorized;

But the route also sits behind the host's bot challenge, so you cannot trigger it from a terminal with curl. That is why pnpm indexnow exists as a peer rather than a wrapper around the HTTP call. It imports the same module:

pnpm indexnow                      # every canonical URL in the sitemap
pnpm indexnow --dry-run            # print the list, submit nothing
pnpm indexnow /blogs/my-new-post   # just these paths

The path form validates against the host before sending, because a URL on the wrong hostname is a 422 you would rather read as an English sentence:

if (url.hostname !== new URL(baseUrl).hostname) {
  throw new Error(`${target} is not on ${new URL(baseUrl).hostname}; IndexNow would reject it`);
}

And it deliberately does not load .env.local, because a localhost value in there produces a submission the endpoint rejects, from a command whose entire purpose is to run against production.

The takeaway

IndexNow takes an afternoon. The list of URLs takes forever, if you let it become its own artefact. Import the sitemap, assert equality in a test, and the next content cluster announces itself.