Our IndexNow key sits in the repo on purpose, and our own auth gate was the only thing that could break it

Our site publishes 72 pages. A sitemap is a standing statement about what exists, so a crawler reads it whenever it next decides to. IndexNow is the other kind of signal: an event that says "this URL just changed", and the engines that participate act on it in minutes rather than whenever they next come round. Bing, Yandex, Seznam and Naver participate. Google does not, so this is additive to the Search Console sitemap rather than a replacement for it.

The protocol is three facts: a key, a file at https://yourhost/<key>.txt whose body is that key, and a POST that quotes both back at the API.

export type IndexNowSubmission = {
    host: string
    key: string
    keyLocation: string
    urlList: string[]
}

That is all of it. The interesting part is not the payload, it is that everything which went wrong for us went wrong inside our own app, before the request left.

The key is not a secret, and pretending otherwise costs you

Our key is a UUID with the dashes removed, and it is committed to the repository:

export const INDEXNOW_KEY = '2382c7da66fd467eb45e9c97aadf605a'
export const INDEXNOW_KEY_PATH = `/${INDEXNOW_KEY}.txt`

You can read it right now:

curl https://pub-trivia.app/2382c7da66fd467eb45e9c97aadf605a.txt

That is not an oversight, it is the mechanism. IndexNow verifies ownership by requiring you to serve the key at a URL on the host you are claiming. Anyone can fetch it. What nobody else can do is make our host serve it. Moving the key into an environment variable would protect nothing, and it would separate the key from the one thing it has to agree with, which is the route that serves it.

The thing worth protecting is that agreement. So the route reads the constant instead of repeating the string:

import { INDEXNOW_KEY } from '@/lib/indexnow'

export const dynamic = 'force-static'

export function GET() {
    return new Response(INDEXNOW_KEY, {
        headers: {
            'Content-Type': 'text/plain; charset=utf-8',
            'Cache-Control': 'public, max-age=86400',
        },
    })
}

A test asserts that the directory name, the constant and the served body are the same string. Note the directory: app/2382c7da66fd467eb45e9c97aadf605a.txt/route.ts. The filename has to be the key literally, and in the App Router the way to put a literal filename at a literal URL is to name the route segment after it, extension included.

The part that actually broke

Our first submission came back 403. The documented meaning of 403 is "the key file could not be fetched or did not match", which covers both halves of the mechanism and therefore tells you nothing about which half.

It was neither. It was our own middleware.

/<key>.txt is not a file in public/, it is a route, so it goes through the proxy, so it reaches the auth gate. The gate did not recognise the path, did what it does for any unrecognised path, and answered with a 307 to /login. From IndexNow's side the key file was unfetchable. From our side the file was obviously fine, because a browser following the redirect eventually renders something and curl without -i happily prints the login page's HTML.

We had already been bitten by the same shape of bug twice. robots.txt and sitemap.xml are also routes in a Next app, served from app/robots.ts and app/sitemap.ts rather than being files on disk, and they too were auth-gated into a redirect for every crawler that asked. So the fix is the same fix, in the list that exists because of those two:

export const PUBLIC_ROUTES = [
    // ...
    '/robots.txt',
    '/sitemap.xml',
    INDEXNOW_KEY_PATH,
] as const

Worth noticing which of the three is also excluded from the middleware matcher and which is only in this list. robots.txt and sitemap.xml are fetched by every crawler on every pass, so they are pattern-excluded from the matcher and skip the Redis and Supabase round trip entirely. The key file is read a handful of times a year, so it pays that round trip and stays out of the regex: hard-coding a key into a middleware matcher to save a request almost nobody makes is a bad trade, and it would put the key in a second place that has to agree with the first.

So check the key file yourself, with redirects turned off

The submission does not have to be the thing that tells you the key file is broken. Ours checks first:

export async function verifyKeyFile(): Promise<KeyFileCheck> {
    const url = absoluteUrl(INDEXNOW_KEY_PATH)
    const response = await fetch(url, { redirect: 'manual' })

    if (response.status >= 300 && response.status < 400) {
        const target = response.headers.get('location') ?? 'elsewhere'
        return {
            ok: false,
            url,
            detail: `redirects (${response.status}) to ${target}, the key file must return 200 with the key as its body`,
        }
    }
    if (response.status !== 200) {
        return { ok: false, url, detail: `returned ${response.status}` }
    }

    const body = (await response.text()).trim()
    if (body !== INDEXNOW_KEY) {
        return {
            ok: false,
            url,
            detail: `served ${JSON.stringify(body.slice(0, 60))}, expected the key`,
        }
    }

    return { ok: true, url, detail: 'serves the key' }
}

redirect: 'manual' is the entire value of this function. fetch follows redirects by default, so the obvious version of this check gets a 200, reads the login page's HTML, and reports that the body is wrong rather than that the request never arrived at the route. Which is true, and useless. The failure we actually had is a 3xx, and you only ever see a 3xx if you refuse to follow it.

The rest is the lesson of that 403 written down: every documented status mapped to what it means, so the script can say it out loud instead of printing a number.

export const INDEXNOW_STATUS_MEANING: Record<number, string> = {
    200: 'accepted',
    202: 'accepted, key validation pending',
    400: 'bad request, the payload is malformed',
    403: 'forbidden, the key file could not be fetched or did not match',
    422: 'unprocessable, a URL is not on this host, or the key does not match',
    429: 'rate limited, too many submissions',
}

The URL list is the sitemap's list, not a copy of it

The remaining way to get a 422 is a URL on the wrong host, and the spec rejects the whole submission if a single URL in it is off-origin. So the paths are checked before the request rather than discovered by the API, and they are read from the same constant the sitemap is generated from:

pnpm indexnow                          # every URL in the sitemap
pnpm indexnow /guides/pub-quiz-format  # only the paths given
pnpm indexnow --dry-run                # print the payload, send nothing

Naming a path that is not in the sitemap is an error rather than a submission, because at that point it is either a typo or a page that should have been added to the sitemap first.

The default is only right once, though. The whole sitemap is what a first submission wants. Everything after that should name what changed: IndexNow's entire content is "this URL is new or different", and re-sending 72 unchanged pages on every deploy is how you teach the receiving end to discount the signal.

Try it

• curl https://pub-trivia.app/2382c7da66fd467eb45e9c97aadf605a.txt returns one line with no trailing newline, exactly the string in the repo. That is a published secret doing its job.
• curl -s -o /dev/null -w "%{http_code}\n" https://pub-trivia.app/sitemap.xml should print 200. Run the same two commands against your own Next app if it has auth middleware. The redirect is silent, it only affects clients that are not browsers, and it is the reason a perfectly good sitemap can sit unread for months.
• The pages that got submitted are at pub-trivia.app/guides and pub-trivia.app/tools, if you want to see what the whole exercise was for.