Our deploy CLI default origin is the one origin that can never work

The marketing site for Notifio has a few hundred pages, most of them generated from data: one per rental site under /alerts, one per competitor under /compare, plus the guides at /guides. After a deploy that changes any of them, a script tells IndexNow, so Bing, Yandex, Seznam and Naver re-crawl the changed pages instead of waiting for their own schedule.

I have written about the protocol's traps before, in a 200 from IndexNow does not mean it read your key, and about only pinging pages that actually changed in IndexNow, but only when the page actually changed.

This post is about something else entirely: the script has five separate ways to refuse to run, and writing them turned out to be most of the work. Not the HTTP call. The refusals.

The default is the one value that can never work

Here is the default origin, resolved the obvious way:

site: process.env.NEXT_PUBLIC_APP_URL ?? DEFAULT_SITE,

NEXT_PUBLIC_APP_URL is http://localhost:3001. It says so in .env, because that is what the Next dev server binds to and every other consumer of that variable wants exactly that.

So the default origin for this command, in the environment where I will actually be typing the command, is a host the search engines cannot reach. IndexNow works by having the engine fetch a key file from your domain over the public internet. Point it at localhost and there is no version of the request that can succeed.

Without a guard, that failure arrives as a fetch failed on the key file check, which reads like a network problem and sends you off to look at your connection. The guard says the actual thing:

/**
 * The engines have to fetch the key file over the public internet, so a local
 * origin can never work. NEXT_PUBLIC_APP_URL is localhost in development, which
 * makes this the likeliest way to run the command wrongly.
 */
if (!isPublicHost(host)) {
  die(
    `${opts.site} is not a publicly reachable origin, so the engines could never fetch the key file.\n` +
      `  Pass the live origin explicitly, for example: pnpm indexnow --site ${DEFAULT_SITE} ...`,
  );
}

The thing I would not have predicted is how much of the module that one check justifies. isPublicHost is not a URL validator, it is specifically a "could a stranger's server reach this" test, and that means enumerating the private ranges:

export function isPublicHost(host: string): boolean {
  const hostname = host.replace(/:\d+$/, "").toLowerCase();

  if (hostname === "localhost" || hostname.endsWith(".localhost")) return false;
  if (hostname === "::1" || hostname === "[::1]") return false;
  if (hostname.endsWith(".local") || hostname.endsWith(".internal")) return false;

  // Private and loopback IPv4 ranges, per RFC 1918 and RFC 5735.
  const ipv4 = hostname.match(/^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$/);
  if (ipv4) {
    const [a, b] = ipv4.slice(1, 3).map(Number);
    if (a === 127 || a === 10 || a === 0) return false;
    if (a === 192 && b === 168) return false;
    if (a === 172 && b >= 16 && b <= 31) return false;
    if (a === 169 && b === 254) return false;
    return true;
  }

  // A bare name with no dot cannot be a registrable domain.
  return hostname.includes(".");
}

Twenty lines to catch a mistake that costs thirty seconds of confusion. Worth it, and it earned its keep a second time in a place I did not write it for. The --init subcommand generates a key and then tells you where to go and confirm it, and that instruction is useless if it points at localhost:

// On a dev .env the origin is localhost, which is useless in the "now go
// confirm this URL" line, so fall back to the production site for display.
const display = isPublicHost(new URL(origin).host) ? origin : DEFAULT_SITE;

The general point: if your command's default comes from the environment, check whether the environment you will be typing it in supplies a value that works. Config inherited from a dev server is usually right for the dev server and wrong for anything that talks to the outside world. This is not defensive programming, it is the difference between a tool being usable and being a thing you have to remember a flag for.

When the remote is all-or-nothing, partial success is a lie

The protocol requires every URL in a submission to belong to the host you name in the payload, and a single foreign URL causes the engine to reject the whole batch. Not to skip that URL. To reject the request.

That fact decides the behaviour of the client, and the library's own header says why the filtering has to happen before the request rather than after:

/**
 *  - Every URL in one request must live on the same host as `host`, and the
 *    whole request is rejected if any single URL does not. Filtering therefore
 *    has to happen before submitting, not after a 422 comes back.
 */

The interesting decision is what the CLI does with a stray. It could drop it and submit the rest. It does not:

const { onHost, offHost } = partitionByHost(resolved, host);
if (offHost.length > 0) {
  // A single foreign URL makes the engines reject the entire batch, so this is
  // fatal rather than a warning: submitting the remainder silently would hide a
  // mistyped hostname behind a successful-looking run.
  die(`These URLs are not on ${host}:\n  ${offHost.join("\n  ")}`);
}

A mistyped hostname in the argument list is almost always a page I meant to submit. Filtering it out and printing a tick would mean the run "succeeded" while the page I cared about was never announced, and I would not find out for weeks. The library still returns the strays rather than discarding them, for the same reason: they are evidence, not noise.

Compare that with what is right next to it, which is deliberately not fatal:

const unique = [...new Set(onHost)];
if (unique.length < onHost.length) {
  const dropped = onHost.length - unique.length;
  console.log(`• Removed ${dropped} duplicate ${dropped === 1 ? "URL" : "URLs"}`);
}

Duplicates are untidy. Off-host URLs are a rejection. The test I ended up applying to every input problem: would the remote refuse this, or is it merely inelegant? Refusal is fatal, inelegance is a note.

Check the thing that fails invisibly

A wrong key does not produce a helpful error. It produces a 403 with no body, and nothing on your site looks broken, so the failure mode is "pages quietly stopped being re-crawled by everything except Google". That is why the submission is preceded by a fetch of your own key file:

/**
 * Confirms the hosted key file exists and matches the key being submitted.
 *
 * Worth doing before every run: a key mismatch is the single most common way
 * this integration breaks, and it fails as a 403 on the submission itself,
 * which is far harder to read than "the file at this URL says something else".
 */

And the check reports the mismatch in the most boring, most useful way available, by quoting what it found:

const body = (await response.text().catch(() => "")).trim();
if (body !== key) {
  const preview = body.length > 60 ? `${body.slice(0, 60)}...` : body || "(empty)";
  return { ok: false, url, message: `contains ${preview}, expected ${key}` };
}

The refusal that follows tells you the consequence and the escape hatch in two lines:

die(
  `Key file ${check.url} ${check.message}.\n` +
    "  Every submission will 403 until it matches. Fix it, or pass --skip-verify to submit anyway.",
);

Naming the escape hatch in the error message matters more than it sounds. A guard with no documented way past it is a guard somebody will eventually delete, because the one time it is wrong they will be in a hurry. --skip-verify exists so that the answer to "the check is wrong and I need to ship" is a flag rather than a commit.

Refuse the ambiguous invocation instead of guessing

Two flavours of this, and neither is clever:

if (opts.sitemap && opts.urls.length > 0) {
  die("Pass either --sitemap or an explicit list of URLs, not both.");
}
if (!opts.sitemap && opts.urls.length === 0) {
  die("Nothing to submit. Pass one or more URLs, or --sitemap.");
}

--sitemap with extra URLs could plausibly mean "the sitemap plus these", so it could have been a union. I made it an error because I cannot tell which of the two the person typing it meant, and a tool that guesses at intent in a command that talks to four search engines is a tool that will one day submit 235 URLs when you meant three.

The empty case is the same principle from the other side. Zero URLs is not a successful run of nothing, it is an invocation that did not say what it wanted.

Unknown engines get the same treatment, with the valid set in the message so you do not have to go and read the source:

if (!Object.hasOwn(INDEXNOW_ENDPOINTS, opts.engine)) {
  die(`Unknown engine: ${opts.engine}. Expected one of ${Object.keys(INDEXNOW_ENDPOINTS).join(", ")}.`);
}

Object.hasOwn rather than in or a truthiness check, so --engine constructor is an unknown engine rather than a crash.

Exit immediately, or exit at the end

There are two ways this script can fail, and they use different mechanisms on purpose.

Everything above is die(), which prints and calls process.exit(1) straight away. Nothing has been sent at that point, so there is nothing to finish and no reason to continue.

Once the submission loop starts, it switches:

let failed = 0;
for (const [index, batch] of batches.entries()) {
  ...
  if (!result.meaning.ok) failed++;
}

if (failed > 0) {
  console.error(`✗ ${failed} of ${batches.length} batches failed`);
  process.exitCode = 1;
}

A failed batch does not abort the run, because the other batches are independent and half a submission is better than none. And the exit status is set rather than taken, so the pending output flushes before the process leaves.

That split is worth adopting as a habit. process.exit while you still have work or unflushed output is how you lose the line you needed; process.exitCode while you are validating arguments is how you accidentally continue into the thing you just decided not to do.

The refusal that is only a comment

One thing the script does not enforce, and I went back and forth on it. The docblock says:

 * Submitting the same URL repeatedly with no change to the page is the one
 * documented way to get throttled, so prefer passing the handful of pages a
 * deploy actually touched over running --sitemap out of habit.

--sitemap exists and pulls every URL out of the live sitemap.xml, which for this site is a few hundred pages. Running it after every deploy would be the easy habit and the wrong one.

I could have made the script refuse --sitemap unless some page actually changed, and there is a separate mechanism that answers that question. But putting that logic here would mean this command needed to know about build output and content hashes, which is a lot of coupling for a script whose job is one POST. So it stays a sentence in the usage text, and the enforcement lives where it belongs.

Not every rule you want should be a guard. The ones worth building are the ones where the mistake is silent, and "I submitted too many URLs" is not silent, it is throttling with a 429 and a documented cause. The five refusals above all exist because their failures were invisible.