DEV Community

Luca
Luca

Posted on

Every IndexNow ping from our Cloudflare Worker got 429: shared egress IPs and a relay

I build IndexCyborg, a tool that automates the indexing chores of a website: it reads the sitemaps, checks pages, asks Google's URL Inspection API what it knows, and notifies search engines through IndexNow when pages are new or changed. It runs on Cloudflare Workers.

On November 5 every IndexNow notification sent from production came back 429 Too Many Requests. Not some of them: every one, for hours. The same POST, with the same key and the same URLs, sent from my own PC got 202. Sent from a Supabase Edge Function it got 200.

This post is what we tried, what didn't work, and the small relay that fixed it.

IndexNow in one paragraph

IndexNow is a protocol that Bing, Yandex, Seznam, Naver and a few other engines share. You host a key file at https://example.com/<key>.txt, then POST a JSON body to an endpoint:

{
  "host": "example.com",
  "key": "your-key",
  "keyLocation": "https://example.com/your-key.txt",
  "urlList": ["https://example.com/new-page"]
}
Enter fullscreen mode Exit fullscreen mode

The participating engines share submissions with each other, so one accepted notification reaches all of them. Google doesn't take part.

The documented answers are worth knowing by heart:

  • 200 received, 202 received with key validation still pending
  • 400 bad format, 403 key not valid, 422 URLs don't belong to the host or the key doesn't match
  • 429 too many requests, "potential spam"

200 and 202 mean the engine received the URLs. They don't mean anything was crawled or indexed.

Why a tiny sender got rate-limited

We were sending a handful of notifications per site per day, so volume wasn't the problem. What changed between "refused" and "accepted" was only where the request came from.

IndexNow doesn't document how it rate-limits, so this part is an inference: the limit is applied per caller IP, and the egress IPs of a serverless platform are shared with a lot of other customers. Some of them presumably send IndexNow traffic too. From the engine's side, our request was one more POST from a busy address.

If your serverless function talks to an API that throttles by IP, you inherit the reputation of everyone you share an exit with.

Attempt 1: try the next engine

Any participating engine can take the notification, so the first fix was a fallback: if api.indexnow.org refuses, try www.bing.com/indexnow with the same body.

The important detail is which failures move on. A 429, a 5xx or a network error might be different elsewhere. A 403 about the key or a 422 about the host would be the same at every engine, so retrying those just repeats the refusal:

export const INDEXNOW_ENDPOINTS = ["https://api.indexnow.org/indexnow", "https://www.bing.com/indexnow"];

export function isTransientIndexNowFailure(result: IndexNowResult): boolean {
  if (result.ok) return false;
  if (result.statusCode === null) return true; // network error
  return result.statusCode === 429 || result.statusCode >= 500;
}
Enter fullscreen mode Exit fullscreen mode

It didn't help. Bing answered 429 too, which fits the IP theory: same caller, same verdict.

Attempt 2: stop hammering

Every refused attempt also wrote a "submission failed" row and an activity event, so a single notification could leave a trail of failures in the dashboard. And every retry was one more request to an endpoint that was already saying "too many".

Now a transient refusal is retried with a delay that grows with the job's age, for six hours, and only the final outcome is recorded:

export const RETRY_WINDOW_MS = 6 * 3600 * 1000;

// 10 minutes at first, then half the job's age, at most an hour.
export function retryDelayMs(jobAgeMs: number): number {
  return Math.min(Math.max(10 * 60 * 1000, jobAgeMs / 2), 60 * 60 * 1000);
}
Enter fullscreen mode Exit fullscreen mode
const transient = isTransientIndexNowFailure(result);
const jobAgeMs = Date.now() - new Date(job.created_at).getTime();
if (transient && jobAgeMs < RETRY_WINDOW_MS) {
  throw new RetryLater(`IndexNow transient failure: ${result.error}`, retryDelayMs(jobAgeMs));
}
Enter fullscreen mode Exit fullscreen mode

When the window runs out, or when the refusal is permanent, the job throws a PermanentFailure instead of finishing quietly. A notification job that reads "completed" should mean an engine accepted it.

That made the failures honest. It didn't make them succeed.

The fix: send it from somewhere else

Since the refusals followed the IP, the notification had to leave from a different one. We already use Supabase, and a request from a Supabase Edge Function had been accepted, so the relay became an Edge Function: the Worker hands it the notification, it posts to the engines and reports what they answered.

A relay that posts arbitrary JSON to arbitrary URLs on request would be an open proxy, so it's deliberately narrow.

It authenticates callers with a shared secret, and only the hash lives in the source:

async function sha256Hex(text: string): Promise<string> {
  const digest = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(text));
  return Array.from(new Uint8Array(digest), (b) => b.toString(16).padStart(2, "0")).join("");
}

// Both sides are fixed-length hex digests; compare without stopping at the first difference.
function sameDigest(a: string, b: string): boolean {
  if (a.length !== b.length) return false;
  let diff = 0;
  for (let i = 0; i < a.length; i++) diff |= a.charCodeAt(i) ^ b.charCodeAt(i);
  return diff === 0;
}
Enter fullscreen mode Exit fullscreen mode

The function is deployed with Supabase's JWT check turned off, because the caller is our Worker, not a Supabase user. The plaintext secret exists only as a Worker secret.

It rebuilds the notification from the four fields IndexNow takes, and refuses URLs that aren't on the notified host (ignoring a leading www.):

if (!onSite(keyLocation)) return "keyLocation must be a URL on host";
if (!urlList.every(onSite)) return "every URL must be on host";

return { host, key, keyLocation, urlList: urlList as string[] };
Enter fullscreen mode Exit fullscreen mode

It tries both engines and reports every attempt, so the Worker can log what happened at the relay, not just the last status code:

for (endpoint of INDEXNOW_ENDPOINTS) {
  last = await post(fetchImpl, endpoint, notification);
  attempts.push({ endpoint, statusCode: last.statusCode });
  if (!isTransient(last.statusCode)) break;
}
return json(200, { ...last, endpoint, attempts });
Enter fullscreen mode Exit fullscreen mode

The Worker side

The Worker tries the relay first when it's configured, then falls back to posting directly. The order matters: the Worker's own IP is the one being refused, so trying it first only adds refused requests.

const relay = indexNowRelay(); // null unless INDEXNOW_RELAY_URL and INDEXNOW_RELAY_SECRET are set
if (relay) {
  const viaRelay = await postViaRelay(relay, payload);
  last = viaRelay.result;
  if (!isTransientIndexNowFailure(last)) return last;
  tried.push(...viaRelay.tried);
}

for (const endpoint of INDEXNOW_ENDPOINTS) {
  last = toResult(await postIndexNow(payload, endpoint), endpoint);
  if (!isTransientIndexNowFailure(last)) return last;
  tried.push(`${new URL(endpoint).host}: ${last.statusCode ?? "network error"}`);
}
Enter fullscreen mode Exit fullscreen mode

One distinction kept debugging sane: "the relay refused me" is not an IndexNow answer. A wrong secret, an undeployed function or a malformed reply is reported as relay unusable, separately from what the engines said. Without that, a typo in a secret would look like IndexNow rate-limiting us again.

The submissions table also records which endpoint answered, so "accepted via the relay" and "accepted directly" are visible later.

Testing a Deno function from a Node test suite

Edge Functions run on Deno, the app's tests run on Vitest. The relay's logic lives in a handler.ts with no Deno APIs (only fetch, Request, Response and Web Crypto), and the Deno entry point is three lines:

import { handleRelay } from "./handler.ts";

const SECRET_SHA256 = "<hex sha-256 of the secret>";

Deno.serve((req: Request) => handleRelay(req, { secretSha256: SECRET_SHA256 }));
Enter fullscreen mode Exit fullscreen mode

The handler takes an optional fetchImpl, so tests pass a stub and check the secret check, the host filter and the fallback order without touching the network.

Did it work?

Yes. The first production notification sent through the relay was accepted with a 200, after a day in which nothing had got through.

It's not free of trade-offs. It's another moving part with its own secret to rotate. Supabase's egress is shared too, so nothing guarantees it stays welcome forever. That's why the direct endpoints stay as a fallback and every attempt is recorded. If it starts failing, the logs will say where.

Checklist if you call IndexNow from serverless

  1. Treat 429, 5xx and network errors as "try elsewhere or later". Treat 403 and 422 as "fix the key or the URLs".
  2. Back off for real. Fast retries against a rate limit make it worse.
  3. Record the final outcome once, not every attempt.
  4. If refusals follow the platform rather than your volume, send from a different egress: a relay function on another provider, or any machine with its own IP.
  5. Keep the relay narrow: authenticated, limited to the payload IndexNow takes, limited to your own hosts.
  6. Don't tell users "indexed" when an engine says 202. It means "received".

I wrote this with the help of an AI assistant, working from the project's commits and code. The snippets come from the repository, trimmed for length.

Top comments (0)