I build and run AI Change Watch: it crawls 15 AI vendors — OpenAI, Anthropic, Google, AWS Bedrock, Azure and the rest — and records every model deprecation, API change and price move as a dated, searchable event. That works out to a couple of thousand pages, each changing on its own schedule rather than on mine.
Building it also generates a steady supply of problems that took me longer to work out than they should have, so I'm going to start writing them up here: Cloudflare Workers, crawling at scale, edge caching, and the indexing plumbing that ties them together. This is the first one.
That volatility is what makes IndexNow a pretty natural fit. You POST a list of URLs, and Bing, Yandex, Naver and Seznam learn that those pages changed — no crawl budget negotiation, no waiting. It costs nothing and takes about twenty lines of code.
I run it from a Cloudflare Worker. For weeks, the worker reported success.
Bing had never received a single URL.
The submitter that couldn't tell you it was broken
The first problem wasn't network-level at all. It was this:
export async function submitIndexNow(urls: string[]): Promise<boolean> {
const list = [...new Set(urls)].slice(0, 10000);
if (list.length === 0) return false; // <-- nothing to send
// ...
for (const endpoint of ENDPOINTS) {
const res = await fetch(endpoint, { method: 'POST', body });
if (res.ok) return true;
}
return false; // <-- everything rejected us
}
Both of those false returns surfaced to the caller as submitted: 0. "There was nothing new to send" and "every endpoint refused the batch" were the same observable event. A quiet day and a completely broken submitter looked identical in the logs, which is why this went unnoticed for weeks.
So before diagnosing anything, I made the two states say different things:
console.warn(JSON.stringify({
at: 'indexnow', outcome: 'rejected', endpoint,
status: res.status, urls: list.length,
body: (await res.text()).slice(0, 200),
}));
The very next run printed this:
{"at":"indexnow","outcome":"rejected","endpoint":"https://www.bing.com/indexnow",
"status":429,"urls":150,
"body":"{\"errorCode\":\"TooManyRequests\",\"message\":\"We're sorry, but you have sent too many requests to us recently.\"}"}
Bing was returning 429 TooManyRequests. Every time. And because my endpoint list falls through on failure, the batch was quietly landing on Yandex instead — the one search engine I wasn't submitting for.
If you take one thing from this post: never let "nothing to do" and "the request failed" produce the same log line.
Proving it's the sender, not the payload
A 429 usually means "slow down", so the obvious suspects are batch size, frequency, or a bad host/key pair.
To rule those out, I kept the payload, the key, the keyLocation and the timing constant, and changed only where the request originated. All three of these went out within the same minute:
| Sender | bing.com | api.indexnow.org | yandex.com |
|---|---|---|---|
| Cloudflare Worker | 429 | 429 | 200 |
| Residential IP (my laptop) | 200 | 200 | 200 |
| GitHub Actions runner | 200 | 200 | 200 |
Same key. Same body. Same minute. The only variable was the sender.
Volume isn't the trigger either. Going back through older logs, a 4-URL batch got the same 429 three days earlier. A 150-URL batch gets it today. There is no batch size small enough to slip under it.
What this proves, and what it doesn't
Worth separating, because the two are not the same strength of claim.
What the experiment establishes:
- The same submission succeeds or fails depending purely on where it is sent from.
- The rejection is not a function of batch size, and not a problem with the key or
keyLocation— those were identical in all three runs. - Therefore no amount of backoff, jitter, or chunking on my side would have fixed it.
What I'm inferring: that this is about shared egress IP reputation or per-IP rate limiting. Cloudflare Workers egress from address ranges shared with a very large number of other tenants, many of whom also submit to IndexNow, so a per-client limit would be consumed collectively.
That's a working hypothesis, not a demonstrated fact. Bing doesn't publish how its IndexNow endpoint rate-limits, and I have no way to observe it from outside. The official Bing IndexNow docs don't cover this case.
Fortunately the fix doesn't depend on the hypothesis being right. Whatever the mechanism, the actionable finding is the same: the request has to leave from somewhere else.
The fix: move the POST, keep the policy
The obvious move is "submit from somewhere else". The trap is rewriting the whole submitter in that somewhere else.
My Worker doesn't just POST a list. It decides what to send: it reads the site's own sitemap, ranks URLs by whether they've never been submitted, whether their lastmod moved, and how long since the last submission, caps the run, and records what landed in KV. That logic is derived from the site's indexing policy. Duplicating it in a CI script means two copies that drift the first time either side changes.
So I split the use case in two, and left both halves in the Worker:
// Reads the sitemap, picks the batch. Records NOTHING.
async plan(): Promise<SitemapPlan> { /* ... */ }
// Records a batch as sent. Called only after it was accepted.
async commit(batch: SitemapUrl[]): Promise<number> { /* ... */ }
Exposed as two endpoints, the GitHub Actions job becomes a courier that never interprets what it carries:
const plan = await admin('/admin/indexnow/pending?scope=sitemap');
if (plan.outcome === 'sitemap-unavailable') fail('the sitemap could not be read');
if (plan.batch.length === 0) { log({ outcome: 'nothing-due' }); process.exit(0); }
const urlList = plan.batch.map((u) => u.loc);
// ... POST urlList to the IndexNow endpoints ...
if (!accepted) fail(`every endpoint rejected ${urlList.length} URLs`);
// Hand the batch straight back, verbatim.
await admin('/admin/indexnow/commit', {
method: 'POST', body: JSON.stringify({ batch: plan.batch }),
});
Two details that matter more than they look:
plan() records nothing. If the submission fails, the same URLs come back in the next plan. An earlier version marked URLs as sent before the POST, which meant one rejected batch silently retired 150 URLs for the whole refresh window — while reporting success.
commit() re-reads its state and drops foreign hosts. The batch makes a round trip through an external job, so on the way back it is untrusted input, not a trusted snapshot.
How it fits together
Cloudflare Worker ──plan()──▶ batch of URLs
(sitemap, ranking, │
KV state) ▼
▲ GitHub Actions
│ │
│ │ POST
│ ▼
│ IndexNow
│ (Bing first)
│ │
└────commit(batch)─────────────┘
on success only
The workflow itself is unremarkable, which is the point:
on:
schedule:
- cron: '17 */3 * * *' # odd minute: GitHub queues heavily at :00
workflow_dispatch:
concurrency:
group: indexnow # two runs would fetch the SAME batch
cancel-in-progress: false
That concurrency block is load-bearing. Since nothing is recorded until a submission is accepted, two overlapping runs would each ask for a plan, receive the identical batch, and submit it twice.
First real run:
{"at":"indexnow","via":"actions","outcome":"submitted",
"endpoint":"https://www.bing.com/indexnow",
"submitted":150,"recorded":150,"listed":1906,"due":1006}
endpoint is the first one in the chain. Bing took it directly.
A side observation: Yandex is fast, Bing is patient
While all of this was still landing on Yandex, I pulled six hours of crawler traffic (1,000-event sample):
| Crawler | Hits |
|---|---|
| YandexBot | 185 |
| PetalBot | 37 |
| Applebot | 13 |
| bingbot | 3 |
| Googlebot | 1 |
YandexBot went from background noise to dominant right after those submissions. IndexNow demonstrably causes crawling — Yandex just acts on a ping far more eagerly than Bing does.
One clarification worth making, because I had it wrong myself at first: IndexNow shares the notification between participants, not the crawl. A URL you submit to Yandex is advertised to Bing as well, but Bing still has to send bingbot and build its own index. "Yandex crawled it" never becomes "Bing indexed it".
And if you're wondering whether any of this reaches Google: it doesn't. Google has been evaluating IndexNow since 2021 and still doesn't participate. Sitemaps and Search Console remain the only levers there.
What I changed
The final setup:
- The Cloudflare Worker owns URL selection and submission state.
- GitHub Actions owns the outbound POST.
- A failed submission is never recorded as sent, so the batch is safe to retry.
- "Nothing to submit" and "submission failed" are separate, named outcomes.
The important part wasn't moving the request to GitHub Actions. It was separating policy from transport — and noticing that only the transport had a problem.
Takeaways
- A successful
fetch()is not the same thing as a successful submission. - Log "nothing to do" and "request rejected" separately. Everything else here was invisible until that one change.
- When debugging a 429, send the identical request from a different network before assuming it's your rate.
- Keep selection and state out of the transport layer, so relocating the transport costs you one small script.
- Don't mark work as done until the receiving service says it accepted it.
All the numbers above are from production logs on 2026-08-09. This is also part of why AI Change Watch is built around event-level change detection rather than periodic re-crawling: when you record what changed and when, questions like "was this ever actually submitted?" have an answer you can look up.
Top comments (0)