IndexNow Batch Mode: The lastmod Diff That Took 286 URLs Per Publish Down to 0

Bing Webmaster Tools put a banner on our IndexNow page: "IndexNow is in batch mode." The recommendation underneath was to stream instead — send URLs as they change rather than announcing the whole site at once. We were announcing 286 URLs on every publish, several times a week, because our submit script read sitemap-0.xml and POSTed every <loc> in it. Two new articles, 286 URLs.

The fix is about 40 lines, and it is not the part of IndexNow the protocol docs spend time on. The docs cover the key file, the POST body shape, and the 10,000-URL cap per request. They do not cover deciding which URLs belong in the request, which is the entire problem the moment your sitemap is larger than your publish.

What batch mode is actually measuring

IndexNow has no per-day quota to blow through. The endpoint accepts up to 10,000 URLs in one urlList and returns a 2xx either way. Nothing rejects a full-sitemap submission — you get a warning in a dashboard, and the stated cost is load on the engine plus slower handling of the changes you actually care about.

That framing decides what you do about it. This is not an error you can detect from the API response. Our script logged 200 OK on every one of those full-sitemap runs, for months. The only signal lived in a dashboard nobody opens daily.

A success response from api.indexnow.org means your payload parsed and your key file resolved. It says nothing about whether the engine will act on the URLs, and nothing about whether you are submitting sensibly. If you wired up IndexNow once and moved on, open the IndexNow page in Bing Webmaster Tools before assuming it is working.

The diff: lastmod is the state you already have

The precondition comes first, because it decides whether any of this works: your sitemap's lastmod has to reflect content changes, not build times. A sitemap integration will happily derive lastmod from file mtime, and our article generator rewrites post files on every run whether the prose changed or not. So astro.config.mjs reads each post's updatedAt frontmatter at config-load time and serializes that as lastmod instead. If lastmod is effectively new Date() at build, every URL differs from stored state on every build, the diff skips nothing, and you have written a slower version of the same batch submission.

Given a lastmod you can trust, the change is bookkeeping. Parse per-<url> blocks rather than bare <loc> tags, so location and timestamp stay paired:

function parseSitemapEntries(xml: string): Record<string, string> {
  const out: Record<string, string> = {};
  for (const block of xml.match(/<url>[\s\S]*?<\/url>/g) ?? []) {
    const loc = block.match(/<loc>([^<]+)<\/loc>/)?.[1]?.trim();
    if (!loc) continue;
    out[loc] = block.match(/<lastmod>([^<]+)<\/lastmod>/)?.[1]?.trim() ?? '';
  }
  return out;
}

const state = await loadState();
const urls = Object.entries(entries)
  .filter(([loc, lastmod]) => state[loc] !== lastmod)
  .map(([loc]) => loc);

State goes to a gitignored scripts/.indexnow-submitted.json. The first run after the change announced 286 URLs and wrote the file. The second run, with nothing published in between, printed Streaming mode: 0 changed, 286 unchanged (skipped) and sent no request at all. An --all flag forces the full list back for recovery.

Two ways this quietly does nothing

Both of these are live in our own script. Neither shows up on the happy path.

29 of our 286 URLs carry no lastmod at all. The built sitemap has 286 <url> blocks and 257 <lastmod> elements. The 29 without are the homepage, /about/, and the category and tag listings — routes the sitemap integration emits with changefreq and priority only, because the lastmod map is keyed on post frontmatter and these are not posts. They store as an empty string in state, so the filter compares '' !== '', gets false, and skips them permanently. The homepage changes on every single publish, since it lists the newest articles, and it now gets announced exactly once ever. The listing pages are the ones most worth streaming and they are precisely the ones the diff drops. The fix is to give those routes a real lastmod — the max of the posts they contain — not to special-case the empty string.

State is written before the POST is confirmed. Our script records entries to the state file and then calls submit(), which logs a non-2xx and moves on; the whole thing exits 0 by design so a syndication hiccup cannot break a deploy. Put those two properties together and a 403 from a rotated key marks all 286 URLs as announced while announcing none of them. The next run diffs clean and sends nothing. Recovery is one --all run, but you have to notice first, and nothing tells you.

If a script swallows errors on purpose, keep every durable state write behind the success check. Ours does not yet — that is the next change, and it is a two-line move, not a redesign.

The channels, and when to skip all of this

Three separate mechanisms get conflated. They are not interchangeable:

headers={['Channel', 'Auth', 'Ceiling', 'Best shape']}
rows={[
['IndexNow (Bing, Yandex, Seznam)', 'Key file at /.txt, no account', '10,000 URLs per request', 'Diff on publish'],
['Bing URL Submission API', 'API key from Webmaster Tools', '100/day, 1,300/month on our site', 'Daily cron draining a backlog'],
['Google', 'No public request-indexing API', 'Manual clicks in Search Console', 'Sitemap and patience'],
]}
/>

A sitemap ping tells an engine to re-read a file it already polls. IndexNow names specific URLs. Streaming logic only applies to the second — there is nothing to diff about the first.

If you would rather not own any of this, a hosted CMS maintains sitemap timestamps and search-engine pings for you, and the whole problem disappears along with the 40 lines.

The condition that flips it: if you publish from a repo and want lastmod bound to a frontmatter field you control rather than to a save event in an editor, hand-rolling wins, and 40 lines is the entire cost.

What we did not test: whether streaming changed indexing outcomes. Two days is not a result. Every claim here is bounded to submission behaviour — 286 down to 0 on an unchanged run, verified from the script's own output — not to crawl rate or index coverage. Search Console showed 633 pages crawled-and-not-indexed against 37 indexed on 2026-08-17, and if that number moves, IndexNow batching will be one of a dozen changes made in the same window. We will not be able to attribute it, and neither should you.

---

Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.