DEV Community

Никита Кривда
Никита Кривда

Posted on

Your CI is probably pinging IndexNow wrong

#devops #webdev #seo #astro

Bing Webmaster Tools threw a recommendation at my site last week that I had never seen before:

IndexNow is in batch mode.

No error, no broken page, nothing red. Just a note that the way I was submitting URLs was the wrong shape. It took me a minute to understand what it was actually complaining about, and the fix turned out to be four lines.

What IndexNow is for

IndexNow is a tiny protocol: you POST a list of changed URLs, and Bing, Yandex and Seznam get told to come look. You prove ownership by serving a key file at your domain root, so there is no OAuth, no login, no API console. That is the whole appeal.

POST https://api.indexnow.org/indexnow
{
  "host": "example.com",
  "key": "<the key you also serve at /<key>.txt>",
  "keyLocation": "https://example.com/<key>.txt",
  "urlList": ["https://example.com/page/"]
}
Enter fullscreen mode Exit fullscreen mode

Because it is so easy, the obvious move is to wire it into CI and forget about it. That is exactly what I did:

- name: Deploy
  run: wrangler pages deploy dist
- name: Ping IndexNow
  run: npm run indexnow
Enter fullscreen mode Exit fullscreen mode

And npm run indexnow, with no arguments, submitted every URL in the sitemap. All 693 of them. On every push.

Why that is the wrong shape

The protocol allows up to 10,000 URLs per request, so I was not breaking any rule. But "allowed" and "useful" are different things.

The signal IndexNow is designed to carry is this specific thing changed, now. If you send your entire sitemap on every deploy, you are not carrying that signal anymore. You are sending a daily dump where 692 entries are noise and one is news, and the crawler has no way to tell which is which. Do it often enough and the sensible thing for a search engine to do is stop treating your pings as urgent, which is precisely the value you wired it up for.

Bing's own guidance is to submit URLs at the time they change. The "batch mode" flag is what you get when your submissions look like a scheduled dump instead of a change feed.

The fix

My sitemap already carried <lastmod> on every page that has a meaningful modification date. So "what changed today" was sitting right there in a file I was already reading:

/** URLs whose <lastmod> is `day` (YYYY-MM-DD). Entries without <lastmod> never match. */
export function urlsChangedOn(xml: string, day: string): string[] {
  return [...xml.matchAll(/<url>(.*?)<\/url>/g)]
    .filter((m) => m[1].includes(`<lastmod>${day}</lastmod>`))
    .map((m) => m[1].match(/<loc>([^<]+)<\/loc>/)![1].replace(/&amp;/g, '&'));
}
Enter fullscreen mode Exit fullscreen mode

And the caller:

const today = new Date().toISOString().slice(0, 10);
const urls = args.length
  ? args
  : urlsChangedOn(await readFile('dist/sitemap.xml', 'utf8'), today);
if (!urls.length) {
  console.log(`indexnow: nothing with lastmod ${today}, skipping`);
  return;
}
Enter fullscreen mode Exit fullscreen mode

Yes, that is a regex over XML, and yes, I know. It is my own sitemap, generated by my own code, in a script that runs in my own CI. Pulling in a parser to walk a file I emit myself would be the actual mistake here.

Next deploy:

indexnow: submitted 7 URL(s) -> 200
Enter fullscreen mode Exit fullscreen mode

Seven, not 693. That day's news article, in seven locales. Exactly the thing that changed.

The part worth stealing

The lastmod trick only works if your sitemap has honest lastmod values. Plenty of generators stamp every entry with the build time, which makes the field worthless for this and mildly harmful for crawling in general: if everything changed, nothing did.

Mine is generated per entry from the content's own updated field:

const lastmod = (d?: string) => (d ? `<lastmod>${d}</lastmod>` : '');
Enter fullscreen mode Exit fullscreen mode

Static hub pages get no lastmod at all, which is the honest answer for a page whose content is assembled from other pages. It also means they never trip the daily filter, which is fine: they get recrawled on their own schedule anyway, and if I ever need to force one I can pass URLs explicitly.

So the checklist, in order of how much it matters:

  1. Emit lastmod from real content dates, or omit it. Never from build time.
  2. Submit what changed, not what exists.
  3. Keep the explicit-URL path, because sometimes you know better than the sitemap.

Two of those are one line each. The third was already there.

If you want to see the output of the thing, it is a GTA 6 fact database in seven languages: heistatlas.com. The sitemap generator and the IndexNow script are the same ones quoted above.

Top comments (0)