DEV Community: Никита Кривда

Your CI is probably pinging IndexNow wrong

Никита Кривда — Fri, 14 Aug 2026 13:01:30 +0000

Bing Webmaster Tools threw a recommendation at my site last week that I had never seen before:

IndexNow is in batch mode.

No error, no broken page, nothing red. Just a note that the way I was submitting URLs was the wrong shape. It took me a minute to understand what it was actually complaining about, and the fix turned out to be four lines.

What IndexNow is for

IndexNow is a tiny protocol: you POST a list of changed URLs, and Bing, Yandex and Seznam get told to come look. You prove ownership by serving a key file at your domain root, so there is no OAuth, no login, no API console. That is the whole appeal.

POST https://api.indexnow.org/indexnow
{
  "host": "example.com",
  "key": "<the key you also serve at /<key>.txt>",
  "keyLocation": "https://example.com/<key>.txt",
  "urlList": ["https://example.com/page/"]
}

Because it is so easy, the obvious move is to wire it into CI and forget about it. That is exactly what I did:

- name: Deploy
  run: wrangler pages deploy dist
- name: Ping IndexNow
  run: npm run indexnow

And npm run indexnow, with no arguments, submitted every URL in the sitemap. All 693 of them. On every push.

Why that is the wrong shape

The protocol allows up to 10,000 URLs per request, so I was not breaking any rule. But "allowed" and "useful" are different things.

The signal IndexNow is designed to carry is this specific thing changed, now. If you send your entire sitemap on every deploy, you are not carrying that signal anymore. You are sending a daily dump where 692 entries are noise and one is news, and the crawler has no way to tell which is which. Do it often enough and the sensible thing for a search engine to do is stop treating your pings as urgent, which is precisely the value you wired it up for.

Bing's own guidance is to submit URLs at the time they change. The "batch mode" flag is what you get when your submissions look like a scheduled dump instead of a change feed.

The fix

My sitemap already carried <lastmod> on every page that has a meaningful modification date. So "what changed today" was sitting right there in a file I was already reading:

/** URLs whose <lastmod> is `day` (YYYY-MM-DD). Entries without <lastmod> never match. */
export function urlsChangedOn(xml: string, day: string): string[] {
  return [...xml.matchAll(/<url>(.*?)<\/url>/g)]
    .filter((m) => m[1].includes(`<lastmod>${day}</lastmod>`))
    .map((m) => m[1].match(/<loc>([^<]+)<\/loc>/)![1].replace(/&amp;/g, '&'));
}

And the caller:

const today = new Date().toISOString().slice(0, 10);
const urls = args.length
  ? args
  : urlsChangedOn(await readFile('dist/sitemap.xml', 'utf8'), today);
if (!urls.length) {
  console.log(`indexnow: nothing with lastmod ${today}, skipping`);
  return;
}

Yes, that is a regex over XML, and yes, I know. It is my own sitemap, generated by my own code, in a script that runs in my own CI. Pulling in a parser to walk a file I emit myself would be the actual mistake here.

Next deploy:

indexnow: submitted 7 URL(s) -> 200

Seven, not 693. That day's news article, in seven locales. Exactly the thing that changed.

The part worth stealing

The lastmod trick only works if your sitemap has honest lastmod values. Plenty of generators stamp every entry with the build time, which makes the field worthless for this and mildly harmful for crawling in general: if everything changed, nothing did.

Mine is generated per entry from the content's own updated field:

const lastmod = (d?: string) => (d ? `<lastmod>${d}</lastmod>` : '');

Static hub pages get no lastmod at all, which is the honest answer for a page whose content is assembled from other pages. It also means they never trip the daily filter, which is fine: they get recrawled on their own schedule anyway, and if I ever need to force one I can pass URLs explicitly.

So the checklist, in order of how much it matters:

Emit lastmod from real content dates, or omit it. Never from build time.
Submit what changed, not what exists.
Keep the explicit-URL path, because sometimes you know better than the sitemap.

Two of those are one line each. The third was already there.

If you want to see the output of the thing, it is a GTA 6 fact database in seven languages: heistatlas.com. The sitemap generator and the IndexNow script are the same ones quoted above.

Every claim on my site carries its sources. Here is the schema that forces it.

Никита Кривда — Thu, 30 Jul 2026 09:43:58 +0000

I run a fact-check site for an unreleased game. That genre is a swamp: half the pages you find are somebody's guess reprinted six times until it reads like news. I wanted the opposite, so I made provenance a schema requirement instead of an editorial habit. If a claim has no source, the build fails.

Here is how that works in Astro, and what it cost me.

Sources live in the content schema, not in the prose

Every entity on the site is a YAML file validated by a Zod schema. The interesting part is that sources is not optional:

const sourceSchema = z.object({
  url: z.string().url(),
  date: z.string(),          // when the source said it, not when I read it
});

const base = {
  status: z.enum(['confirmed', 'trailer-spotted', 'rumor', 'debunked']),
  updated: z.string(),
  sources: z.array(sourceSchema).min(1),
};

export const entitySchema = z.object({
  name: z.string(),
  description: z.string(),
  sections: z.array(z.object({
    heading: z.string(),
    text: z.string(),
    status: z.enum(['confirmed', 'trailer-spotted', 'rumor', 'debunked']),
    sources: z.array(sourceSchema).min(1),   // per section, not per page
  })).optional(),
  ...base,
});

Two decisions in there matter more than they look.

Sources are per section, not per page. A page usually mixes a confirmed fact with a plausible reading of a trailer. One source list at the bottom lets those blur together. Per-section sources force me to say which sentence rests on what.

Status is a required enum, not a boolean. rumor and debunked are first-class. The page renders a badge from the same field, so the reader sees the confidence level next to the claim instead of a disclaimer nobody scrolls to.

The cost is real: adding a paragraph means finding a citable source for it. Several times I have deleted a nice sentence because I could not back it. That is the feature working.

Seven locales, and the empty ones stay invisible

The site ships in seven languages, and translations land at different times. Nothing is worse for trust than a language switcher that dumps you on an English page, so the content layer gates on what actually exists in your locale:

export async function byLocale(collection: DbCollection, locale: Locale) {
  const all = await getCollection(collection);
  return all.filter((e) => e.id.startsWith(`${locale}/`));
}

// A section appears in the nav only where it has content in this locale.
export async function nonEmptyDbCollections(locale: Locale) {
  const pairs = await Promise.all(
    DB_COLLECTIONS.map(async (c) => [c, (await byLocale(c, locale)).length]),
  );
  return pairs.filter(([, n]) => n > 0).map(([c]) => c);
}

Everything downstream reads that one function: the nav, the hub pages, the sitemap. The practical payoff is that half-finished sections cannot leak. I have collections sitting in the repo right now with a validated schema, a rendering template and zero entries, because the game they describe is not out. They are invisible until the first file lands, and then the hub, the pages, the sitemap entries and the JSON-LD all appear at once. No launch-day checklist to forget.

The same gate feeds hreflang. Alternate links are generated only for locales that really have the page, and a CI script re-reads the built HTML and fails the build if any page claims a translation that does not exist. Hreflang lying about itself is a mistake you cannot see by looking at your source, only by looking at your output.

What I would do differently

I underestimated titles. Long headlines got clipped in the SERP by the search engine's own truncation, which means the engine chose the cut, not me. Clamping the <title> at a word boundary and leaving the <h1> full length is four lines of code I should have written on day one:

const clamp = (s, n) => {
  const x = s.trim();
  if (x.length <= n) return x;
  const cut = x.slice(0, n - 1);
  const sp = cut.lastIndexOf(' ');
  return (sp > n * 0.6 ? cut.slice(0, sp) : cut).replace(/[\s.,;:!?-]+$/, '') + '…';
};

Decorative images are a trap in both directions. I had scene art behind a video player marked alt="" aria-hidden="true", which is correct for a purely decorative wash. Two different site auditors flagged it as a missing alt anyway. The right fix was not to silence the auditors but to admit the art was not decorative: it is the only visual representing that page, so it deserved a real description.

Static output plus IndexNow is a good pairing. The build pings IndexNow with the URLs from the freshly generated sitemap as the last CI step, marked best-effort so a bad ping never fails a green deploy. Ownership is a key file at the domain root, so there is no dashboard login in the pipeline.

The part that has nothing to do with code

None of the above moves you up the results page on its own. I have a technically clean site sitting on page two, because a young domain with no links is a young domain with no links, and no amount of schema fixes that. What the discipline does buy is the thing you can act on later: when a journalist or a wiki editor checks whether you are worth citing, per-claim sources are the difference between a maybe and a yes.

If you want to see the shape of it, the sourced fact list is the rawest view of the data, and this piece on the game's setting shows the status-per-claim idea applied to an article where about half the interesting statements are honestly unprovable.

Happy to answer questions about the Astro side in the comments.