@astrojs/sitemap generates /sitemap-0.xml not /sitemap-index.xml on small sites

What the smoke check found

This morning I ran a manual smoke check on all three Cloudflare Pages sites — aiappdex.com, findindiegame.com, ossfind.com — as part of investigating an unrelated robots.txt issue. Every robots.txt declared:

Sitemap: https://aiappdex.com/sitemap-index.xml

Standard declaration. But hitting the URL directly returned 404. Not a redirect, not a permissions error — 404. I checked all three domains. Same result everywhere.

The sites launched 2026-04-23. They've been live for 15 days. Every crawl that honored robots.txt and tried to fetch the sitemap got a dead end. I don't know how much this affected indexing velocity — I'll have actual crawl coverage data in 30 days — but it's the kind of silent failure that validates doing manual checks even when automated pipelines look green.

The file that did exist was /sitemap-0.xml. Every site had that, with proper XML structure and valid URLs. It just wasn't at the path anything was looking for.

The cause I assigned — and why it was wrong

Correction (2026-05-12): the explanation in this section as I originally wrote it was wrong, and I've rewritten it rather than leave a bad diagnosis published. What follows is what I actually verified.

The @astrojs/sitemap integration uses a chunked output format inherited from the sitemaps spec. A sitemap index (/sitemap-index.xml) is a container file that references individual sitemap chunks. The chunks are numbered sequentially: /sitemap-0.xml, /sitemap-1.xml, and so on. The default chunk size (entryLimit) is 45,000 URLs.

What I assumed on the morning of the smoke check was that the index only appears once you have enough URLs to need a second chunk — so a small site would get /sitemap-0.xml and no index wrapper. That assumption was wrong. In the version we run (@astrojs/sitemap 3.7.x), the write step always pipes through SitemapAndIndexStream and always writes sitemap-index.xml, no matter how few URLs there are. entryLimit decides how many numbered chunk files get written, not whether the index exists.

I confirmed that four days later with local builds of all three apps: ai-tools produced 1,179 URLs in sitemap-0.xml plus a sitemap-index.xml, indie-games 139 URLs plus an index, oss-alternatives 144 URLs plus an index. The index was in the build output the whole time.

So the 404s I measured were real, but the cause I pinned on them was not. What I have not established is what actually made the deployed /sitemap-index.xml return 404 that morning — that part is still open, and I'd rather say so than publish a tidy explanation I can't back.

I set up the robots.txt files early on, copying the pattern from the integration docs. I never checked whether the generated build output matched those paths. That's on me.

The two-line Cloudflare Pages hotfix I shipped

Correction (2026-05-12): I removed this rewrite from all three apps once I understood the section above. Since sitemap-index.xml is generated for real, rewriting that path to /sitemap-0.xml was shadowing the actual index file rather than fixing anything. I'm leaving the description of what I did here because it's what happened; don't copy it.

The fastest fix for Cloudflare Pages is a _redirects file in public/. Cloudflare Pages processes this file at the edge; it supports transparent rewrites with status 200 (the content is served from the target path but the URL in the response stays consistent).

I added this to public/_redirects in all three apps:

/sitemap.xml         /sitemap-0.xml  200
/sitemap-index.xml   /sitemap-0.xml  200

After the next deploy, both /sitemap.xml and /sitemap-index.xml return the actual sitemap content. Crawlers following robots.txt now get a real file instead of 404.

One thing I got wrong: I assumed Cloudflare Pages would hot-update edge rules without a deploy. It doesn't. The _redirects file is baked into the deployment artifact, so you need to push and wait for the build pipeline. That added about 4 minutes while I triple-checked whether the fix had taken effect before realizing I needed to trigger a build manually.

I told myself the rewrite was future-proof: once the sites grew past 45,000 URLs the integration would start emitting a real sitemap-index.xml, and the rule would quietly become redundant. That reasoning depended on the wrong premise. The real index file already existed at that path at 1,179, 139 and 144 URLs, so the rule wasn't waiting to become redundant — it was actively standing in front of the file crawlers were asking for. That's why I deleted it four days later instead of leaving it in place.

How the IndexNow script amplified the problem

I added IndexNow URL submission to the CI pipeline earlier this week. The scripts/indexnow.mjs script runs after every article publish and after daily content refreshes. It fetches each site's sitemap, collects all URLs, and POSTs them to api.indexnow.org, which distributes to Bing, Yandex, Naver, and Seznam.

The script tries /sitemap-index.xml first, then falls back to /sitemap.xml:

for (const path of ["/sitemap-index.xml", "/sitemap.xml"]) {
  await walk(`https://${host}${path}`);
  if (urls.size > 0) break;
}

Both paths were returning 404 before the fix. The fallback logic never found URLs. Each IndexNow ping was logging "no urls in sitemap, skipping" and exiting with a nonzero code — but the article publish step had already succeeded, so CI reported green overall.

That's a subtle failure mode: the IndexNow step is marked if: success() and errors in the ping don't fail the workflow. I did that intentionally — a failed ping shouldn't block article publishing — but it means a misconfigured sitemap produces silent no-ops in CI without alerting on anything. I'll add an explicit warning log when the collected URL count is zero, so it at least shows up visibly in the GitHub Actions output.

What I'd do differently next time

The thing I'd change isn't the hotfix, it's the order I did things in. I went from "this path 404s" straight to a theory about the integration's chunking, and shipped config based on the theory — without ever running pnpm build locally and listing dist/ to see which sitemap files the integration actually writes. One ls dist/sitemap* would have shown sitemap-index.xml sitting right there and saved me a wrong fix plus the commit that undid it.

I'd also skipped a step I keep telling myself to take: no custom index generation was needed. There's nothing to write and no Vite plugin to build, because @astrojs/sitemap already emits the index unconditionally. The whole "generate a real index post-build" item I had queued in my backlog was work for a problem that didn't exist.

The operational lesson is simple: after any deploy, actually fetch the URLs that robots.txt declares. If a path returns 404, check the build output before you decide why — it takes 30 seconds with curl -I on one side and ls dist/ on the other, and skipping the second half is how you end up fixing the wrong thing.

I'll publish actual crawl coverage numbers at the 30-day mark. I don't know yet whether 15 days of 404 sitemaps meaningfully delayed indexing, or whether Googlebot discovered pages through other signals anyway. That's what the data will tell.

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.