DEV Community: Issa Hadjidj

Your sitemap and your noindex tags disagree. Here's how to make that impossible.

Issa Hadjidj — Mon, 17 Aug 2026 14:38:53 +0000

I found a sitemap.xml sitting in the public/ folder of a site I work on. It declared three URLs. The site had seventeen pages.

Fourteen pages had never been in the sitemap. Nobody wrote them out — they were simply added after someone typed that file by hand, and no one thought to go back.

That's the normal fate of a hand-written sitemap. It is correct on the day you write it and wrong from the next commit onward.

Generate it from the filesystem
In Astro, any file in src/pages that isn't a .astro page can be an API route. A sitemap.xml.ts file becomes /sitemap.xml, and it runs at build time.

import.meta.glob gives you every page file in the project, so the sitemap can read the same source of truth the router reads:

interface Entry {
loc: string;
changefreq: 'weekly' | 'monthly' | 'yearly';
priority: string;
lastmod?: string;
}

const DEFAULTS: Omit = { changefreq: 'monthly', priority: '0.5' };

const SETTINGS: Record> = {
'/': { changefreq: 'weekly', priority: '1.0' },
'/blog/': { changefreq: 'weekly', priority: '0.8' },
'/legal-notice': { changefreq: 'yearly', priority: '0.1' },
};

const PAGES: Entry[] = Object.keys(import.meta.glob('./*/.astro'))
.filter((path) => !path.includes('[')) // dynamic routes handled below
.map((path) => {
const route = path
.replace(/^./, '')
.replace(/\/index.astro$/, '/') // blog/index.astro -> /blog/
.replace(/.astro$/, ''); // about.astro -> /about
const loc = route === '' ? '/' : route;
return { loc, ...(SETTINGS[loc] ?? DEFAULTS) };
});
The important detail is the fallback. A page missing from SETTINGS still enters the sitemap, with default values. Forgetting to tune a page costs you an approximate priority — not its indexing. That inversion is the whole point: the failure mode has to be harmless, or you're back to maintaining a list by hand.

Dynamic routes are filtered out because [slug].astro has no URL of its own. Its real URLs come from your content source:

const posts = await getPosts(); // CMS, in our case Sanity

const blogEntries = posts.map((post) => ({
loc: /blog/${post.slug}/,
changefreq: 'monthly' as const,
priority: '0.7',
lastmod: post.updatedAt ?? post.publishedAt,
}));
Add a page, publish an article: both land in the sitemap on the next build. Nobody has to remember anything.

The part that actually matters
Here's the bug that survives every "generate your sitemap automatically" tutorial.

A sitemap is a request. You are telling Google: please index these URLs. A noindex tag is the opposite instruction: do not index this page.

If a page carries noindex and appears in your sitemap, you are sending two contradictory instructions about the same URL. Google resolves it — noindex wins — but you've spent crawl budget asking for something you refuse, and Search Console will report the conflict back to you as an error you then have to triage.

Automatic discovery makes this more likely, not less, because the glob knows nothing about your rendering logic.

So the noindex condition has to be evaluated in the sitemap too, from the same source:

// A draft or unpublished article is noindex — it must not be requested.
const blogEntries = posts
.filter((post) => !post.seo?.noindex)
.map((post) => ({ /* … */ }));

// A page behind a feature flag is noindex until the flag flips.
let pages = FEATURE.seo.noindex
? PAGES.filter((p) => p.loc !== '/feature-page')
: PAGES;

// A hub page with nothing published yet has nothing to show either.
if (!items.some(isPublished)) {
pages = pages.filter((p) => p.loc !== '/hub/');
}
Three different reasons for the same rule: if a page can go noindex, the sitemap has to know why.

The alternative is a sitemap that drifts in the other direction — technically complete, semantically wrong, quietly asking Google to index pages that reject it.

Give lastmod something true to say
lastmod is only useful if it reflects a real change. A build timestamp on every URL is worse than no lastmod at all: it says everything changed, every deploy, which tells the crawler nothing.

Take the date from the content:

lastmod: post.updatedAt ?? post.publishedAt
For pages that aggregate — an author page, a category index — the freshest thing they contain is the right answer:

lastmod: author.posts[0]?.updatedAt ?? author.posts[0]?.publishedAt
And when a page has no meaningful date, omit the field. It's optional, and an omitted lastmod is more honest than a fabricated one.

What changed
Seventeen pages in the sitemap instead of three. Two pages correctly absent from it, because they're noindex. New articles enter on publication, new pages on the next build.

The maintenance cost is now zero, which is the only maintenance cost anyone actually pays.

Your CMS migration killed URLs you don't know about. The Wayback Machine knows which ones.

Issa Hadjidj — Mon, 17 Aug 2026 14:26:12 +0000

We moved voxtrend.fr from Webflow to Astro. The redirects were written, the build was green, the site was live. Two months later I went looking for something else and found eleven URLs still sitting in Google's index, all returning 404.

None of them were in the redirect map. Not because anyone was careless : because nobody remembered they existed.

The failure mode
When you migrate, you write redirects for the URLs you know about. You get them from the old CMS export, or from the sitemap, or from memory.

That covers the pages you published. It misses:

pages you deleted a year before the migration, which Google still has
demo content from the CMS template you never cleaned up
URL variants the old CMS generated on its own
pages in a language or section you dropped along the way
Every one of those may still hold external links. A 404 throws that away.

The Wayback Machine has the list
The Internet Archive exposes a CDX API that returns every URL it ever captured for a domain. No key, no account, no rate limit worth worrying about for a one-off.

curl -s "https://web.archive.org/cdx/search/cdx?url=voxtrend.fr&matchType=domain&output=json&fl=original&collapse=urlkey&filter=statuscode:200&limit=500" -o cdx.json
The parameters that matter:

matchType=domain — the whole domain, not just the exact URL
collapse=urlkey — one row per URL instead of one per capture
filter=statuscode:200 — only pages that actually resolved
fl=original — return just the URL column
Then reduce it to unique paths and drop the assets:

import json, re, urllib.parse

rows = json.load(open('cdx.json'))[1:] # first row is the header
paths = set()

for (url,) in rows:
p = urllib.parse.urlparse(url).path.rstrip('/') or '/'
if re.search(r'.(png|jpe?g|svg|css|js|woff2?|ico|webp|pdf)$', p, re.I):
continue
paths.add(p)

for p in sorted(paths):
print(p)
What that turned up
Fifty-two historical paths. Most were already redirected. Eleven were not, and they fell into two very different groups.

Real pages, still linked from outside:

/about-us
/politique-de-confidentialite
/privacy-policy
/blog-2
/connexion
/creation_de_compte
Leftovers from the Webflow starter template:

/blog-post/best-prototyping-tools-12
/blog-post/best-prototyping-tools-13
/career-post/interface-designer-2
/integrations-post/blossom
/blog-categories/banking
Someone had launched from a template and never deleted the demo content. It sat there long enough to get crawled.

Check which ones are still live before you write anything:

for p in $(cat paths.txt); do
printf "%-44s " "$p"
curl -s -o /dev/null -w "%{http_code}\n" "https://voxtrend.fr$p"
done
Redirect the first group. Let the second one 404.
This is the part people get wrong, and it's counter-intuitive.

For the real pages, redirect to the closest equivalent. /about-us to the homepage if there's no about page anymore. /privacy-policy to the French version. Those pages existed, people linked to them, the link should land somewhere meaningful.

For the template junk, do nothing. Let it 404.

Bulk-redirecting a hundred dead URLs to your homepage is a documented soft-404 pattern. Google treats a redirect to an irrelevant page the same as a 404, and you've added a hop for nothing. A page that never had real content has nothing to preserve. A clean 404 is the correct answer.

Two things I checked while I was in there
Vercel's permanent: true emits a 308, not a 301. If your SEO tool flags "308 instead of 301", ignore it : Google treats them identically. Both are permanent, both pass signals. The difference is that 308 preserves the HTTP method, which is irrelevant for page redirects.

The canonical host was inconsistent. astro.config.mjs declared site: 'https://voxtrend.fr' while every page and the sitemap used https://www.voxtrend.fr. Nothing was broken, because nothing read Astro.site yet. The day you add @astrojs/sitemap or an RSS feed, they emit non-www URLs that all bounce through a redirect. Worth fixing before it becomes a real bug.

The whole thing takes twenty minutes
One API call, one script, one loop of status checks. It found six real URLs that had been quietly discarding whatever external links pointed at them.

If you migrated a site in the last year and only redirected what was in the CMS export, run the CDX query. The list is longer than you think.