Your sitemap and your noindex tags disagree. Here's how to make that impossible.

#astro #typescript #webdev #seo

I found a sitemap.xml sitting in the public/ folder of a site I work on. It declared three URLs. The site had seventeen pages.

Fourteen pages had never been in the sitemap. Nobody wrote them out — they were simply added after someone typed that file by hand, and no one thought to go back.

That's the normal fate of a hand-written sitemap. It is correct on the day you write it and wrong from the next commit onward.

Generate it from the filesystem
In Astro, any file in src/pages that isn't a .astro page can be an API route. A sitemap.xml.ts file becomes /sitemap.xml, and it runs at build time.

import.meta.glob gives you every page file in the project, so the sitemap can read the same source of truth the router reads:

interface Entry {
loc: string;
changefreq: 'weekly' | 'monthly' | 'yearly';
priority: string;
lastmod?: string;
}

const DEFAULTS: Omit = { changefreq: 'monthly', priority: '0.5' };

const SETTINGS: Record> = {
'/': { changefreq: 'weekly', priority: '1.0' },
'/blog/': { changefreq: 'weekly', priority: '0.8' },
'/legal-notice': { changefreq: 'yearly', priority: '0.1' },
};

const PAGES: Entry[] = Object.keys(import.meta.glob('./*/.astro'))
.filter((path) => !path.includes('[')) // dynamic routes handled below
.map((path) => {
const route = path
.replace(/^./, '')
.replace(/\/index.astro$/, '/') // blog/index.astro -> /blog/
.replace(/.astro$/, ''); // about.astro -> /about
const loc = route === '' ? '/' : route;
return { loc, ...(SETTINGS[loc] ?? DEFAULTS) };
});
The important detail is the fallback. A page missing from SETTINGS still enters the sitemap, with default values. Forgetting to tune a page costs you an approximate priority — not its indexing. That inversion is the whole point: the failure mode has to be harmless, or you're back to maintaining a list by hand.

Dynamic routes are filtered out because [slug].astro has no URL of its own. Its real URLs come from your content source:

const posts = await getPosts(); // CMS, in our case Sanity

const blogEntries = posts.map((post) => ({
loc: /blog/${post.slug}/,
changefreq: 'monthly' as const,
priority: '0.7',
lastmod: post.updatedAt ?? post.publishedAt,
}));
Add a page, publish an article: both land in the sitemap on the next build. Nobody has to remember anything.

The part that actually matters
Here's the bug that survives every "generate your sitemap automatically" tutorial.

A sitemap is a request. You are telling Google: please index these URLs. A noindex tag is the opposite instruction: do not index this page.

If a page carries noindex and appears in your sitemap, you are sending two contradictory instructions about the same URL. Google resolves it — noindex wins — but you've spent crawl budget asking for something you refuse, and Search Console will report the conflict back to you as an error you then have to triage.

Automatic discovery makes this more likely, not less, because the glob knows nothing about your rendering logic.

So the noindex condition has to be evaluated in the sitemap too, from the same source:

// A draft or unpublished article is noindex — it must not be requested.
const blogEntries = posts
.filter((post) => !post.seo?.noindex)
.map((post) => ({ /* … */ }));

// A page behind a feature flag is noindex until the flag flips.
let pages = FEATURE.seo.noindex
? PAGES.filter((p) => p.loc !== '/feature-page')
: PAGES;

// A hub page with nothing published yet has nothing to show either.
if (!items.some(isPublished)) {
pages = pages.filter((p) => p.loc !== '/hub/');
}
Three different reasons for the same rule: if a page can go noindex, the sitemap has to know why.

The alternative is a sitemap that drifts in the other direction — technically complete, semantically wrong, quietly asking Google to index pages that reject it.

Give lastmod something true to say
lastmod is only useful if it reflects a real change. A build timestamp on every URL is worse than no lastmod at all: it says everything changed, every deploy, which tells the crawler nothing.

Take the date from the content:

lastmod: post.updatedAt ?? post.publishedAt
For pages that aggregate — an author page, a category index — the freshest thing they contain is the right answer:

lastmod: author.posts[0]?.updatedAt ?? author.posts[0]?.publishedAt
And when a page has no meaningful date, omit the field. It's optional, and an omitted lastmod is more honest than a fabricated one.

What changed
Seventeen pages in the sitemap instead of three. Two pages correctly absent from it, because they're noindex. New articles enter on publication, new pages on the next build.

The maintenance cost is now zero, which is the only maintenance cost anyone actually pays.

DEV Community

Your sitemap and your noindex tags disagree. Here's how to make that impossible.

Top comments (0)