Our sitemap is 516 URLs and not one of them comes from the database any more

munchable.app/sitemap.xml is 516 URLs this morning. You can open it and count: 461 ingredient answer pages, 38 recipes, 8 condition guides, and 9 pages typed out by hand. Not one of those URLs is produced by a database query, and that is the whole point of this post, because for a long time nearly all of them were.

What it used to be

Munchable had a public product namespace: one page per barcoded product, generated from rows in our own catalogue. Tens of thousands of them. A sitemap of that size does not fit in a file, because the spec caps a single sitemap at 50,000 URLs, so the shape it forces on you is a sitemap index plus chunked child sitemaps.

That meant:

• a hand-rolled index route, listing the chunks
• dynamic child routes, one per chunk, so the rows could be paged out of Postgres at request time
• about 120 lines of XML serialisation, which is to say 120 lines where a stray ampersand in a product name becomes an invalid document
• a custom cache header on those routes, because the query behind a chunk was a ranking query, and a crawler happily fetching chunk after chunk would re-run it every time

None of that is hard. All of it is yours to maintain, including the escaping and the content type, and all of it exists because the URL set was too big to know at build time.

Why the product pages went

The browse list of most-scanned products answered no search anybody actually makes. Nobody types a barcode into Google. Tens of thousands of near-identical pages, each one a product name and an ingredients list, was the wrong bet: it is a lot of crawl budget spent on pages that are thin by construction and that compete with each other for the same weak queries.

What people do search is the question. "Is semolina gluten free." "Is onion low FODMAP." Those are the pages we kept, and there are 461 of them because that is how many (ingredient, condition) pairs we have an answer for, not because somebody decided 461 was a good number. The same goes for the recipes and the condition guides: every URL in the sitemap now comes from a TypeScript array somewhere in the repo.

What replaced it

Next's file convention. app/sitemap.ts, one exported function, returns an array of objects. The file is 73 lines and most of them are comment. The framework owns the serialisation, the escaping, the content type and the route, and because the whole URL set is known from code, it can prerender the document at build time instead of computing it per request.

export default function sitemap(): MetadataRoute.Sitemap {
  if (!IS_INDEXABLE) return [];

  return [
    { url: absoluteUrl('/'), lastModified: new Date(), changeFrequency: 'weekly', priority: 1.0 },
    { url: absoluteUrl('/conditions'), changeFrequency: 'monthly', priority: 0.8 },
    ...CONDITION_PAGES.map((page) => ({ url: absoluteUrl(`/conditions/${page.slug}`), ... })),
    ...ANSWER_PAGES.map((page) => ({ url: absoluteUrl(`/${page.question}`), ... })),
    // and so on
  ];
}

Two things in there are worth more than the deletion.

The first is IS_INDEXABLE. Every preview and branch deploy on Vercel gets its own hostname and is functionally identical to production, so robots.ts disallows everything on them. If the sitemap kept publishing 516 absolute production URLs from a preview host, the two files would be giving a crawler opposite instructions from the same deploy. Returning an empty array is how they agree.

The second is that there are 516 <priority> elements, 516 <changefreq> elements, and exactly one <lastmod>. Google has said plainly that it ignores the first two, and we emit them anyway because they cost nothing and other crawlers are not Google. The single lastmod is the homepage. Every other URL in that file is generated from source code, and the date a source file changed is not the date the page's content changed, so there is no honest value to put there. A lastmod that moves on every deploy is worse than none: it teaches the crawler that your dates mean nothing.

The thing I did not expect to delete

The prerendered product pages left a mark in a place with no obvious connection to a sitemap.

Our ingredient taxonomy has an overlay snapshot read through three tiers: an in-process memory cache, Redis, then Postgres. Redis here is Upstash, which is an HTTP client. A no-store fetch during a prerender makes Next abandon static rendering for that page, and the render fails. Postgres goes over a TCP connection, which Next does not treat as a dynamic data source at all, so it is fine.

So the reader function grew a noRedis option, added for exactly one caller: the prerendered product pages. Every other call site had to know not to need it. When the pages went, the option went with them, and the comment left in its place says what to do if a statically rendered caller ever turns up again: put the check inside the Redis tier, where it belongs, rather than back in the signature. A flag describing a property of the environment does not belong in a per-call-site argument.

That is the part of the clean-up I would have missed if I had only been thinking about XML. A big dynamic URL set does not only cost you the code that serialises it. It reaches back into shared functions and leaves options there, and those options are load-bearing until the day they are not.

If you want to see what is left, the sitemap is right there, and the answer pages it mostly consists of are grouped by condition on one index page, because a page that is only linked from a sitemap gets crawled slowly and ranked badly.