My Next.js Sitemap Was Missing Pages for Weeks. I Fixed It by Ditching next-sitemap

#nestjs #javascript #seo #webdev

I spent two weeks wondering why Google Search Console kept showing "Discovered — currently not indexed" for entire sections of my Next.js app. The pages were live. Lighthouse passed. Everything looked fine locally.

Then I found it: every URL inside an App Router route group — like (marketing)/about or (shop)/products/[slug] — was completely absent from my sitemap. Not an error. No warning. Just silently missing from the XML file next-sitemap was generating.

This article is about that bug, why it exists, and the proper fix. I'll also show you a few things about Next.js sitemap generation that most tutorials skip over entirely.

The Problem: next-sitemap and App Router Route Groups

next-sitemap is the most downloaded sitemap package for Next.js. Around 416,000 weekly downloads. It has been the go-to choice for years — and honestly, for a long time it was excellent.

But it was built for the Pages Router. Its architecture is a postbuild script that crawls your built output and writes static XML files into public/. That model predates the App Router entirely.

GitHub issue #700 — open since August 2023 — documents that pages inside route groups like (marketing), (shop), or (auth) don't appear in the generated sitemap. Route groups are a standard App Router pattern. You use them to organize layouts without affecting the URL structure.

The dangerous part isn't that the bug exists. It's that there's no error message. The build succeeds. The sitemap generates. You deploy. And then quietly, over days and weeks, Google doesn't index those pages because they're not in your sitemap.

I only caught it during a routine Google Search Console audit. By then, it had been broken for almost three weeks.

Here's how to check if you're affected:

# After your build, inspect the generated sitemap
cat public/sitemap.xml | grep "(marketing)"
# If your route groups use parentheses, this should return nothing
# But those pages SHOULD be in your sitemap

# Better check: count your sitemap URLs vs your actual pages
cat public/sitemap.xml | grep -c "<loc>"

If the URL count is lower than your actual page count — and you're using route groups — you've likely hit this bug.

Why next-sitemap Can't Easily Fix This

The root issue is architectural. next-sitemap uses fs and path to crawl your built .next/server/pages and .next/server/app directories at build time. Route group folder names include parentheses, and the crawler strips them when constructing URLs (because that's the correct behavior for actual URLs) — but in doing so, it loses track of which files it has actually processed.

This is a hard fix. The logic for resolving route group paths to actual URLs is tightly coupled to how Next.js resolves them internally, and next-sitemap doesn't use Next.js internals directly.

Additionally, next-sitemap's last published version is 4.2.3 from approximately 2022. The project has not had a meaningful update since App Router shipped as stable.

The Fix: App Router Route Handlers

The cleanest solution for App Router projects is to stop using a build-time crawling script entirely and generate the sitemap dynamically via a Route Handler.

You keep full control over which URLs are included. Route group logic is irrelevant because you are building the URL list directly from your database or CMS — not inferring it from the filesystem.

Here's a minimal working example:

// app/sitemap.xml/route.ts
import { NextResponse } from 'next/server';

const hostname = 'https://yoursite.com';

export async function GET() {
  const staticUrls = [
    { loc: '/', priority: 1.0, changefreq: 'daily' },
    { loc: '/about', priority: 0.8, changefreq: 'monthly' },
    { loc: '/blog', priority: 0.9, changefreq: 'weekly' },
  ];

  const xml = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  ${staticUrls.map(url => `
  <url>
    <loc>${hostname}${url.loc}</loc>
    <changefreq>${url.changefreq}</changefreq>
    <priority>${url.priority}</priority>
  </url>`).join('')}
</urlset>`;

  return new Response(xml, {
    headers: { 'Content-Type': 'application/xml' },
  });
}

This works. But you're immediately writing XML escaping, URL validation, and changefreq type safety by hand. For a 5-page site, that's fine. For a product catalog with 10,000 dynamic URLs and image extensions, you want a library.

Adding Dynamic URLs with a Sitemap Library

For larger projects, I switched to @power-seo/sitemap — a newer package from CyberCraft Bangladesh that is built specifically around the Route Handler pattern. Zero Node.js-specific dependencies (so it works on edge runtimes), TypeScript-first, and supports image, video, and news sitemap extensions.

Here's the same product catalog example using it:

// app/sitemap.xml/route.ts
import { generateSitemap, validateSitemapUrl } from '@power-seo/sitemap';
import { db } from '@/lib/db';

export const dynamic = 'force-dynamic';

export async function GET() {
  const products = await db.product.findMany({
    select: { slug: true, updatedAt: true, imageUrl: true, name: true },
  });

  const urls = [
    { loc: '/', changefreq: 'daily' as const, priority: 1.0 },
    { loc: '/products', changefreq: 'daily' as const, priority: 0.9 },
    ...products.map((p) => ({
      loc: `/products/${p.slug}`,
      lastmod: p.updatedAt.toISOString(),
      changefreq: 'weekly' as const,
      priority: 0.8,
      images: [{ loc: p.imageUrl, caption: p.name, title: p.name }],
    })),
  ];

  // Catch bad data before Google does
  const invalid = urls.filter((url) => !validateSitemapUrl(url).valid);
  if (invalid.length > 0) {
    console.warn(`${invalid.length} invalid sitemap URLs detected`);
  }

  const xml = generateSitemap({ hostname: 'https://yoursite.com', urls });

  return new Response(xml, {
    headers: {
      'Content-Type': 'application/xml',
      'Cache-Control': 'public, max-age=3600, stale-while-revalidate=86400',
    },
  });
}

A few things worth noting here:

The image extensions actually matter for Google Images. If you have product photos, adding <image:image> tags to your sitemap entries is how Google Images picks them up alongside regular search results. next-sitemap doesn't support this in additionalPaths. Writing it by hand in raw XML is error-prone.

Runtime data means your sitemap is always fresh. A product added at 2pm appears in the sitemap the next time Google crawls /sitemap.xml — no rebuild required. With next-sitemap, your sitemap is frozen at build time.

validateSitemapUrl() is a safety net. One malformed URL in a large catalog can cause Google to reject the entire sitemap file. Catching bad data before you serve it is worth the two extra lines.

Handling Large Catalogs (50,000+ URLs)

The XML Sitemap spec caps a single file at 50,000 URLs. If you're building a large e-commerce site, a news site, or a marketplace, you'll need a sitemap index pointing to multiple child sitemaps.

@power-seo/sitemap handles this with splitSitemap():

// scripts/generate-sitemaps.ts
import { splitSitemap } from '@power-seo/sitemap';
import { writeFileSync, mkdirSync } from 'fs';

async function main() {
  const allUrls = await fetchAllUrls(); // your 75,000 SitemapURL[]

  const { index, sitemaps } = splitSitemap(
    { hostname: 'https://yoursite.com', urls: allUrls },
    '/sitemaps/chunk-{index}.xml'
  );

  mkdirSync('./public/sitemaps', { recursive: true });
  writeFileSync('./public/sitemap.xml', index);

  sitemaps.forEach(({ filename, xml }) => {
    writeFileSync(`./public${filename}`, xml);
  });

  console.log(`Generated ${sitemaps.length} child sitemaps + sitemap index`);
}

main();

For memory-constrained environments or very large catalogs, the streamSitemap() generator keeps memory usage constant by yielding XML chunks one <url> at a time — around 5x lighter on heap than building the full string in memory.

robots.txt Without next-sitemap

One thing next-sitemap does that @power-seo/sitemap intentionally doesn't: robots.txt generation. If you're migrating, the clean replacement is Next.js's native app/robots.ts convention:

// app/robots.ts
import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/' },
    sitemap: 'https://yoursite.com/sitemap.xml',
  };
}

Eight lines. No extra package. Native Next.js. Done.

What I Actually Learned From This

Silent failures are the most dangerous kind. A build error stops you. A missing warning doesn't. Always verify your sitemap against your actual page list — not just "does the file exist."
Build-time sitemap generation has real limitations. For content that changes frequently (products, blog posts, user profiles), a static sitemap file ages poorly between deployments.
The route group pattern is too important to work around. Route groups are core App Router structure. Choosing a sitemap tool that doesn't handle them means choosing a tool that doesn't work with how modern Next.js is meant to be used.
Image sitemaps are often neglected but easy to add. If you have product images, recipe photos, or editorial images, <image:image> extensions directly impact Google Images traffic. It's two extra fields per URL.

If you want to try the approach above, the source for @power-seo/sitemap is at: Power SEO Repo

Discussion

Is your Next.js sitemap actually showing all your pages — or are you just assuming it is?

If you're on App Router and using next-sitemap, it's worth a quick check: run cat public/sitemap.xml | grep -c "<loc>" after your next build and compare that count to your actual routes. The numbers might surprise you.

Are you using App Router route groups like (marketing) or (shop)? Have you checked if those pages are in your sitemap?

Drop your findings in the comments. I'm especially curious whether anyone has found a working workaround for the next-sitemap route group bug — I looked and couldn't find one that didn't require manually maintaining a URL list anyway (which defeats the purpose of a crawler-based tool).