375 URLs in our sitemap come from seven sources, and the 132 biggest are found by reading a directory off disk

CogniPrep's app/sitemap.ts returned 375 URLs this morning. Grouping them by where the code got them from:

Source	URLs
readdirSync(app/employers)	132
blogPostMeta	99
ALL_PROVIDERS	56
CHEATING_GUIDES	56
AC_GUIDES	7
TEST_TYPES	5
INTERVIEW_GUIDES	5
typed out by hand	15

Six of those are TypeScript registries that other parts of the app also read, which is the arrangement you want: a provider that exists has a page, a search entry, a landing card and a sitemap row, all from one array, and the tests assert that parity. The 15 hand-written entries are the top-level pages.

The interesting one is the first row, because it is the only group the code discovers rather than reads:

const employersDir = join(process.cwd(), 'app', 'employers');
const employerSlugs = readdirSync(employersDir, { withFileTypes: true })
  .filter((entry) => entry.isDirectory())
  .map((entry) => entry.name);

132 directories in the app tree, turned into 132 public URLs by fs.readdirSync at build time.

Why that works at all

Reading the app directory at runtime on a serverless platform is exactly the trap this project has already hit once: Next traces imports, not file paths, so a route that reads files off disk deploys without them and fails in production while working perfectly in next dev. That is why next.config.mjs carries outputFileTracingIncludes entries for the data directories two API routes read at request time.

The sitemap escapes that because it never runs on a request. A sitemap.ts with no dynamic data access is evaluated during the build and written out as a file, and you can see that from outside:

curl -sI https://cogniprep.app/sitemap.xml | grep -iE 'last-modified|etag|age|accept-ranges|content-disposition'

accept-ranges: bytes
content-disposition: inline; filename="sitemap.xml"
etag: "a54c662d9e44b8cc70c4ae100c4eb130"
last-modified: Tue, 20 Oct 2026 12:49:08 GMT
age: 156983

accept-ranges, a content-disposition filename and a last-modified that sits at the deployment rather than at now: that is a static asset, not a function. The readdirSync ran once, on a build machine, with the whole repository on disk.

Which also means the group has a hidden dependency on staying static. The day someone adds a dynamic read to this file, the employer rows stop resolving and the sitemap quietly loses a third of the site. Nothing in the type system or the test suite says so. The comment above the code says the directory scan is there "so new pages are picked up automatically", which is true, and is not the part that needs writing down.

The other difference: what each source does when it is wrong

A registry row with no page gives you a sitemap URL that 404s. This project has tests for that, because a 404 in a sitemap is the kind of thing Search Console tells you about weeks later.

The filesystem group inverts the dependency: it cannot name a page that does not exist, but it will name any directory, and the only filter is isDirectory(). A route group such as (marketing), a private _components folder, or a directory holding a layout and no page would all become URLs. I checked today, and we are clean:

employer directories:            132
employer URLs in the sitemap:    132
directories with no page.tsx:      0

The page.tsx that renders the employer index sits in that same directory and is correctly skipped, because it is a file. So the current state is fine and the invariant is unasserted, which is the same position the registries were in before someone wrote the tests.

The dates are the actual bug

lastModified in this file is hand-maintained, one date per group, with a comment asking whoever edits a page to update it. Here is that promise meeting production.

# what the sitemap says about the cookie policy
curl -s https://cogniprep.app/sitemap.xml | grep -A1 '<loc>https://cogniprep.app/cookies</loc>'
# <lastmod>2026-05-01T00:00:00.000Z</lastmod>

# what the cookie policy says about itself
curl -s https://cogniprep.app/cookies | grep -o 'Last updated[^<]*'
# Last updated: 12 September 2026

Two public statements about the same document, four months apart, in the same deployment. The page is right and the sitemap is wrong.

It gets better on the legal pages, where both dates are wrong together. /terms and /privacy both carry lastmod 2026-10-03 and both print "Last updated: 3 October 2026", and both of them now contain wording about school email domains that shipped on 18 October:

curl -s https://cogniprep.app/terms | grep -o 'schools have a discount with us[^<]*'

So the content moved, and neither date followed it. The 99 blog URLs are the only group with dates I would trust, because they come from each post's own registry entry rather than from a constant in the sitemap. Everything else is a date somebody typed and nobody revisited: 133 of the 375 URLs share 2026-07-16.

The same file also emits priority and changeFrequency on all 375 rows, which Google has said plainly that it ignores. Harmless, and a good illustration of why lastmod is worth getting right: it is the one hint in the format that a crawler still reads, and it only works if it is consistently honest. Ours is honest in one group out of seven.

What I would change

1. Derive lastModified from git. git log -1 --format=%cI -- <path> at build time is the date the content changed, with no human in the loop. For groups where that is not meaningful, omit lastmod rather than guess: no date is a better signal than a date four months stale.
2. Keep the filesystem scan and assert it. One test: every directory under app/employers has a page.tsx, and every URL in the sitemap resolves to one. It would have been the cheapest of today's checks to automate.
3. Leave priority and changeFrequency. They cost nothing, and deleting fields to prove a point about Google's documentation is how you spend an afternoon.

See it

All of it is public. The sitemap is the 375 URLs, and the three grep commands above reproduce each number. The contradiction is easiest to see with two tabs: the cookie policy prints its own last-updated date at the top, and the sitemap entry for the same URL is four months behind it. The employers index is the page whose children are discovered by readdirSync, so every card on it is one directory on a build machine, and the blog index is the 99 URLs whose dates are derived rather than typed.