Our most valuable tests do not test code, they assert that our content is true

CogniPrep is a content heavy Next.js app: a few hundred practice tests, a hub page per provider, a guide cluster, a blog, employer pages and format pages. Nearly all of it is driven by registries, which are plain TypeScript arrays and objects that other code derives routes, prices, search and navigation from.

Registries have a specific failure mode. They do not crash. The comment at the top of one of our test files says it better than I can:

A game with no registry entry renders a "coming soon" placeholder rather than
failing, a trait game left out of TRAIT_GAMES reports a percentile for a test
with no right answers, and a missing hub or cheating page is a 404 from several
places at once. None of that is caught by the typechecker.

That last sentence is the whole argument. ProviderId is a union type, so the compiler will stop you using a provider that does not exist. It will not notice that the provider exists everywhere except in the one array that the sitemap is built from.

The suite is 304 files and 6,563 tests, and it runs in about 16 seconds with Vitest. 40 of those files are what we call registration tests: one per provider, and they assert almost nothing about behaviour.

What a registration test actually asserts

A representative one, for a single provider:

it('has its public pages on disk and in dark mode with its siblings', () => {
  for (const page of [
    ['app', 'games', 'clevry', 'page.tsx'],
    ['app', 'cheating', 'clevry', 'page.tsx'],
    ['app', 'blogs', 'clevry-test-tips', 'page.tsx'],
    ['app', 'employers', 'royal-mail', 'page.tsx'],
  ]) {
    expect(existsSync(join(process.cwd(), ...page)), page.join('/')).toBe(true);
  }
  expect(FORCE_DARK_PATHS).toContain('/games/clevry');
});

There are 193 existsSync assertions like that across the 40 files. They exist because a registry entry is a promise that a route exists, and nothing else in the toolchain checks that promise. The sitemap is derived from the registry, so a broken promise ships as a URL in sitemap.xml that returns a 404, which is the most expensive kind of mistake on a site that lives on search traffic.

The same file also checks things that cross a boundary the typechecker cannot see:

it('uses icon names both ICON_MAPs know', () => {
  const sources = [
    readFileSync(join(process.cwd(), 'components', 'GameCard.tsx'), 'utf8'),
    readFileSync(join(process.cwd(), 'components', 'dashboard', 'ProviderGamesView.tsx'), 'utf8'),
  ];
  for (const game of getGamesByProvider('clevry')) {
    for (const source of sources) expect(source, game.id).toContain(`${game.lucideIcon}:`);
  }
});

Two components each hold a map from icon name to component. A name present in one and missing from the other renders a card with no icon in exactly one place in the app. Grepping two source files in a test is not elegant. It is three lines, it runs in milliseconds, and it catches a class of bug that only a human clicking through two surfaces would otherwise find.

And it checks derivation rather than values:

it('lands on the full price tier by derivation, not by hand', () => {
  const playable = getGamesByProvider('clevry').filter((g) => g.implemented !== false);
  expect(playable.length).toBeGreaterThan(SMALL_PROVIDER_MAX_GAMES);
  expect(getProviderPrice('clevry')).toBe(PROVIDER_TOKEN_PRICE);
});

The test does not assert that the price is a number. It asserts that the price is the one implied by the catalogue, so that adding a test to a provider cannot silently leave it on the wrong tier.

The assertions I find most interesting are the negative ones

Two kinds of test here exist to stop us from saying something untrue.

it('keeps Criterion out of the Criteria provider, which is a different company', () => {
  expect(PROVIDER_META.criteria.description).not.toMatch(/criterion/i);
});

Criterion is a Clevry product. Criteria is an unrelated assessment company. The two names differ by two characters, both appear in our search index, and a well meaning copy edit that adds "Criterion" to the Criteria description would be a factual error about a real company on a page Google indexes. A not.toMatch is a cheap way to make that mistake loud.

// Expert averages are unpublished and must never be guessed.
expect(GAME_CONFIG.clvVerbalExpert.publishedAverageSeconds).toBeNull();

That one asserts the absence of data. The field exists to hold a number the provider has published; for some tests no such number exists, and the only correct value is null. Without the test, the first person who wants a nicer looking page fills it with a plausible figure. The test turns "we do not know this" into a property of the system.

The off-by-one that was hiding in a length check

Every registration test checks the page title and meta description length, because the site rule is a title strictly under 60 rendered characters and a description under 160. For a while the assertion read:

expect(guide!.metaTitle.length).toBeLessThanOrEqual(48);

The root layout sets title: { template: '%s | CogniPrep' }, and | CogniPrep is exactly 12 characters. So a 48 character title renders at exactly 60, which the rule does not allow. Every one of those assertions is now toBeLessThan(48), or toBeLessThanOrEqual(47) where it was written that way.

It is a one character change in seven files, and it is the kind of thing a length budget always has: the number you assert on is not the number you care about. If a template, a prefix or a separator sits between your string and the rendered output, write the arithmetic down next to the assertion, because the next person will read 48 and assume it is 60 minus nothing.

A test over a literal is also not the final word, so the length rule is checked twice: once in these tests, and once after a real build by extracting <title> and <meta name="description"> from the prerendered HTML. The second one is the audit, the first one is the fast feedback.

See it

Open any of these and read the tab title, or run document.title.length in the console:

• The Clevry hub: 57 characters rendered, description 154.
• The Clevry guide: 58 and 146.
• A Clevry blog post: 55 and 157.
• The TestGorilla hub: 55 and 153.
• A Royal Mail employer page: 42 and 150.

Every one of those is under the limit, and the descriptions sitting at 154 and 157 are the reason the test exists. There is no slack to absorb a mistake.

If you have a content registry

Three questions worth asking of your own:

1. What does a missing registry entry do? If the answer is "renders a placeholder" or "404s somewhere else", you need a test per entry, not a type.
2. Which promises does an entry make about files that exist? Assert them with existsSync. It feels crude and it is the highest value test in our suite.
3. What must never be said? Write that as not.toMatch. The compiler has no opinion about facts.