DEV Community

Daniel Pertu
Daniel Pertu

Posted on

Our website's whole test suite is 140 tests in five files, they import 8 of its 79 source files, and three of the five are about money

notifio.app is a Next.js site with 38 public pages, eight API routes and a Stripe integration that charges people in 24 currencies. I wrote about the runner a while back, which is node --test with no framework and the one real cost being that @/ path aliases do not resolve. I never wrote down what the tests cover.

$ npm test
...
ℹ tests 140
ℹ suites 34
ℹ pass 140
ℹ fail 0
ℹ duration_ms 1830.847167
Enter fullscreen mode Exit fullscreen mode

The whole suite:

__tests__/pricing/display-currency.test.ts     461 lines   12 suites   79 asserts
__tests__/seo/indexnow.test.ts                 292 lines    9 suites   73 asserts
__tests__/stripe/checkout-session.test.ts      255 lines    5 suites   56 asserts
__tests__/stripe/managed-payments.test.ts      222 lines    6 suites   37 asserts
__tests__/seo/metadata-length.test.ts          158 lines    2 suites   12 asserts
Enter fullscreen mode Exit fullscreen mode

Five files, 1,388 lines, 140 tests, 257 assertions, green in 1.8 seconds. Against that: 79 TypeScript files and 12,007 lines in src/.

$ grep -h 'from "\.\./\.\./src' __tests__/*/*.ts | sort -u | wc -l
8
Enter fullscreen mode Exit fullscreen mode

The tests import eight of the 79 files. Nothing stubs, nothing mocks a module graph, nothing renders a component. Eight modules are tested and 71 are not.

What the five files are actually about

Three of the five, and 938 of the 1,388 lines, and 172 of the 257 assertions, are about taking money correctly: the currency a visitor is quoted in, the amount Stripe is told to charge, and the tax behaviour of Stripe Managed Payments.

One is about the IndexNow script that pings search engines after a deploy: sitemap parsing, host partitioning, URL chunking, key file checks.

One, the smallest, is the only file that knows the site has pages at all.

Which produces the honest summary of this repository's priorities: the two most heavily tested things on the website are charging the right amount and telling Bing about it. Nothing else has ever broken in a way that cost me anything, and the tests I have are the scars.

The things with no test at all

Listed plainly, because the list is the point:

  • all 8 API routes, including checkout, the licence validator, the contact form handler and the installer redirect
  • all 16 page components under src/app
  • the 4 animated product demos on the homepage
  • the email templates, including the alert mail
  • lib/license.ts, the function that decides whether somebody has paid
  • proxy.ts and the maintenance switch
  • the database layer, the Redis client, the rate limiters

Some of those I am at peace with. Page components on a static marketing site break visibly and immediately: if /pricing fails to render, the build fails, and a build that fails does not deploy. That is a real test, it is just not in __tests__.

The licence resolver is the one that bothers me. It is 15 lines, it is the single gate between paying and not paying, and the only thing standing behind it is that the route calling it answers a 403 while another route answers a 200 for the same failure, which I now know because I counted, not because anything asserts it.

The small file that does the most unusual work

metadata-length.test.ts enforces the rule that every rendered title is under 60 characters and every description under 160. To do that it has to know what the renderer does, so it reads the renderer as text:

/** The suffix the root layout's title template appends to every child page. */
function titleTemplateSuffix(): string {
  const layout = readFileSync(LAYOUT, "utf8");
  const match = layout.match(/template:\s*"(.*?)"/);
  assert.ok(match, "root layout no longer declares a title template");
  const [, template] = match;
  assert.ok(template.includes("%s"), `unexpected title template: ${template}`);
  return template.replace("%s", "");
}
Enter fullscreen mode Exit fullscreen mode

The root layout sets title.template to "%s | Notifio", which spends ten of every page's sixty characters before the page has written a word. Rather than restating those ten characters in the test, it greps them out of the layout and fails loudly if the template stops existing.

It does two more things I would not have thought to write and am glad are there:

const SITES_DIR = path.join(import.meta.dirname, "../../src/lib/alerts/sites");
const ALERT_SITES = (
  await Promise.all(
    readdirSync(SITES_DIR)
      .filter((name) => name.endsWith(".ts"))
      .map((name) => import(path.join(SITES_DIR, name))),
  )
).flatMap((module) => Object.values(module).flat());
Enter fullscreen mode Exit fullscreen mode

It globs the per-site content directory instead of importing the barrel, so a new market file is covered the day it is created rather than the day somebody remembers to add it to a list. And it guards against finding nothing:

it("finds the static pages to check", () => {
  assert.ok(pages.length >= 10, `only found ${pages.length} pages under src/app`);
});
Enter fullscreen mode Exit fullscreen mode

A directory-walking test that silently finds zero files passes. That one line is the difference between a test suite and a suite that has quietly stopped testing anything, and it is the assertion I would copy into any codebase doing this.

Its one real weakness is that it parses export const metadata object literals with a regex and returns null for anything it cannot read, and a null title is reported as unparseable rather than passed. Pages using generateMetadata are skipped entirely and covered by the data-driven half instead. It works because every page on this site writes its metadata the same way, and it is one refactor away from quietly skipping a page.

The assertion that refuses to import a constant

My favourite four lines in the suite:

/** The margin the module adds. Restated here on purpose: a test that imported */
/** the constant would pass if the constant were deleted. */
const MARGIN = 1.04;
Enter fullscreen mode Exit fullscreen mode

Every foreign price on the site is the pound price times a fixed rate times that margin, which covers Stripe's conversion fee. If the test imported MARGIN from the module, then deleting the margin would change both sides of the comparison and the test would still be green. Restating the number is the only way the assertion means anything.

The suite built on it is also the slowest thing in the run: 1,629 ms of the 1,830 ms total goes into one suite, looping every list price across every supported currency and checking that no local figure nets less than the pound price. Ninety per cent of my test time is spent on the one property I cannot afford to get wrong, which is an accident I would not change.

No CI, and what that actually costs

$ ls .github
ls: .github: No such file or directory
Enter fullscreen mode Exit fullscreen mode

These 140 tests have never run anywhere except this laptop. There is no workflow, no pre-push hook, no check on the Vercel build, which means the only thing enforcing them is me typing npm test, and the only reason that has been enough is that the repository has 55 commits and I wrote all of them.

The fix is about six lines of YAML and I keep not writing them, so here is the honest ranking of what I would do to this suite, in order:

  1. Run it on push. Six lines, removes the only failure mode that is purely about me forgetting.
  2. Test the licence resolver, including the case where the email matches and the token does not.
  3. A smoke test per API route that asserts the status code for a malformed body, since ten distinct codes come out of those eight routes and nothing has ever checked one of them.

What I would not do is chase the ratio. Eight of 79 files sounds like a scandal and mostly is not: a static content site's failure modes are a build error, a wrong price, and a sentence that stopped being true. The build catches the first, these tests catch the second, and nothing catches the third, which is why I keep auditing the live pages by hand and writing down what I find.

Top comments (0)