Notifio is a desktop app that watches rental search pages and emails you when a new listing appears. The app and the website are two codebases in one repository, and they have two very different relationships with testing. The desktop side has no test framework at all, just three scripts and a rule about imports. The website has a real suite, and I had never counted what it covers.
The suite, measured
$ node --test
ℹ tests 140
ℹ suites 34
ℹ pass 140
ℹ fail 0
ℹ duration_ms 1865
Five files, 1,388 lines, 257 assertions. No Jest, no Vitest, no mocking library, no DOM, no database, no network. The runner is Node's own, which reads the TypeScript directly and charges one price for it: the @/ path alias does not exist, so every test imports with a relative path and a .ts extension.
file it() cases lines
pricing/display-currency.test.ts 42 46 461
seo/indexnow.test.ts 31 31 292
stripe/managed-payments.test.ts 23 23 222
stripe/checkout-session.test.ts 14 18 255
seo/metadata-length.test.ts 4 22 158
114 it() calls produce 140 tests, because some of them are written inside loops over a data table and each row becomes its own case. That is the only indirection in the whole suite.
What the suite can reach
Every import in every test file, deduplicated, is eight source modules:
src/lib/pricing/display-currency.ts
src/lib/pricing/constants.ts
src/lib/stripe/checkout-session.ts
src/lib/stripe/managed-payments.ts
src/lib/seo/indexnow.ts
src/lib/compare/targets.ts
src/lib/articles/audiences.ts
src/lib/articles/guides.ts
Eight of the 79 TypeScript files under src. 2,271 lines of 12,007. The last three are data rather than code, so the number of actual functions under test is smaller than it already looks.
Put the subject matter next to the file list and the shape is unmistakable. Five files: what currency to quote a visitor, how to build a Stripe line item, when a sale is misrouted for tax, what to tell Bing, and whether a title fits in a search result. The website has 140 tests and not one of them is about rental listings.
That is deliberate, and here is the rule it follows
The suite covers the failures that are silent.
If the currency logic picks wrong, the desktop app shows a price the server told it to show and the buyer is charged something else. Nobody files that bug; they just do not buy. If a title drifts two characters over the budget, Google truncates it and the page quietly earns less. If the IndexNow submitter posts a URL that does not exist, the response is still a 200. Every one of those is a problem you find in a dashboard, months later, with no way to tell when it started.
The untested half of the codebase fails loudly instead. The eight route handlers are:
/api/validate licence check from the desktop app
/api/notify the alert email the app asks us to send
/api/checkout buy a licence
/api/checkout/upgrade buy the auto-reply upgrade
/api/contact the form on /help
/api/download/[platform] a 302 to a signed installer URL
/api/webhooks/stripe issue a licence after payment
/api/webhooks/resend forward inbound mail
If activation breaks, the next person who buys writes to support within the hour. If the alert email stops, somebody whose rental search went quiet tells us the same day. A test suite is one way to learn about a bug and an annoyed customer is another, and for most of that list the customer is faster and more accurate.
The one place that argument falls down
The Stripe webhook. Its failure mode is a person who paid and has no licence, which is loud, but it is also the worst hour of that person's day and it is the most intricate code in the repository:
} catch (err) {
// Concurrent webhook won the race and inserted first. Fall back to
// the row that won so we still deliver a valid token.
const [row] = await db
.select()
.from(licenses)
.where(eq(licenses.email, email))
.limit(1);
if (!row) throw err; // genuine DB error → 500 so Stripe retries
token = row.token;
Three branches before that one: this exact session already processed, a different session for an email that already has a licence, and a fresh insert. All four paths have to end with the same token reaching the same inbox, because the token is the product. Not one of them is exercised by anything but a real payment.
It needs no database to test. Pass in a fake db with four scripted behaviours and assert the token that comes out. That is the next test file, and it is the only gap on the list that makes me uncomfortable.
The test that checks data instead of code
metadata-length.test.ts is four it() calls in the source, 22 cases at runtime, and the most valuable file of the five. Every page on the site must have a title under 60 characters and a description under 160, and the rendered title is what counts:
/** The suffix the root layout's title template appends to every child page. */
function titleTemplateSuffix(): string {
const layout = readFileSync(LAYOUT, "utf8");
const match = layout.match(/template:\s*"(.*?)"/);
assert.ok(match, "root layout no longer declares a title template");
It reads the root layout with a regular expression to find the | Notifio suffix, rather than trusting anybody to remember that ten of the sixty characters are already spent. Hand-counting is what let three titles drift to exactly 60 before this existed.
It also refuses to be given a list of what to check:
/**
* The alerts barrel uses extensionless imports that Node cannot resolve, so the
* site files are loaded straight from the directory. Globbing rather than
* listing them also means a new market file is covered the day it is added.
*/
A limitation of the runner turned into the better design. Those four calls cover all 15 site pages, the five comparisons, the five guides and the three audience pages, and they will cover the sixteenth site page without an edit.
What I would change
Nothing about the ratio. A 140-test suite that runs in under two seconds with no dependencies gets run, and a suite that needs a database does not. The two things I am taking from the census are the webhook file above, and the fact that these tests only ever run when I type node --test: there is no CI in this repository at all, which I have learned the hard way on a sister project is how you find out two of them are red.
The product the tests are protecting is at notifio.app: a one-time licence rather than a subscription, installers for Windows and both Mac architectures, the per-site setup guides at notifio.app/alerts, and the questions the contact form exists for at notifio.app/help.
Top comments (0)