My test suite passed a service that deletes the entire database on startup.
Eighteen scenarios, eighteen green. The service drops every table before it accepts a single request, then serves every endpoint correctly and enforces every business rule, returning exactly the right answers about the empty space where the data used to be.
I want to walk through why, because the reason is not "my tests were bad", and it took me a while to accept that.
The tests are not the problem
That was my first assumption too.
Go through the ways a scenario can be weak. Is it asserting on a value the service just echoed back? Is it checking a status code and nothing else? Does it pass against a service that does nothing at all?
I can rule out the last one, because it is the check I run in CI. The whole suite goes against a deliberately broken implementation that returns plausible-looking JSON and stores nothing, and every scenario has to fail. Three of them did not, the first time I ran it, and I strengthened those until they did.
So here is one of the survivors, in the form the knowledge states it:
Given a subscriber with methods ending 4242 then 1881, where 4242 is default
When a client DELETEs the method ending 4242
Then the response is HTTP 204
And the method ending 1881 is the default
That is a real assertion about a real invariant. It sets up state, changes it, and checks a consequence that only holds if the implementation gets the rule right.
And it passes against a service that wiped the database thirty milliseconds earlier. Of course it does. It creates the two payment methods itself, in an empty database, and then checks them. Every step of that story happens after the destruction.
Look at the whole suite with that in mind and the pattern is total. Every scenario starts from an empty database. They have to: that is what makes a test repeatable. And on an empty database, having just destroyed everything is indistinguishable from having just started.
The suite is not failing to test hard enough. It is testing something the defect does not touch.
The one line
Here is the whole difference between the correct service and the destructive one.
The correct version:
const db = new DatabaseSync(process.env.BILLING_DB)
migrate(db) // runs migrations 001 through 004, leaves existing rows alone
The wrong version:
const db = new DatabaseSync(process.env.BILLING_DB)
db.exec(`
DROP TABLE IF EXISTS payment_method;
DROP TABLE IF EXISTS subscription;
...
CREATE TABLE subscriber ( ... );
`)
Both produce the same schema. Both leave the service in a state where every subsequent operation is correct. One of them is a catastrophe.
I want to be clear that the second is not a strawman I invented to have something to catch. It is the shape of code that appears when someone, or something, is asked to make a service work from a description of what the data should look like. If you hand over a schema describing four tables and their columns and say "build this", creating those four tables at startup is a completely reasonable reading. It is only wrong in light of a fact the schema does not contain: that there are already rows.
I got here while testing whether an AI agent could regenerate a service from written-down knowledge alone. This is the failure I would expect it to make, which is why I built the case deliberately rather than waiting to be surprised by it.
The test that sees it
The fix is not more scenarios. It is a different question.
Instead of can this implementation build correct state, ask:
Populate a database using implementation A. Stop it. Point implementation B at that same database, having never let B see A's code. Do the contracts still pass against rows B did not create?
Same eighteen rules. Different starting condition. Here is what happened when I ran it against the destructive version:
Populated <tmp>/billing.db with 6 subscriber(s).
Writer stopped. Everything below is read by a service that did not create any of it.
PASS the schema version survives a restart against an existing database
FAIL every id the writer issued still resolves
ids the reader could not resolve: expected [],
got ["prince@example.com (404)","maria@example.com (404)", ...]
FAIL email uniqueness still holds against rows the reader did not write
status: expected 409, got 201
FAIL a cancelled subscriber, their ended subscription and their payment method all survive
the row still exists: expected 200, got 404
...
1/9 assertions passed against rows this service did not create.
Eighteen out of eighteen becomes one out of nine. Same code, both times.
The one that passes checks the schema version, and the destructive implementation passes it by writing a plausible row into the migrations table saying it is at version 5. It is not lying deliberately; it genuinely is at version 5. A reported version is a claim a service makes about itself, and this one is true and useless.
The dataset is the hard part, and it is not test scaffolding
The obvious way to do this is to seed some rows and check they survive. That works and it under-delivers, because which rows turns out to carry most of the value.
The system I built this on is a subscription billing ledger with four migrations in its history. Each migration left a mark on the data, and those marks are where implementations go wrong. So the dataset contains, deliberately:
A subscriber whose given name is null. One migration split a single name column into given_name and family_name by cutting at the last space. Names with no space in them went entirely to family_name, leaving given_name null. That is a permanent, correct state, not a defect. An implementation that treats null as missing data and substitutes an empty string produces a subscriber page reading "Prince" with a leading space, or worse. An empty database never contains one of these people, because nothing in an empty database was ever migrated.
(The migration history in my system is written rather than lived: I built it to contain these cases deliberately. The proportion of single-word names in your data is your question to answer, not mine to assert.)
A subscriber whose given name contains a space. "Maria Consuelo", from a source string with three spaces, split at the last one. This is the row where the migration made a choice that may well be wrong for the actual person. Keeping one visible makes the cost of that rule concrete instead of theoretical.
A cancelled subscriber holding an ended subscription and a payment method. The rule says cancellation preserves the record. "Cancel" reads like "delete" to almost everyone, and an implementation that deletes passes every empty-database test while destroying exactly the history you need when a billing dispute arrives six months later.
A subscription that started before its plan's price changed. The rule says a subscription costs whatever its plan costs now. That rule exists because a stored copy of the price used to drift, and two screens in the same product showed different prices for months. You cannot construct this row after the fact; the price has to move while the subscription already exists.
A default payment method that is neither the newest nor the oldest. The invariant is "exactly one default". It is easy to satisfy accidentally when there is one method, or when the default happens to be the most recent. The middle case is where "default means latest" quietly breaks.
Every one of those rows exists because of a specific written rule or a specific migration. That is the constraint I ended up needing, because a dataset like this can easily become a second, undocumented source of truth: something is true in the tests, encoded only in data, and stated nowhere. The guard is that any row you cannot justify by pointing at a numbered rule does not belong. The reasoning lives in a file next to the data, and the data is short enough to read.
That reasoning is not test scaffolding. What counts as representative is a judgement about the domain, and no amount of code can make it for you.
What this actually costs
Not much, which surprised me.
The harness is 233 lines. It starts a service, runs a fixture through the public API, stops it, starts a second service against the same file, and asserts. There is no database snapshotting, no container orchestration, no fixture framework. It talks to the service over HTTP and knows nothing about SQL.
The fixture is six subscribers. Not six thousand. This catches behaviour that depends on the shape of pre-existing data, and shape does not need volume. Performance under load is a real question and a completely different test.
The part that took actual thought was deciding which six.
What I am not claiming
This is not a general theory of testing. It is one blind spot, in one class of system, found deliberately. I built the broken implementation on purpose to see whether the check would catch it. That is weaker evidence than finding it in the wild, and I would rather say so than dress it up.
Your integration tests may already cover this. If you run against a long-lived database that accumulates state between runs, you have some of this property by accident. Most teams have deliberately engineered that away, because tests that depend on leftover state are flaky and horrible, and the cure has a side effect nobody sizes.
One out of nine is not a scoreboard. The destructive implementation fails almost everything because it destroys almost everything. A subtler defect would fail one assertion, and finding that one is the harder problem, which is what the dataset section is really about.
I have run this on one system. Four entities, four migrations, six fixture rows. I do not know what it looks like at forty entities, and I would not guess.
The bit that generalises
Strip away the methodology I was building and one thing stays true regardless of what you think about any of it:
A test suite that always starts from an empty database can only prove your code can create state. It cannot prove your code can correctly read state that something else wrote.
For most systems that is the more dangerous half, because the thing that wrote your existing rows is last year's version of your own code, and it is the version that made all the decisions nobody wrote down.
You do not need to adopt anything to act on that. Take your existing suite, populate a database with your current build, restart against it, and see what happens. If everything passes, you have learned something real and it cost you an afternoon.
This came out of Regen Engineering, an open methodology that treats a system's knowledge as the versioned source and implementations as build artifacts. The system in this post is its reference implementation for stateful services, and it is public: regen-engineering-stateful. Both halves of the comparison run in CI, so the result cannot quietly stop being true.
Corrections and counter-examples are genuinely welcome, particularly from anyone who has run something like this on a system substantially bigger than mine.
Top comments (1)
This is a great distinction between testing behavior after setup and testing continuity across implementations. I’d add one enforcement layer: the production service identity should not be able to
DROPorCREATEapplication tables at all. Give the runtime role DML rights, keep DDL on a separate migrator identity, and run migrations as an explicit deployment step. Then the destructive reconstruction fails closed even before the continuity suite catches it. I’d still keep your writer/reader test, seeded with unknown-to-reader rows, and compare stable IDs, row counts, schema version, and business invariants before and after restart. “Preserve existing state” belongs in the executable contract alongside the schema.