I wrote the script on a Friday afternoon because I was procrastinating on something worse.
It was maybe thirty lines. Walk the spec directory, parse each test file, count how many assertion calls appear inside each test body. I wasn't auditing anything. I was curious, in the idle way you get curious at 4pm on a Friday, and I expected to look at the output for ten seconds and close it.
Of our 400 specs, 61 contained no assertion at all.
Not a weak assertion. Not a bad one. Zero. Sixty-one tests that navigated somewhere, clicked some things, and ended. All of them passing. All of them counted in the number I reported to my manager every sprint.
I sat there for a while.
It gets worse when you look closer
The 61 were the honest ones, in a way. They were obviously empty once you opened them. The larger problem was the tests that looked like they asserted something.
Here are the four patterns I found, roughly in order of how many there were.
The existence check pretending to be a verification.
Order order = orderService.getOrder(orderId);
assertNotNull(order);
This test passes if the order exists. It passes if the order has the wrong total, the wrong line items, the wrong customer, a negative quantity, or a status of CANCELLED. It is not testing the order. It is testing that Java returned an object, which Java is quite reliable about.
I wrote a lot of these. I know exactly why I wrote them, and I'll get to that.
The soft assert that never lands.
SoftAssert sa = new SoftAssert();
sa.assertEquals(cart.getTotal(), 149.97);
sa.assertEquals(cart.getItemCount(), 3);
sa.assertTrue(cart.isTaxApplied());
// end of test
Three assertions. Zero of them can ever fail this test. SoftAssert collects results and only reports them when you call assertAll(), and there's no assertAll() here. Every failure is collected into an object that then goes out of scope and gets garbage collected.
This one genuinely frightened me, because it is invisible in review. It looks more rigorous than a hard assert. There are three assertions in it. A reviewer skimming the diff sees a thorough test. I found nine of these.
The tautology.
assertTrue(response.getStatusCode() >= 200);
Somebody wrote this meaning "the request succeeded." It passes on a 404. It passes on a 500. Status codes start at 100, so this is close to assertTrue(true) with extra steps.
The step counted as a check.
Our reporting counted steps. A test with fourteen steps looked more substantial in the dashboard than a test with three. But page.scrollToBottom() is not a check. page.waitForNetworkIdle() is not a check. We had tests whose step count was in the double digits and whose verification count was zero, and the dashboard rendered them identically to the tests that actually proved something.
Why I wrote them
This is the part I think matters more than the taxonomy, because the taxonomy is just symptoms.
The ticket said "automate TC-4471." The definition of done was "test exists, test is in the suite, test passes in CI." Nowhere in that sentence is the word "proves." I could satisfy every stated requirement of my job by writing a test that navigated to a page and asserted it wasn't null, and I did, repeatedly, for a couple of years, and I was praised for my throughput.
We reported test count in the sprint review. Four hundred automated tests. That number went up and to the right and everybody was pleased with it. Nobody has ever asked me, in any sprint review I've ever sat in, how many of those tests would go red if the product broke.
That's the actual bug. Not assertNotNull. The measurement system rewarded the existence of tests, so I produced the existence of tests. I'd have produced proof if proof was what got counted.
Coverage tooling didn't save me either, and this is worth being precise about. Line coverage tells you a line executed during a test run. It does not tell you that anything checked the result. A test with zero assertions that walks the entire checkout flow will light up hundreds of lines as covered. Coverage measures execution. We were reading it as proof. Those are not the same thing and the report doesn't distinguish them, because it can't.
Empty green and earned green are the same colour in every dashboard I've ever used.
The experiment that settled it
I wanted to know how bad it really was, so I did something I'd recommend to anyone reading this.
I broke the product on purpose.
I went into the discount calculation and changed a >= to a >. One character. This meant orders at exactly the threshold no longer received the discount, which is a real bug of the sort that generates real support tickets. Then I ran the full suite.
It went green. All 400.
There were eleven tests touching that code path. Every one of them executed the changed line. Not one of them checked the number that came out.
This idea has a proper name, and I wish I'd known it earlier: mutation testing. You introduce small deliberate faults into your code and measure how many your suite catches. The ones it doesn't catch are called surviving mutants, and every survivor is a region of your codebase where your tests execute but do not verify. PIT does this for Java, Stryker for JavaScript and TypeScript, mutmut for Python.
It is the only measurement I know of that answers the question coverage pretends to answer. Coverage asks "did this line run." Mutation testing asks "would anyone have noticed if this line were wrong." Only one of those is a test suite's actual job.
Fair warning: it's slow, because it runs your suite once per mutant. Don't point it at everything. Point it at the twenty files where a bug would cost you the most and let it run overnight. The first report will not be a pleasant read.
What I'd tell my earlier self
Ban the bare existence check in review. assertNotNull(x) on its own is not an assertion, it's a null guard that got promoted. If it's genuinely all you can check, write a comment explaining why, and let the awkwardness of writing that comment do its work.
Make missing assertAll() a build failure. This is a lint rule, not a culture change. It took an afternoon and it can never happen again.
Report assertion density, not test count. Assertions per test, and the count of tests with zero. Put it next to the pass rate. The moment that number is visible, the incentive flips, and the incentive is the whole problem.
Run mutation testing on your critical paths once a quarter. Not for the score. For the list of survivors, which is a to-do list written by a machine that cannot be talked into optimism.
The uncomfortable version
I'm aware this reads as a confession of incompetence. I don't think it is, and that's my actual argument.
Every one of those empty tests passed code review. Somebody looked at assertNotNull(order) and approved it, because it satisfied the thing we were all being measured on. The suite was doing exactly what the organisation asked it to do. It just wasn't doing what everybody assumed it was doing, and no tool in our stack was capable of surfacing the gap.
If you want to know where you stand, you don't need my thirty-line script. You need about fifteen minutes.
Go into your codebase. Change one comparison operator in a piece of business logic that matters. Run your suite.
If it stays green, you've learned something about the last two years of your work. I did.
Top comments (3)
One thing missing: test the database directly, not just the in-memory object.
Your discount example should verify:
When you changed >= to >, a real database assertion would have caught
it immediately. Testing only the Java object is testing that Java works,
not that your system works.
Database assertions are slower but honest—they show if your code
actually persisted anything.
I am always looking forward to your posts. God bless you for your impactful posts.
Precise as always