Most of us have seen a green build and felt a small rush of relief. Then someone finds a bug in production, and the relief turns into a quiet question: what did all those passing tests actually tell us?
Here is a hypothetical, so you can picture it. A team has 900 tests and a dashboard that glows green. A pricing bug ships anyway. Every test passed, because none of them checked the one rule that was broken. The tests weren't lying. They just weren't saying what everyone assumed they were saying.
That gap, between what a test says and what we think it says, is what this article is about.
A test is evidence, not proof
A passing test tells you one narrow thing: for this specific input, in this specific environment, the code produced an output that matches what the test author expected.
That's it. It does not tell you the feature works. It does not tell you the code is correct. It tells you a particular check did not fail.
A useful way to think about it: a test is a small experiment. A good experiment can fail for a meaningful reason. If it can't fail, or can only fail for meaningless reasons, it tells you very little.
So the real question for any test is not "does it pass?" but "what would have to be broken for this to fail?"
Three questions to ask of any test
Before trusting a test, I like to ask:
- What behavior does it protect? Can you name it in one sentence?
- What bug would make it fail? If you can't imagine one, it's decoration.
- Would a failure point somewhere useful? Or would you spend an hour guessing?
Let's apply these to some examples.
Example 1: The test that checks the mock
Some terminology first. A mock is a stand-in object that replaces a real dependency (like a database or an API) so a test can run quickly and predictably.
Mocks are useful. They're also an easy way to write a test that proves nothing.
def test_get_discount_calls_service(mocker):
service = mocker.Mock()
service.fetch_rate.return_value = 0.2
result = get_discount(service, user_id=1)
service.fetch_rate.assert_called_once_with(1)
This test verifies that get_discount called fetch_rate. It says nothing about whether the discount is correct. If someone changes the math from price * rate to price + rate, this test still passes.
Compare it to this:
def test_get_discount_applies_rate_to_price(mocker):
service = mocker.Mock()
service.fetch_rate.return_value = 0.2
result = get_discount(service, user_id=1, price=100)
assert result == 20
Same mock, but now the test checks an outcome a user would care about. Break the math and it fails, and the failure message points at the right place.
The lesson: assert on results, not on the internal steps used to get them, unless the step itself is the requirement.
Example 2: The test that can't fail
def test_parse_returns_something():
result = parse("a=1")
assert result is not None
What bug would make this fail? Only a function that returns nothing at all. A parser that returns the wrong dictionary passes happily.
Tighten it:
def test_parse_reads_key_value_pair():
assert parse("a=1") == {"a": "1"}
Now it has a point of view. It also documents the behavior for the next person who reads it.
Example 3: Coverage that looks like confidence
Code coverage is the percentage of lines (or branches) that run while your tests execute. It is a useful signal, but it measures what ran, not what was checked.
A test can execute every line of a function and assert nothing:
def test_checkout_runs():
checkout(cart=sample_cart())
Coverage goes up. Confidence should not. If this test has no assertion, it only fails when the code crashes.
Coverage is better used in the opposite direction: low coverage tells you where nobody is looking. High coverage doesn't tell you anyone is looking carefully.
A practical check: mutation thinking
You can test your tests. The formal technique is called mutation testing: a tool makes small deliberate changes to your code (flip a > to >=, change + to -) and re-runs your tests. If the tests still pass, the change "survived," which means your tests didn't notice a real behavior change.
Tools exist for this in many languages, such as mutmut for Python and Stryker for JavaScript and others. Check current docs for your stack, since setup details change.
You don't need the tool to get the benefit. Try it by hand on a critical function:
- Pick a function with business rules.
- Change one condition or operator.
- Run the tests.
- If everything is still green, you've found a gap.
Five minutes of this on a pricing or permissions function is often more revealing than a coverage report.
What about AI-generated tests?
AI assistants can write tests quickly, which is great for getting a baseline. They also make this problem easier to scale, because generated tests tend to confirm what the code currently does.
That's the catch. If the code has a bug, a test generated from that code may assert the bug as correct behavior. It's a test that agrees with the implementation instead of the requirement.
A few habits help:
- Give the AI the requirement, not just the code. Paste the acceptance criteria or the rule in plain language.
- Ask for edge cases explicitly: empty input, boundary values, invalid types, time zones, rounding.
- Review assertions first. Skim what each test claims, before looking at setup.
- Run the manual mutation check on anything important.
Treat AI-generated tests as a draft from a fast junior colleague. Useful, worth reviewing, not authoritative.
Signs a test suite is telling you the truth
Healthy suites share some traits:
- Failures are specific. The test name and message tell you what broke.
-
Tests read like rules.
test_expired_coupon_is_rejectedbeatstest_coupon_2. - Few tests are flaky. A flaky test passes and fails without code changes. Once people expect random failures, they stop believing real ones.
- Risky areas get more attention. Payments, permissions, and data migrations deserve deeper checks than a settings page.
- Different layers have different jobs. Unit tests check logic quickly. Integration tests check that parts work together. A few end-to-end tests check critical user journeys.
A short checklist for your next pull request
When you review or write tests, try this:
- Can I state the protected behavior in one sentence?
- If I break the logic slightly, does a test fail?
- Does the assertion check an outcome, not just a call?
- Are boundaries covered (zero, one, max, empty)?
- Would the failure message help someone at 5 p.m. on a Friday?
If most answers are yes, the test is probably earning its place.
For engineering leaders and QA managers
Dashboards of pass rates and coverage are easy to report and easy to misread. Consider adding questions to your reviews:
- Which critical rules have a test that would fail if they broke?
- What escaped to production recently, and which test should have caught it?
- How much time goes to investigating flaky failures?
The escaped-defect question is especially useful. Each one is a free lesson about what your suite does not currently say.
The takeaway
A green build means "the checks we wrote did not fail." Whether that's reassuring depends entirely on what those checks were looking for.
So the habit worth building is small: for every test, ask what bug would make it fail. If the answer is "none I can think of," rewrite it or delete it.
What's one test in your codebase you trust the most, and what makes it trustworthy? I'd love to hear how you decide.
Top comments (0)