Six tests that passed without testing what they claimed

I recently built a multi-tenant booking API (FastAPI, SQLAlchemy, Postgres, Redis, and a background worker) with Claude Code, as the running example for a book. Writing good tests was an explicit goal from the first commit. The suite grew to 97 tests against real Postgres and Redis, with no mocked database.

Six of those tests passed without testing what their names claimed. None was careless. Each was plausible, read well, and went green. A green run can't tell a test that checks the behaviour from one that checks nothing, and that's as true when you write the test as when an agent does.

Here they are, in the order we found them.

1. "Expired tokens are rejected"

The test built a token with an exp in the past and asserted a 401. It passed.

It signed the token with a hard-coded secret, and the app used a different one from .env. The token was rejected for its signature, not its expiry. With expiry checking switched off, the test still passed.

The fix is a positive control. Next to the negative assertion, take the same token, change only the property under test, and assert it's accepted:

expired = jwt.encode(claims, settings.jwt_secret, algorithm="HS256")
assert post_with(expired).status_code == 401

still_valid = jwt.encode({**claims, "exp": now + timedelta(minutes=5)},
                         settings.jwt_secret, algorithm="HS256")
assert post_with(still_valid).status_code == 201  # same token, future exp: accepted

If the control fails, the negative test was never testing expiry.

2. "A 2,070-character webhook URL returns 422, not 500"

Contract fuzzing had found a real 500: URLs between 2,049 and 2,083 characters passed the URL validator and then overflowed a 2,048-character column. The regression test used a 2,560-character URL. It got a 422 because the URL validator's own length limit rejected it, so the database was never reached.

The fix: the test asserts its precondition (assert 2048 < len(url) < 2083), so it can't drift out of the window that matters.

3. "Login fails closed when Redis is down"

Login checks two rate limits: per IP and per email. Flipping the per-IP check to fail open left the test green, because the per-email check still failed closed.

The fix: one planted bug per guard, and a test path that exercises each guard alone. Registration only has the per-IP check, and it had no Redis-down test at all.

4. "Two owners removing each other can't leave zero owners"

Two threads, two requests, one assertion. It passed 10 out of 10 runs with the protective lock removed. The threads never actually interleaved.

The fix: force the interleaving. Open a second database connection, lock the rows the request needs, start the request, then commit:

with engine.connect() as other:
    other.execute(text("SELECT 1 FROM memberships WHERE org_id = :o "
                       "AND role = 'owner' FOR UPDATE"), {"o": org_id})
    other.execute(text("DELETE FROM memberships WHERE org_id = :o AND user_id = :u"),
                  {"o": org_id, "u": second_owner})
    pending = pool.submit(remove_first_owner)   # blocks on the lock
    time.sleep(0.5)
    other.commit()
assert pending.result() == 409

With the lock in place, it passes 3 out of 3. Without it, it fails 3 out of 3. That's a race test.

5. "A booking of exactly 12 hours is allowed"

A Hypothesis property test generated random durations between −24 and +24 hours and checked the rule 0 < duration <= 12h. Moving the boundary by one microsecond (<= to <) didn't fail it. Random durations practically never land on exactly 12:00:00.

The fix: pin the boundaries with @example(duration=MAX_BOOKING) and its neighbours. Property tests are great at the edges of the input space you give them, and blind to the edges inside it.

6. No test fetched a single booking

This one isn't a lying test but a missing one. While practising a refactor, we broke GET /bookings/{id}, and the whole suite stayed green. No test had ever fetched one booking successfully.

The one habit that found all six

Every one was found the same way: break the code on purpose and check that a test goes red.

We turned that into a script. It holds a list of realistic bugs (a missing tenant filter, expiry not enforced, the overlap constraint neutralized, a limiter that fails open, and so on). It applies each bug in turn, runs the tests expected to catch it, and fails if any bug survives. There are 22 now, and the script runs in CI. A pull request that deletes or weakens the test for a protection fails the build, even when every remaining test passes.

It isn't a general mutation-testing tool. It's a hand-picked list of the bugs this particular service must never ship. Because each bug runs only the test files expected to catch it, it's fast enough for every pull request: about a minute and a half for all 22 on a laptop.

What this means when an agent writes your tests

AI-written tests fail in exactly these ways, only faster and in greater volume. Three instructions change the outcome more than any amount of "be thorough":

1. "Write the test first and show me it failing, and why." The failure message should name the behaviour, not a setup error or a 404 for a route that doesn't exist yet.
2. "Every negative test needs a positive control."
3. "For each new protection, add a planted bug and show it's caught."

Then read which test went red. A planted bug caught by an unrelated test is a warning, not a success.

---

This is adapted from Chapter 9 of my book, Claude Code for Backend Developers. The book comes with the runnable service these tests are from: https://azadhmhd.gumroad.com/l/claude-code-backend