Ask an AI assistant to "write tests for this endpoint" and you will get tests. Lots of them. They compile, they pass, coverage goes up, and the PR looks great.
The problem is where those tests came from. If the model read the implementation and then wrote tests for it, the tests describe what the code does. They don't describe what the code is supposed to guarantee. Those are not the same thing, and the gap between them is exactly where security bugs live.
Tutorial tests vs. production tests
Tutorial tests answer "does this function return what I expect?"
Production tests, at least the ones that matter for security, answer a different question: "if someone breaks the thing that keeps us safe, will this suite notice?"
A test suite generated from the implementation is very good at the first question and surprisingly bad at the second. Here is how that plays out.
Failure mode 1: the test that mirrors the query
Illustrative example. Say you have a multi-tenant API and this handler:
def get_invoices(user, db):
return db.query(Invoice).filter(Invoice.tenant_id == user.tenant_id).all()
An assistant reads it and writes:
def test_get_invoices_returns_user_invoices(user, db):
seed_invoices(db, tenant_id=user.tenant_id, count=3)
assert len(get_invoices(user, db)) == 3
Green. Now delete the .filter(...) line. The test is still green, because the fixture only ever seeded one tenant. The test was built from the code's happy path, so it inherited the code's blind spot: it never imagined a second tenant existing.
Failure mode 2: asserting what came back, not what shouldn't have
Implementation-derived tests tend to assert on presence. "The response contains the user's name and email." Fine. But the security property is about absence: "the response does not contain ssn, password_hash or internal_notes for a caller without the right role."
If a refactor adds a field to the serializer, a presence-based test keeps passing while you start leaking data.
Failure mode 3: the happy path, three times
Ask for "thorough tests" and you often get the same valid request with slightly different values. What's missing is the hostile input: a missing token, an expired token, a token for a different tenant, an ID that belongs to someone else, an empty list, the maximum page size plus one.
Start from invariants instead
Before generating anything, write down the properties that must hold no matter how the code is implemented. For example:
- A user cannot read another tenant's records.
- An unauthorized request cannot change protected state.
- Sensitive fields are not returned to callers who lack permission.
These are written without looking at the code. That's the point. They come from the threat model and the product requirements, not from the function body, so they can't quietly copy its assumptions.
Now the AI is genuinely useful: give it the invariant and ask it to generate cases across normal, invalid and boundary inputs.
@pytest.mark.parametrize("caller", [
"same_tenant_user",
"other_tenant_user",
"anonymous",
"expired_token",
"admin_of_other_tenant",
])
def test_invoices_never_cross_tenants(caller, two_tenant_db, make_client):
client = make_client(caller)
resp = client.get("/invoices")
returned = {inv["tenant_id"] for inv in resp.json().get("items", [])}
assert returned <= {client.tenant_id}
The fixture deliberately seeds two tenants. The assertion is about what must never appear. Delete the filter and this test fails for same_tenant_user immediately.
Prove the suite can actually fail
This is the step most teams skip. A passing suite tells you nothing until you know it would fail for the bug you care about.
Mutation testing automates that check. Tools like mutmut (Python), Stryker (JS/TS, C#) and PIT (Java) make small changes to your code, such as flipping a comparison or removing a condition, and rerun the tests. If a mutant survives, no test noticed the change.
You don't need to run it on the whole codebase. Point it at your authorization checks, tenant filters and serializers. A surviving mutant in check_permission() is worth more attention than a hundred in your logging helpers.
The reframe
AI didn't create this problem. Tests derived from the implementation have always had it. AI just makes it cheap to produce a lot of them, which makes the gap easier to miss.
A green test suite is only useful if it would turn red for the failure you care about.
How are you deciding what your AI-generated tests should check? Do you write the properties first, or review what the model produced after the fact?
Top comments (0)