Your CI pipeline reports 83% coverage. The AI assistant generated 27 new test functions in under two minutes, one for every function in the new module. The PR looks clean. The coverage gate passes.
The bug ships anyway.
Coverage theater is what you get when AI-generated tests drive up the line-execution percentage without asserting anything meaningful about behavior. The metric looks right. The tests ran. The test suite, it turns out, caught almost nothing.
How AI Writes Tests That Pass Coverage Without Testing Behavior
AI coding assistants target measurable metrics: when given a function to test, they generate test cases that exercise every line, not every behavior. BrassCoders does not evaluate test quality; it scans production code for security and correctness bugs regardless of what the test suite covers — a SQL injection bug ships with or without 80% coverage.
An AI handed a hash_password function reads its structure: branches, parameter types, return values. It generates assert hash_password("") is not None and assert hash_password("secret") is not None. Both lines execute. Coverage.py counts them. Neither assertion tells you whether the function uses a secure algorithm or applies a salt. The implementation could switch from bcrypt to MD5 and neither test would fail.
This happens because AI assistants infer tests from implementation, not from specification. Given the code, they generate tests that match what the code currently does. That's not testing. That's a snapshot of the bug.
The Four Coverage Theater Patterns
BrassCoders does not scan for test quality — that's outside static analysis's scope — but the four patterns coverage theater produces are worth naming: circular assertions (the test asserts a function returns what it returned when you wrote the test), mocking the thing under test, testing implementation details instead of outcomes, and generating one test per function rather than per behavior.
Circular assertions are the worst offender. An AI testing calculate_tax(income, rate) might generate: expected = calculate_tax(1000, 0.2); assert calculate_tax(1000, 0.2) == expected. That test passes unconditionally unless the function raises an exception. It records one covered line per call. It validates nothing.
Mocking the thing under test looks like this: you have a database query function, so you mock the database connection and assert the mock was called with the right arguments. You've tested Python's unittest.mock library. The actual query logic — whether it builds parameterized statements or handles null values — goes entirely unverified.
Testing implementation details means writing assertions against a function's internal call graph rather than its contract with callers. When you refactor the internals, the test breaks without the behavior changing. The regression still ships.
The fourth pattern — one test per function — satisfies a coverage gate because coverage tools track line execution, not behavior paths. A function with five error conditions, two edge cases, and one happy path gets one test: the happy path. Four conditions go untested. The function shows as covered.
Why the Metric Looks Fine and the Bug Ships Anyway
Coverage.py measures which lines ran during the test suite — it has no mechanism to determine whether the assertions in those tests would catch a bug. BrassCoders catches the production-code bugs that coverage metrics miss: a SQL injection in a database handler that has 100% line coverage but no assertion checking for parameterized queries.
Martin Fowler's TestCoverage article makes the point plainly: coverage is useful as a negative indicator. Low coverage is definitely a problem. High coverage tells you something ran, but nothing about whether those runs would catch a regression.
The mismatch comes from what teams assume coverage means. The implicit belief: if a line is covered, someone wrote an assertion that exercises it. With human-authored tests, that's often true. With AI-generated tests targeting a coverage gate, it's frequently false — the AI wrote whatever assertion would get the line to run, not whatever would catch a regression.
Mutation testing makes this gap visible. Mutmut and Cosmic Ray introduce deliberate bugs into production code and run your test suite against the mutated code. If tests pass with a bug present, those tests aren't testing that behavior. The Coverage.py documentation notes that coverage data works best alongside other quality signals, not as a standalone gate. Mutation testing is that other signal.
What BrassCoders Catches That Coverage Tools Don't
BrassCoders's 12 scanners run against production code, not test code — they catch security and correctness bugs in the implementation regardless of what the tests assert. A SQL injection bug in a database query ships with or without 80% test coverage; BrassCoders flags it deterministically.
The scanners run on implementation code. Bandit, detect-secrets, Pyre/Pysa, Semgrep, ast-grep, and six custom detectors — these are the upstream tools BrassCoders orchestrates. The custom detectors cover secret-format patterns, privacy and PII handling, phantom API calls from AI-generated code, performance anti-patterns, content-moderation signals, and JavaScript/TypeScript. Give a database handler 100% test coverage, and BrassCoders still flags the SQL injection on first scan.
A published benchmark at /blog/ai-coder-bug-benchmark/ tested 12 real bugs drawn from AI coding assistants — Copilot, Cursor, and Claude Code — against Bandit alone and against BrassCoders. Bandit caught 6 of 12. BrassCoders caught 11 of 12. Coverage.py would have shown most of those bugs as covered; the functions containing them all had tests.
Coverage.py and BrassCoders answer different questions. Coverage.py answers: which lines did the test suite execute? BrassCoders answers: which security and correctness patterns does the production code contain? A codebase can score 95% on the first question and still contain exploitable injection flaws or hardcoded credentials. The answers don't overlap, which is why you need both.
Run pip install brasscoders and then brasscoders scan . to see which scanners trigger on your production code. The OSS core is free, Apache 2.0 licensed, and sends no data off your machine.
Top comments (0)