A test suite can execute 90% of your code and verify none of it. Coverage counts the lines a test runs. It says nothing about whether the test would notice if one of those lines returned the wrong answer.
That gap is where AI-generated tests live. Ask an assistant for tests and it produces something that runs, imports cleanly, and lights up the coverage report. Whether any of it would catch a real bug is a separate question, and the research says the answer is often no.
Coverage Measures Execution, Not Verification
BrassCoders treats a coverage percentage as an execution count, not a quality signal. Coverage records which lines a test ran; it says nothing about whether the test would fail if those lines returned the wrong answer.
The coverage.py documentation is explicit that the tool monitors which parts of the code have been executed by tests, and identifies code that could have been executed but was not. It makes no claim about whether executed code was checked for correct behavior.
The mutation-testing project PIT puts the distinction in one sentence. Its documentation states that traditional coverage measures only which code is executed by your tests, and does not check that your tests are actually able to detect faults in that code. A test that calls parse_date(input) and asserts nothing executes every line inside parse_date. The coverage report credits all of them. The test would still pass if parse_date returned the wrong date, the wrong type, or None.
The Peer-Reviewed Verdict on Coverage
BrassCoders leans on the largest study of the question to date. Inozemtseva and Holmes generated 31,000 test suites across five systems totaling 724,000 lines of code, and found only a low-to-moderate correlation between coverage and a suite's fault-detection power once the number of test cases is held constant.
Their ICSE 2014 paper states the takeaway directly: coverage is useful for finding under-tested code, but it should not be used as a quality target because it is not a good indicator of test suite effectiveness. Stronger coverage criteria didn't rescue the signal either; branch coverage was no better a predictor than statement coverage.
A companion result points at what does predict fault detection. Zhang and Mesbah composed 6,700 test suites from 24,000 assertions across five real-world Java projects and found, in a paper deliberately titled Assertions Are Strongly Correlated with Test Suite Effectiveness, that the number of assertions strongly correlates with a suite's effectiveness. The pair of findings reads cleanly together. Lines executed is a weak signal. Assertions made is a strong one.
What Happens When AI Writes the Tests
BrassCoders treats AI-generated tests as coverage-rich and assertion-poor by default. A 2023 study of ChatGPT-generated unit tests found only 24.8% ran without execution errors, and 17.3% compiled but failed on the model's own incorrect assertions.
Those numbers come from an empirical evaluation of ChatGPT for unit-test generation, which generated 1,000 tests and reported that 42.1% compiled while just under a quarter executed cleanly. The assertions were the recurring problem: the model wrote checks that looked plausible and encoded the wrong expected value. A test with a wrong assertion is worse than no test, because it turns green and tells you the behavior is confirmed.
Coverage figures for AI tests swing wildly with the codebase. Siddiq and colleagues, in an empirical study of LLM-generated JUnit tests, measured a Codex model above 80% coverage on the HumanEval benchmark and under 2% on the real-world SF110 corpus. They also catalogued the tests' contents and found named test smells, Empty Tests and Duplicated Asserts among them, patterns that raise coverage while checking nothing.
The generators are genuinely good at the coverage number. TestPilot, evaluated on 25 npm packages in a 2024 IEEE Transactions on Software Engineering study, reached a median 70.2% statement coverage. Hitting the number is the thing they do well, which is exactly why the number flatters them.
How To Measure Whether Tests Actually Verify
BrassCoders points teams to mutation testing as the check coverage can't perform. A mutation tool seeds small faults into the code, a flipped comparison or a changed constant, then runs the suite and reports how many faults the tests caught. A test that asserts nothing catches none of them, no matter how much code it executes.
Mutation score is a validated proxy for real-bug detection, not a lab curiosity. Just and colleagues, in Are Mutants a Valid Substitute for Real Faults in Software Testing?, studied 357 real faults across 321,000 lines of code and found a statistically significant correlation between mutant detection and real-fault detection, independent of code coverage. PIT is the reference tool on the JVM; Python projects have mutmut and Cosmic Ray. Run one against an AI-written test file and the assertion-free tests reveal themselves immediately, because they kill almost nothing.
Where BrassCoders Fits
BrassCoders does not run your tests or grade them; it scans the code the tests are supposed to protect. A static analysis pass flags a real SQL injection or an unsafe deserialization call whether or not the test suite covers that line, so a green, high-coverage suite full of hollow assertions can't hide a finding BrassCoders would report.
That makes static analysis an independent signal from the test suite, and independence is the point. Coverage and mutation score measure how good your tests are. A BrassCoders scan measures the code itself, reading it for the bug patterns AI assistants tend to introduce, and it reaches the same verdict on a file with zero tests as on one reporting 100% coverage. The full evidence set behind this post, with every source, is in BrassCoders's AI test quality research index. BrassCoders emits the pattern matches as YAML; the AI assistant reading that file decides which ones matter in context, the same division of labor a scan applies to every finding.
None of this means you should stop asking an assistant for tests. It means the coverage number it produces isn't the thing to trust. Measure the tests with mutation testing, and scan the code with a tool that doesn't care what the tests claim.
BrassCoders scans your code for the bugs a hollow test suite misses, as one of 12 scanners on every commit: pip install brasscoders.
Top comments (0)