Originally published on AI Tech Connect.
Ask a coding agent to add tests to a module and it will do it quickly and in a form that looks entirely convincing. The suite runs green. The coverage number goes up. Somebody screenshots the percentage for the sprint review. The question none of that answers is the only one that matters: if the code under test were wrong, would any of these tests have noticed? That is not a rhetorical flourish. Researchers studying LLM-generated tests on the HumanEval-Java benchmark recorded a case where the generated tests achieved 100% line and branch coverage yet scored only 4% on mutation testing. The suite executed effectively every line and branch while being blind to almost every fault that could be injected, missing corner cases such as leap-year date handling. Perfect coverage. Near-zero fault-…
Top comments (0)