Green But Wrong: Mutation-Testing Agent-Generated Test Suites

The test suite passed in forty-two seconds, which made the production incident that followed especially embarrassing. Our coding agent had generated a set of unit tests alongside a refactor of the patient registration flow, and every assertion was green when the pull request merged. Two days later a compliance report flagged that a 120-year-old patient had been registered, violating a policy that should have rejected anyone at or above that age. The root cause traced back to a boundary condition the tests should have caught, and the worst part was that the tests were not missing; they were simply asserting the wrong boundary.

That shared assumption is the signature weakness of agent-generated tests, because the model encodes one mental model into both the code and its verification. A misunderstanding in the implementation gets faithfully reproduced in the test, producing a false green where the suite passes and the coverage report looks healthy. The bug sails through because the test asserts the buggy behavior as if it were the specification, and coverage percentages only measure which lines executed. They do not measure whether the assertions would notice a broken line.

Mutation testing closes that gap by asking a different question, one that measures how much code the tests can distinguish from a broken version of itself. The tool introduces small faults, called mutations, into the source and runs the suite against each one; if the tests pass with the mutation in place, the mutation survives, and that survival is a direct signal of a testing blind spot. A suite with a high mutation survival rate is not merely incomplete, it is actively misleading, because it reports green while broken code remains undetected. This makes mutation survival a more honest metric than coverage for evaluating agent-generated tests.

I ran this workflow on the registration validator using the open-source MonkeyCode project's current free tier, which includes 10 million tokens and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The setup took about twenty minutes and required no local dependencies beyond an SSH client, because the free server provided an isolated Linux environment where I could install the mutation tooling without touching my laptop's Python installation. The token allowance covered the entire session, including a conversation where I asked the model to explain each surviving mutation and classify it as a real gap or a false alarm.

The module under test was a small age validator that parses a string, rejects negative values, and rejects implausibly large ones:

# validator.py
def validate_age(age_str):
    age = int(age_str)
    if age < 0:
        raise ValueError("Age cannot be negative")
    if age > 120:
        raise ValueError("Age cannot exceed 120")
    return age

The agent generated three tests: one for a valid input, one for a negative input, and one for an input that exceeds the upper bound. The suite passed and coverage reported one hundred percent, which made the mutation results more interesting. The tool introduced a mutation that changed age < 0 to age <= 0, and the suite passed anyway, because no test exercised the boundary value of zero. Another mutation changed age > 120 to age >= 120, and again the suite stayed green, because the test for the upper bound used 121 rather than 120 itself. The survival rate for this tiny module was roughly thirty percent, and every surviving mutation traced to a boundary condition.

The fix was not a longer test file but a sharper testing strategy, and I added two boundary tests, one for zero and one for 120, which dropped the mutation survival rate to zero for this module. The interesting part is that the agent's original tests were not lazy; they were simply shaped by the same boundary blindness that shaped the implementation. The model had written age < 0 in the code and then written tests for -1 and 121, which are the values that make the comparisons true, not the values that probe the comparison itself. Mutation testing exposed that pattern in minutes, where a code review might have missed it entirely.

The same workflow scales to larger codebases with a few practical adjustments, and the first is computational cost, because mutation testing runs the full suite once per mutation. A large project can take hours on a modest machine, so the free server handled this small module comfortably but would struggle with a serious codebase. The free tier's token allowance is generous for interactive sessions, yet it is not a substitute for dedicated test infrastructure, and teams should treat it as a sandbox for experiments rather than a permanent pipeline. Mutation tools also produce false positives, where a surviving mutation reflects equivalent code rather than a real gap, so the results need human interpretation rather than blind automation.

The reusable lesson is that agent-generated tests deserve the same skepticism as agent-generated code, and mutation survival rate is a concrete metric for that skepticism. A green suite from an agent is a starting point, not a conclusion, and the boundary-blindness pattern I found here is common enough to warrant a quick mutation run on any critical module. If you want to reproduce this workflow, the MonkeyCode free tier is a reasonable sandbox for the experiment, though the technique itself is tool-agnostic and will serve you with any mutation framework you already use.

MonkeyCode provides free models that can run this workflow.