Does Your AI-Generated Test Suite Actually Fail? Use Mutation Testing to Find Out

The CI was green. Coverage was at 91%. The team felt great — until a production bug slipped through a function that supposedly had 100% line coverage. The tests were written by an AI assistant, and they were a lie: they asserted the output but never broke the implementation.

That team is not an outlier. As coding agents generate more code, they also generate more tests, and those tests often share the same blind spots as the code. Coverage percentages measure how much code ran, not whether anyone checked that the code does the right thing. For a team evaluating a free model for test generation, coverage is the wrong success metric. Mutation score is the right one.

This post shows how to run a mutation-testing gate on AI-generated unit tests, using a free model to generate the tests and a free server to execute the batch. You get a reproducible workflow, a cost estimate, and a decision table for when this approach makes sense.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Mutation testing in five minutes

Mutation testing deliberately breaks your source code — changing > to <, removing a return, swapping + to - — and then runs your test suite. Each broken version is a mutant. If the tests fail, the mutant is killed. If the tests still pass, the mutant survived, and you found a blind spot.

The score is simple:

mutation score = killed mutants / (total mutants - equivalent mutants)

A score above 80% means your tests actually catch behavior changes. Below 50% means you have a warm blanket of assertions that verify nothing.

Popular tools: mutmut for Python, Stryker for JavaScript/TypeScript, PIT for Java. All can run inside a CI job.

Why free models change the calculation

Generating tests with an LLM is cheap in tokens but expensive in verification time. A decent unit test is 30–80 lines, roughly 400–1,000 tokens. Ten files of generated tests might burn 10–20K tokens per run. That fits comfortably inside a free model tier — MonkeyCode's free models cover this scale without a credit card, and the associated free server can run the mutation batch without touching your laptops or your paid CI minutes.

Here is the workflow that turns a free model into a meaningful quality gate:

1. Generate tests for a module with an LLM prompt that includes the function signature and a few examples.
2. Run the generated tests against the current code. They should pass.
3. Run the mutation tool. Compute the score.
4. If the score is below threshold, expand the prompt with edge cases the mutants exposed.
5. Re-run until the score crosses your bar.

This is not a benchmark from a blog post. It is a gate you can run in your own repository with your own bugs.

A reproducible Python example

Assume you have a small module pricing.py:

def apply_discount(price: float, discount: float) -> float:
    if discount < 0 or discount > 1:
        raise ValueError("discount must be between 0 and 1")
    return price * (1 - discount)

A naive AI-generated test might look like this:

def test_apply_discount():
    assert apply_discount(100, 0.1) == 90
    assert apply_discount(50, 1) == 0

That test executes both lines. Coverage is 100%. But raise a ValueError or test a discount of exactly 0, and the test suite stays green because the assertions never exercise the guard clause.

Run mutmut on it:

pip install mutmut
mutmut run --paths-to-mutate pricing.py
mutmut results

You will see mutants like discount < 0 changed to discount <= 0, and the tests will still pass. The mutation score will drop well below 60%, exposing the gap.

Then you iterate the prompt:

Write unit tests for apply_discount. Include:
- a discount below 0 (expect ValueError)
- a discount above 1 (expect ValueError)
- boundary values 0 and 1
- a non-numeric input if type hints allow it

Regenerate, re-run mutmut, and the score climbs.

Budgeting tokens for this workflow

Use this formula per module:

Tokens per iteration ≈ (base prompt 800) + (source lines × 15) + (generated test lines × 5)

For a 200-line module, one iteration costs roughly 800 + 3000 + 800 = ~4.6K tokens. Three iterations per module, 8 modules: about 110K tokens. That is well inside the free tier of MonkeyCode's free models, and the free server handles the compute without a GPU on your side.

A word of caution: the free server is best for bursty batch jobs, not for continuously hot endpoints. If you wire this into every commit, the queue may grow. Keep it on a nightly schedule or a manual trigger.

Decision table: should you adopt this gate?

Team situation	Run mutation gate on AI tests?	Why
1–5 devs, early-stage project	Yes, on the critical module only	Cheap, catches the worst blind spots
20+ devs, monorepo, many AI-generated PRs	Yes, as a required CI check	Scales quality without human review time
Tests are purely integration/E2E	No, mutation adds little	Integration tests already exercise behavior end-to-end
Security-sensitive code	Yes, with a strict 90% threshold	Mutation score is a proxy for adversarial thinking
No tests exist at all	No, write a few baseline tests first	Mutation needs a baseline to break

The gate is a conversation tool, not a number to worship. It fails on untested legacy code, but that failure is information: you now know where your safety net is thin.

Limitations and who should not use this

Free models have smaller context windows and may truncate long test files. Keep each module prompt under 2K tokens of source. The free server has rate limits, so a burst of 50 modules in one minute will likely fail; batch them in chunks of five.

Teams with strict data-residency requirements should not send proprietary business logic to a hosted free model. For them, a self-hosted small model is the better route, even if the mutation gate still applies. And if your team has zero tolerance for false positives in test generation, a human-written suite with a high mutation score is still the gold standard.

Run it once before you judge the model

Do not trust a vendor's benchmark for test generation. Clone a small service, ask the free model to write tests for one module, and run the mutation score yourself. That thirty-minute run tells you more than any leaderboard.

MonkeyCode's free models and free server give you the resources to run that experiment today. The cost is zero; the evidence is yours.

Set the gate on one module. Watch the score. Then decide whether the free model stays in your test pipeline.