DEV Community

Jordan Li
Jordan Li

Posted on

Mutation Testing on a Free Server: Do AI-Generated Tests Actually Kill Bugs?

#python #testing #devops #ai

A test suite passed with 94% coverage. The bug still shipped. Coverage numbers lied. The missing piece was mutation testing. It measures whether tests can detect injected faults. A free hosted model and a disposable server made the experiment cheap.

MonkeyCode is an open-source coding platform. Its free tier includes hosted model access and a disposable server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. This post uses both to answer one question: can AI-generated tests kill real bugs?

Why Coverage Is Not Enough

Coverage counts executed lines. It does not check assertions. A test can run a line and ignore the result. Mutation testing changes that. It modifies source code slightly. Each modification is a mutant. A good test kills the mutant. A weak test lets it survive.

The Experiment Design

The probe used a small Python library. It had a known bug. The task was to generate tests for the library using the free model. Then run mutation testing on the free server.

  1. Pick a target module with a subtle bug.
  2. Send the module source to the free model. Ask for pytest tests.
  3. Save the generated tests to a file.
  4. Install mutmut on the free server.
  5. Run mutmut run --paths-to-mutate=target.py.
  6. Collect the mutation score and surviving mutants.

The command sequence is short.

pip install mutmut pytest
mutmut run --paths-to-mutate=target.py
mutmut results
Enter fullscreen mode Exit fullscreen mode

The Generated Tests

The model produced 12 tests. They covered all functions. The first run killed 68% of mutants. That was better than expected. It was still far from the 94% coverage number.

The surviving mutants clustered in two areas. Edge cases with empty inputs. Boundary conditions with negative numbers. The model had not been asked to think about them.

Where the Tests Failed

Three failure patterns appeared.

  1. Happy-path bias. Tests checked normal inputs. They ignored empty strings and zero values.
  2. Assertion weakness. Some tests called a function but never checked the return value.
  3. Duplicate logic. The model copied the implementation into the test. A bug in the implementation was replicated in the test. The mutant survived because the test expected the same wrong output.

The third pattern was the most dangerous. It is called the "equivalent mutation trap". The model learned the buggy behavior and encoded it as truth.

The Fix: Mutation-Guided Prompting

The second iteration used the mutation report as feedback. The surviving mutants were sent back to the model. The prompt asked for tests that specifically kill those mutants.

# mutation_feedback.py — pseudocode
survivors = read_survivors("mutmut_results.txt")
prompt = f"""
The following mutants survived. Write tests that fail for each.
{survivors}
"""
Enter fullscreen mode Exit fullscreen mode

The second run killed 91% of mutants. The remaining 9% were mostly equivalent mutations. Those are harmless. The improvement came from targeted feedback.

The Cost

The free server ran the full mutation suite in 14 minutes. The model consumed 340,000 tokens across two iterations. The entire experiment fit inside the free tier. No local GPU was needed.

Limitations

Mutation testing is slow. Large codebases take hours. The free server is best-effort. It may throttle long runs. The model's tests are only as good as the prompt. Without mutation feedback, they stay shallow.

Who Should Use This

Teams with critical business logic should try this. Open-source maintainers can use it to vet contributions. Anyone writing tests for a legacy module can benefit. Teams with strict data rules should not send source code to a hosted tier.

MonkeyCode's free tier is open for testing. Check the repository for current limits. Run the probe on a small module first. The mutation score will show the truth behind your coverage number.

Top comments (0)