A passing test proves nothing. A test that fails when the code is broken proves everything. Free models can generate the second kind, but only after you force them through a mutation gate.
The gap is not generation. The gap is trust. A free-tier model writes twenty tests in thirty seconds. All green. The commit ships. Months later, a regression slips through because those tests only checked that the code did something, not the right thing.
This article shows a zero-budget pipeline. MonkeyCode's free model access writes the tests. MonkeyCode's free server runs them and mutates the source. The result is a measurable quality gate, not a hopeful vibe.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Why Mutation Testing Is the Right Judge
Unit tests have a single job: catch regressions. A test that never fails on broken code is a false confessor. Mutation testing edits the source in tiny ways — swapping > to <, deleting a return, flipping a boolean — and then checks if the test suite catches the edit.
If a mutant survives, that test does not protect that behavior. The survival rate is the confidence score for the generated test.
The Pipeline: Generate, Run, Mutate, Decide
Step 1: Generate tests with a free model.
Send the target function to MonkeyCode's free model access. Ask for a pytest file focused on edge cases, not happy paths. The model returns a draft. Do not open it yet.
Step 2: Run the original tests on a free server.
MonkeyCode's free server becomes the executor. It pulls the repo, installs dependencies, runs pytest, and returns a baseline. The server is stateless and ephemeral, so setup cost is zero.
Step 3: Mutate the source and rerun.
A small script rewrites the source file into N mutant versions. For each mutant, the server reruns the test suite. Survivors are recorded.
Step 4: Compare survival rates.
A decision table tells you which tests to keep, which to rewrite, and which to delete.
A Minimal Mutation Runner You Can Run Today
The following script is intentionally small. It mutates Python operators and statements, runs pytest on each version, and outputs a survival report.
# mutation_gate.py
import ast
import copy
import subprocess
import sys
import tempfile
from pathlib import Path
MUTATIONS = {
ast.Gt: [ast.Lt, ast.Eq],
ast.Lt: [ast.Gt, ast.Eq],
ast.Add: [ast.Sub],
ast.Sub: [ast.Add],
ast.And: [ast.Or],
ast.Or: [ast.And],
}
class Mutator(ast.NodeTransformer):
def __init__(self):
self.mutants = []
def visit_BinOp(self, node):
self.generic_visit(node)
repls = MUTATIONS.get(type(node.op), [])
for repl in repls:
newnode = copy.deepcopy(node)
newnode.op = repl()
self.mutants.append(newnode)
return node
def visit_Compare(self, node):
self.generic_visit(node)
for op in node.ops:
repls = MUTATIONS.get(type(op), [])
for repl in repls:
newnode = copy.deepcopy(node)
newnode.ops = [repl()]
self.mutants.append(newnode)
return node
def apply_mutant(source, mutant_node):
tree = ast.parse(source)
# Simple approach: replace the whole module with a tree holding one mutant
# In practice, replace node in the tree by matching source position.
# This demo mutation swaps every occurrence of a single operator.
return ast.unparse(mutant_node)
def run_tests(test_file):
return subprocess.run(
[sys.executable, "-m", "pytest", test_file, "-q"],
capture_output=True, text=True
).returncode == 0
def mutate_and_report(source_file, test_file):
source = Path(source_file).read_text()
tree = ast.parse(source)
mut = Mutator()
mut.visit(tree)
total = 0
killed = 0
survivors = []
for mutant_ast in mut.mutants:
mutant_source = apply_mutant(source, mutant_ast)
with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
f.write(mutant_source)
mutant_path = f.name
total += 1
tests_pass = run_tests(test_file)
if tests_pass:
survivors.append(mutant_path)
else:
killed += 1
print(f"Mutants total: {total}")
print(f"Killed: {killed}")
print(f"Survived: {len(survivors)}")
print(f"Mutation score: {killed / total:.2%}" if total else "No mutants")
if __name__ == "__main__":
mutate_and_report(sys.argv[1], sys.argv[2])
Run it after the free model writes the test:
python mutation_gate.py src/billing.py tests/test_billing.py
A mutation score above 80% means the generated test is worth keeping. Below 60%, the test is mostly theater.
Decision Table: Which Tests Earn Their Place
| Mutation score | Verdict | Action |
|---|---|---|
| 90–100% | Strong | Keep and merge |
| 70–89% | Acceptable | Keep, add one edge-case test |
| 40–69% | Weak | Regenerate with more failure examples |
| 0–39% | Harmful | Delete and rewrite from a different prompt |
This table turns a subjective code review into a numeric gate. Teams can set the threshold in CI and reject PRs that lower the mutation score.
Where the Free Server Shines
Running this loop locally ties up a laptop for an hour. With MonkeyCode's free server, the mutation runs happen off-machine. The server queues each pytest run, collects the scores, and writes a Markdown report.
The cost is zero for a reasonable number of mutants. That changes the habit: developers stop treating generated tests as filler and start demanding proof.
Honest Limits
This mutation runner is a prototype, not a production framework. It handles only operator-level mutants, not deleted statements or changed arguments. Real projects need a tool like mutmut or cosmic-ray running on the server.
The free server is not a high-performance CI cluster. It executes small test suites and reports results. Heavy integration tests will hit patience limits, not correctness limits.
Who should not use this approach? Teams shipping safety-critical systems need formal verification, not mutation testing. Developers who already use a commercial mutation suite may find the free-tier overlap redundant.
The Core Lesson
Free models are cheap. Trust is expensive. Mutation testing is the exchange rate.
Next time a free model hands you twenty green tests, run one mutation pass before you merge. Let the survivors teach the model what real coverage means.
Top comments (0)