Opinion: Spend Free AI Compute on Killing Your Tests, Not Writing Code

We argue about whether an AI badge measures anything, but we rarely ask whether our test suite can measure a fault at all. Free model access changes the economics of verification more than it changes the economics of code generation. The marginal cost of producing a patch is near zero, so the bottleneck moves downstream to the tests that must reject bad patches. Mutation testing is the cheapest way to find out whether those tests can fail at all, and a free server is exactly where that arithmetic should run.

The argument

Most review gates for AI-generated code assume the test suite is trustworthy, and that assumption is almost never measured. Running a suite once per injected fault multiplies CI minutes by the number of mutants, which is why mutation testing is routinely skipped on paid runners. A free server changes the decision from "skip it" to "run it nightly," and the output is a concrete list of test gaps instead of another diff to read.

Feeding known-bad patches to your review gate tests one decision at a time. Mutation testing is the systematic version of that idea, because it enumerates the fault classes your suite can see without depending on a reviewer's imagination. Generated tests are not a substitute, because they tend to mirror the assumptions in the code they test, and mutants exist precisely to break those assumptions.

This is not about the cost per verified change; it is about whether the verification step can detect a change at all. If your suite cannot fail on a flipped operator, it cannot be trusted to judge an AI patch either.

The artifact

Here is a minimal, runnable mutation tester that flips one comparison operator per mutant and runs pytest against each one. It requires Python 3.9 or newer because it uses ast.unparse, and it deliberately mutates only the first operator of each comparison to stay readable. The script restores the original file after every run, so a failed mutant never leaks into your working tree.

#!/usr/bin/env python3
"""mutate.py — flip one comparison operator per mutant and run pytest.

Usage:
    python mutate.py target.py test_target.py
"""

import ast
import copy
import subprocess
import sys
from pathlib import Path

FLIPS = {
    ast.Lt: ast.Gt,
    ast.Gt: ast.Lt,
    ast.LtE: ast.GtE,
    ast.GtE: ast.LtE,
    ast.Eq: ast.NotEq,
    ast.NotEq: ast.Eq,
}


def mutant_sites(tree):
    for node in ast.walk(tree):
        if isinstance(node, ast.Compare) and node.ops and type(node.ops[0]) in FLIPS:
            yield node


def main() -> int:
    target_path, test_path = Path(sys.argv[1]), Path(sys.argv[2])
    original = target_path.read_text()
    base = ast.parse(original)

    total = killed = 0
    for site in mutant_sites(base):
        old = type(site.ops[0])
        tree = copy.deepcopy(base)
        for candidate in ast.walk(tree):
            if (isinstance(candidate, ast.Compare)
                    and candidate.lineno == site.lineno
                    and candidate.col_offset == site.col_offset):
                candidate.ops[0] = FLIPS[old]()
                break

        target_path.write_text(ast.unparse(tree))
        try:
            result = subprocess.run(
                [sys.executable, "-m", "pytest", str(test_path), "-q"],
                capture_output=True,
                text=True,
            )
            total += 1
            if result.returncode != 0:
                killed += 1
                print(f"KILLED   line {site.lineno}: {old.__name__} flipped")
            else:
                print(f"SURVIVED line {site.lineno}: {old.__name__} flipped")
        finally:
            target_path.write_text(original)

    if total == 0:
        print("No comparable operators found; nothing to mutate.")
        return 1

    print(f"\n{total} mutants, {killed} killed, score {killed / total:.0%}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The workflow

1. Save the script as mutate.py in the same directory as your target module and its tests.
2. Run it once locally to confirm the output format: python mutate.py pricing.py test_pricing.py
3. Read the survivor list; each survivor is either an equivalent mutant or a genuine test gap.
4. Schedule the same job on MonkeyCode's free server option so it runs nightly without consuming paid CI minutes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access is useful for the classification step below, because classifying a survivor is a small, verifiable task.
5. For each survivor, ask the free model to classify it with a short prompt, then confirm the reasoning by reading the code yourself.

A worked example

Consider a shipping-cost function with two boundary comparisons.

def shipping_cost(weight_kg: float, express: bool) -> float:
    if weight_kg <= 1.0:
        base = 4.0
    elif weight_kg <= 5.0:
        base = 7.0
    else:
        base = 10.0
    return base * 1.5 if express else base

And a test file that covers only the obvious cases.

from pricing import shipping_cost


def test_light_standard():
    assert shipping_cost(0.5, False) == 4.0


def test_heavy_express():
    assert shipping_cost(6.0, True) == 15.0

Run the mutator and you get this report.

KILLED   line 2: LtE flipped
SURVIVED line 3: LtE flipped

2 mutants, 1 killed, score 50%

The first mutant dies because test_light_standard pins the 1.0 kg boundary. The second mutant survives because both tests use weights on the same side of the 5.0 kg boundary, so the flipped condition returns identical results for every input in the suite. That survivor is a missing test case, not an equivalent mutant, and the free model should say so when you feed it this prompt.

Original code:
    elif weight_kg <= 5.0:
        base = 7.0

Mutated code:
    elif weight_kg > 5.0:
        base = 7.0

Result: pytest passed, so the mutant survived.
Question: is this an equivalent mutant or a missing test case?
Answer in one sentence, and propose a test if the mutant is meaningful.

The verification is simple: shipping_cost(5.0, False) returns 7.0 in the original and 10.0 in the mutant, so the boundary is real. Add the missing test, rerun the mutator, and the score moves to 100%.

def test_boundary_five_kg():
    assert shipping_cost(5.0, False) == 7.0

Reading the survivor table

Use the table as a triage guide rather than a verdict. A survivor under <= or >= almost always means the exact boundary value is missing from your inputs, while a survivor under == or != usually means one side of the equality path is never reached. The pattern across many survivors matters more than any single line.

Survivor pattern	Likely meaning	Action
Boundary operator (<=, >=) survives	The exact boundary value is never tested	Add a test at the boundary
Equality (==) survives	The equal path is never exercised	Add a test where values match
Inequality (!=) survives	The unequal path is never exercised	Add a test where values differ
All mutants in a class are killed	The suite is sensitive to that fault class	Extend mutation to other operator classes

Limitations

Mutation testing has three honest limitations. Equivalent mutants produce false alarms that still need human classification, because a flipped operator can leave observable behavior unchanged. Operator-flip mutation cannot find missing logic, since you cannot mutate code that does not exist in the first place. And a high kill rate measures sensitivity rather than correctness, so a suite can kill every mutant and still miss an integration failure.

Flaky tests make the results noisy, so run each mutant more than once if your suite is not stable. Production tools such as mutmut for Python, Stryker for JavaScript, and PIT for Java enumerate the full mutation matrix, while this script stops at the first operator of each comparison. Treat the score as a trend line across commits, not a pass/fail gate, because a single nightly run on a small module is a sample, not a census. Who should not use this approach: teams without a test suite at all, and teams whose tests are mostly end-to-end UI flows, because unit-level mutants will die for reasons unrelated to the code under test.

The takeaway

The cheapest way to know whether your review gate can fail is to make it fail on purpose. Run the mutator once, classify the survivors, and you will know exactly which tests to write before the next AI patch arrives. That is a better use of free model access than another generated function, and the free server is the right place to do that arithmetic every night.