Score LLM-Written Tests with Mutation Testing on a Free Server

A test that never fails is not a test, and LLM-generated tests often pass because they assert nothing. I use mutation testing on a free server to score those tests: small source changes should make a good suite fail, and the mutation score tells me whether the generated tests are worth keeping.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with a hosted free tier. I call a free model through its OpenAI-compatible endpoint on a free server so the generate-and-score loop stays cheap. The current grant includes 10 million tokens—enough for thousands of test-generation calls—but quantity does not guarantee quality. A single weak test wastes the whole run. Mutation testing is the filter I apply before I keep anything.

Why mutation testing beats line coverage for LLM tests

Code coverage is a poor proxy. A test can cover every line and still miss a bug. Mutation testing checks behavior, not lines. It measures whether the tests would fail if the implementation changed. That is the property I actually care about when a free model writes the assertions.

I keep three outcomes in mind:

• Score 0: the tests are useless. They pass, but they assert nothing that would break under a small change.
• Partial score: some mutants die, some survive. I inspect the survivor and tighten the prompt.
• Full score on my mutant set: the tests detect every change I applied. That is a strong keep signal, not a proof of correctness.

Coverage asks whether a line ran. Mutation testing asks whether the test would fail if that line meant something else. For LLM-written tests, the second question catches empty asserts, tautologies, and examples that only exercise the happy path. I still run the original tests first. If pytest fails on the unmodified function, I stop. Broken tests are noise, not a quality signal.

For the technique itself I point people at Wikipedia’s mutation testing overview. For a real operator catalog beyond naive string replacements I look at mutpy. I execute generated suites with pytest.

The generate, run, and mutate pipeline on free servers

Here is the pipeline I run against free servers and free models:

1. Send a function to the free server. Ask a free model to write pytest tests and return only the test code.
2. Run those tests against the original function.
3. Mutate the function and re-run. Count how many mutations the tests catch. That count is the mutation score.

Free LLM tiers make this verification loop affordable. I iterate on the prompt until the score improves without burning a paid quota on every retry. I use free models for generation and a free server for the hosted endpoint. The pipeline is endpoint-agnostic: any OpenAI-compatible base URL works if I pass a key and a model name.

Ten million tokens is enough for thousands of test-generation calls. I still treat each call as untrusted output. The free model is a generator, not a reviewer. The mutation score is the reviewer.

This loop is for developers who already generate tests with an LLM and need a cheap filter before committing them. Teams that treat coverage percentages as proof should not use this as a substitute. Mutation testing is a heuristic. It does not guarantee correctness. It only tells me that the tests can detect certain changes.

A working Python script for a free model

The script below implements the loop. It uses the OpenAI-compatible endpoint that MonkeyCode exposes. I provide a function file, a base URL, a key, and a model. I keep temperature at 0.2 so the free model stays closer to deterministic test code, and I cap max_tokens so a rambling reply cannot swamp the pytest file.

# mutation_check.py
import argparse
import subprocess
import tempfile
import os
from openai import OpenAI

PROMPT = """Write pytest tests for the function below.
Return only the test code.

python
{code}

"""

def generate_tests(client, model, code):
    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": PROMPT.format(code=code)}],
        temperature=0.2,
        max_tokens=500,
    )
    return resp.choices[0].message.content

def run_pytest(code, tests):
    with tempfile.NamedTemporaryFile('w', suffix='.py', delete=False) as f:
        f.write(code + '\n\n' + tests)
        path = f.name
    try:
        result = subprocess.run(['pytest', '-q', path], capture_output=True, text=True, timeout=30)
        return result.returncode == 0
    except subprocess.TimeoutExpired:
        return False
    finally:
        os.unlink(path)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--code', required=True, help='Path to the Python function file')
    parser.add_argument('--base-url', required=True)
    parser.add_argument('--key', required=True)
    parser.add_argument('--model', required=True)
    args = parser.parse_args()

    original = open(args.code).read()
    client = OpenAI(base_url=args.base_url, api_key=args.key)

    tests = generate_tests(client, args.model, original)
    print('--- generated tests ---')
    print(tests)

    original_pass = run_pytest(original, tests)
    print('original tests pass:', original_pass)
    if not original_pass:
        print('stop: generated tests are broken')
        return

    mutations = [
        ('==', '!='),
        ('% 2', '% 3'),
        ('return n % 2 == 0', 'return n % 2 != 0'),
    ]
    killed = 0
    for old, new in mutations:
        mutated = original.replace(old, new)
        if mutated == original:
            continue
        caught = not run_pytest(mutated, tests)
        killed += int(caught)
        print(f'mutation {old!r} -> {new!r}: caught={caught}')

    print(f'mutation score: {killed}/{len(mutations)}')

if __name__ == '__main__':
    main()

shell

I run it like this:

python mutation_check.py --code is_even.py --base-url https://free-server.monkeycode.ai/v1 --key $KEY --model your-model

A sample function to feed the script:

# is_even.py
def is_even(n):
    return n % 2 == 0

The generated tests might look like this:

def test_even():
    assert is_even(2) is True

def test_odd():
    assert is_even(3) is False

These tests pass on the original. That is necessary and not sufficient. The next section is where I decide whether to keep them.

How I read the mutation score and improve the prompt

I apply the first mutation: change == to !=. The mutated function returns True for odd numbers. test_odd now fails. That mutation is killed. The second mutation changes % 2 to % 3. The mutated function returns True for multiples of 3. Both tests may still pass if they only check 2 and 3. That mutation survives. The third mutation flips the evenness predicate in the return statement. If the tests only encode two literals, the score often lands at 1/3.

I treat that number as a prompt-debug signal, not a grade for the function:

• A score of 0 means I should not commit the tests. They pass, but they assert nothing that mutation can see.
• A score of 3 on this tiny set means the tests detect every change I applied. I still do not call that correctness; I call it a keep candidate.
• A score of 1 or 2 means partial coverage. I look at which mutation survived and add a case that would have failed—another odd non-multiple of 3, a negative number, or a non-integer if the function should reject it.

Then I change the prompt, not the production code. Typical edits that raise the score in this loop:

• Ask for tests that cover boundary values, not only 2 and 3.
• Forbid tautologies such as assert is_even(2) == is_even(2).
• Require the model to return only pytest code, which this script already does, so pytest is not parsing markdown fences as source.

Mutation testing has limits. The mutations here are naive string replacements. Real tools like mutpy generate hundreds of operators. The principle still holds: if my tests cannot kill even these three edits, they will not catch a real regression either. LLM-generated tests may also contain non-deterministic assertions. Flaky tests ruin the score. I run each candidate a few times before I trust the result.

The free tier changes over time. The 10 million token grant is today’s offer. I verify the current quota before I depend on it. The pipeline itself does not care which free server I point at, as long as the API is OpenAI-compatible.

Start with a small function. Run the script against a free model on a free server. Read the mutation score. Improve the prompt. Repeat. That loop turns free tokens into a measurable quality signal—then commit only the suites that actually fail when the code lies.