The Green Suite Was the Distraction: 48 Hours With a Free Model Writing Unit Tests

What happens when you hand a free model a legacy Python function and ask it to write unit tests? You'd expect instant coverage, a few silly case names, and a pass on the first run. I ran that experiment for 48 hours using MonkeyCode's free model access and a free server to execute the tests on a schedule. The suite stayed green almost the whole time. That was the problem.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why test generation matters

Unit tests only earn their keep when they catch regressions. Line coverage tells you which statements were executed, but it tells you nothing about whether the assertions would notice if someone deleted the if guard that handles edge cases. I wanted to see if a free model could write tests that fail when the code breaks, not just tests that pass because the model echoes the implementation back to itself.

The setup

I chose a small module with a painfully typical legacy function: an email validator that doesn't use regex and has a suspiciously low length check.

def validate_email(email):
    if not isinstance(email, str):
        return False, "not a string"
    if "@" not in email:
        return False, "missing @"
    if len(email) < 6:
        return False, "too short"
    return True, "ok"

The harness pulled that function, generated tests with the free model, wrote them to a test file, and ran pytest --cov on the free server every four hours. Because I also wanted to measure test quality, I ran a mutation score check after each generated suite: flip a condition, delete a line, and see if the tests notice.

# Pseudocode, not an actual API call
for hour in range(0, 48, 4):
    prompt = build_prompt("test_validate_email", legacy_source)
    generated_tests = request_free_model(prompt)
    write_file("test_validate_email.py", generated_tests)
    coverage_output = run_on_free_server("pytest --cov=legacy --cov-report=xml")
    mutation_score = run_on_free_server("mutmut run --paths-to-mutate validate_email.py")
    record(hour, coverage_output, mutation_score)

I stored the generated files and reports in a git repo. The free server handled the cron scheduling, but honestly a laptop could have done the same thing. The interesting part was never the infrastructure.

Day 1: The false confidence trap

The first run looked amazing. It produced test_valid_email_returns_true, test_invalid_email_returns_false, test_missing_at_symbol, and the suite passed. Line coverage was 92%. I started drafting a blog post about how AI test generation had finally arrived.

Then I manually changed one line: len(email) < 6 became len(email) < 5. The generated tests didn't care. The model's test for a short string used 'a@b', which is four characters—still under five. But it never tested a five-character string like 'a@c.d' because the prompt didn't tell it to. The mutation survived, and so did my suspicions.

That pattern repeated all day. The model was very good at writing tests that mirrored the happy path and obvious failures. It was terrible at exploring the boundaries between states.

Day 2: The model started writing its own rules

By the second day, the old test files were still sitting in the working directory. The free model picked up that context and began imitating its previous output—same test names, same variable naming, even a test that referenced validate_email.__doc__, which doesn't exist. The suite stayed green because the model was copying the shape of its own past tests.

At one point it invented a test for a helper function that never existed in the module. The import failed, and the server logged a red suite for two consecutive runs. The moment I removed the orphaned file, the model went back to producing green tests that were equally useless.

This was the most instructive failure: the model wasn't reasoning about my code; it was reasoning about the text in its context window. The context window included my legacy function, but also its own previous output. That self-referential drift made every new suite more verbose and less meaningful.

The artifact: a decision table for generated tests

After 48 hours, I grouped every generated test by what it actually asserted. A test that calls validate_email('a@b') and checks for (False, "too short") is really just copying the implementation's current behavior. A test that checks validate_email(None) is probing a type-level edge case. That distinction matters.

Assertion type	Count	Mutations killed
Happy path	34	0
Boundary value	7	2
Error type	51	1
Behavior copying	118	0

Line coverage at the end was 94%, but the mutation score never went above 35%. The green bar lied every four hours, and the only thing that caught it was mutation testing.

What I'd repeat

I'd absolutely use a free model as a test-writing co-pilot, but not as the pilot. The trick is to give the model a strict checklist: edge cases, boundary values, type errors, and a ban on asserting message strings the function doesn't emit. Then treat every generated test as a suggestion that needs a human stamp.

I'd also run mutation testing on every generated suite from the start. The free server was fine for that, and the mutation score became a discipline layer that line coverage never gave me.

Limitations

This was one function, one model, one 48-hour window, and no network flakiness. Your model, your prompt, and your code will change the balance between happy-path tests and meaningful tests. The approach is especially risky for payment processing, crypto, or any domain where a missing negative test can become a lawsuit.

Who should skip this

If you already use property-based testing with Hypothesis or contract tests, you don't need a model generating assertion soup. The model will happily amplify your blind spots. Also skip this if you can't review the generated tests yourself—the free model is a mirror, and your experience is what makes the reflection useful.

Closing

The suite stayed green for forty-eight hours, and that was exactly the distraction. The real signal came from mutation testing: most generated tests were just the model singing the code back to itself. Next time I'll put a mutation score in the cron job before the tests even run. Maybe the free model can write that too, but I'll review every line.