My AI-Written Test Suite Hit 94% Coverage. Then I Mutated the Code.

I have a small Python library that parses CSV files and extracts course schedules. It has no tests. I know, I know. So last week, I decided to see if a free model could write them for me.

I used MonkeyCode's free model access and free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The goal was simple: generate a test suite for my library, run it, and measure coverage. If the tests caught real bugs, great. If not, I wanted to know why.

The Setup

The library is small. It reads a CSV, skips header rows, and returns a list of dictionaries. There's a function for handling missing values and another for date parsing.

# schedule_parser.py
import csv
from datetime import datetime

def parse_schedule(path):
    with open(path) as f:
        reader = csv.DictReader(f)
        rows = []
        for row in reader:
            if row.get('Course') and row.get('Time'):
                row['Time'] = parse_time(row['Time'])
                rows.append(row)
        return rows

def parse_time(value):
    try:
        return datetime.strptime(value, '%H:%M').time()
    except ValueError:
        return None

I wrote a prompt that asked the model to generate pytest tests for this module. I gave it the source code and asked for edge cases.

PROMPT = """
Write pytest tests for this module. Cover normal cases, missing values, and invalid dates.
Return only the test code.
"""

The Generated Tests

The model produced 20 tests. They looked reasonable at first glance.

# test_schedule_parser.py
import pytest
from schedule_parser import parse_schedule, parse_time

def test_parse_time_valid():
    assert parse_time("09:30").strftime("%H:%M") == "09:30"

def test_parse_time_invalid():
    assert parse_time("25:00") is None

def test_parse_schedule_normal():
    rows = parse_schedule("sample.csv")
    assert len(rows) == 2
    assert rows[0]["Course"] == "CS101"
    assert rows[0]["Time"].strftime("%H:%M") == "10:00"

def test_parse_schedule_missing_time():
    rows = parse_schedule("missing.csv")
    assert len(rows) == 1
    assert rows[0]["Time"] is None

def test_parse_schedule_empty():
    rows = parse_schedule("empty.csv")
    assert rows == []

I ran them on the free server with pytest and coverage.py.

pip install pytest coverage
coverage run -m pytest test_schedule_parser.py
coverage report -m

The report said 94% coverage. Line 12, the row.get('Course') check, was the only miss. I felt good. Then I decided to mutate the code.

The Mutation Test

Mutation testing is simple: introduce a small change (a mutation) and see if the tests catch it. If they don't, the mutation survives, and you've found a blind spot.

I manually changed row.get('Course') to row.get('Course') or True. That's a mutation. The tests still passed. The coverage report still said 94%.

Why? Because none of the generated tests had a row without a 'Course' key. They tested missing 'Time' but not missing 'Course'. The model had seen the code and written tests for the obvious paths, but it didn't think about the case where a row has no 'Course' column at all.

I tried another mutation: parse_time returning datetime.now().time() instead of None on invalid input. The tests caught that one, because test_parse_time_invalid expected None. So the model did handle invalid dates.

But the first mutation survived. That's the lesson: coverage measures lines executed, not behaviors verified. The model generated tests that executed almost every line, but it didn't generate tests that would fail if the logic changed in a specific way.

The Blind Spot

Let me show you the exact mutation that slipped through.

# Original
def parse_schedule(path):
    with open(path) as f:
        reader = csv.DictReader(f)
        rows = []
        for row in reader:
            if row.get('Course') and row.get('Time'):  # line 12
                row['Time'] = parse_time(row['Time'])
                rows.append(row)
        return rows

# Mutated
def parse_schedule(path):
    with open(path) as f:
        reader = csv.DictReader(f)
        rows = []
        for row in reader:
            if (row.get('Course') or True) and row.get('Time'):  # mutation
                row['Time'] = parse_time(row['Time'])
                rows.append(row)
        return rows

With this mutation, any row with a missing 'Course' key would still be processed. The tests didn't catch it because every test fixture included a 'Course' column. The model assumed the key was always there — because the code used .get() but never tested the absence.

What I Learned

Three things stuck with me.

First, free model access is genuinely useful for bootstrapping a test suite. Twenty tests in seconds is better than zero tests. But they're a starting point, not a finish line.

Second, coverage is a proxy, not a guarantee. A high number can make you feel safe when you're not. Mutation testing is a better check, even if you do it manually like I did.

Third, the model's tests reflect the model's understanding of the code. It saw row.get('Course') and assumed it was always present. It didn't ask "what if this key is missing?" because the code didn't hint at that possibility. A human might have asked that question, but the model just followed the prompt.

Who Should Skip This

If you're writing tests for code that handles money, health data, or anything where a missed edge case has real consequences, don't rely on an AI-generated suite. Use property-based testing, mutation testing, and human review.

But if you're a student with a small project and zero tests, this is a great way to start. The free model and free server from MonkeyCode made it possible to iterate quickly. I just wish I had mutated the code before I trusted the coverage number.

Try it yourself: take a small module, ask a free model to write tests, then break the code in one small way and see if the tests catch it. You'll learn more from the mutation than from the coverage report.