Build a Tiny Mutation Check Before You Trust Generated Tests

Last Thursday I had a green test suite and a broken function. That pairing is nastier than a red bar, because it feels like a receipt.

When you run the script in this post, I want this printout in your terminal. Not a vibe. A printout.

baseline: PASS (2 tests)
mutant always_one: SURVIVED
mutant always_zero: KILLED
mutant first_integer: SURVIVED
survivors: 2/3
verdict: suite is too weak to grade the function

Here is the only question I care about. If cartoon replacements of my code still keep the tests green, did I test the assignment — or did I test the model's favorite example?

I am Alex Chen, an AI/CS student in Halifax. I keep a folder of helpers that look finished at 1 a.m. This one started as a waitlist parser for a campus-club inbox. Incoming mail looks like You are #12 of 40 on the CSCI waitlist. I needed an integer rank. I did not need an agent framework. I needed to know whether the tests I had accepted could notice a lie.

You know that shoulder-drop when the runner prints two dots? That was me. Then a ruder question showed up. What if the function always returned 1?

Background

I had asked a free model to write tests for extract_rank. The reply looked adult: a happy-path assertion, a type check, a confident comment. I pasted it. Both tests passed. I almost committed.

The implementation I was protecting was a one-liner in disguise. If the string contained #, return 1. That is not parsing. That is superstition with a hash character.

Why did the suite stay green? Because the generated tests used the same toy sentence the model likes to invent: You are #1 of 10. Rank 1 is a terrible oracle. It agrees with a broken function, a constant, and a parser that grabs the first digit it sees.

This is the cheap-code problem in miniature. Tests are easy to emit. Checking whether they can fail is the part that still costs attention.

I drafted those first tests with MonkeyCode's free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I froze the weak suite as a fixture so you can reproduce the miss without an account, a key, or a moving target.

Goal

I wanted a lab I could rerun on a laptop. Baseline tests must pass. Then I replace the function with three mutants. If a mutant still passes, the suite did not pin down the behavior. A surviving mutant is not a score. It is a counterexample with a name.

Mutation testing, in the research sense, is a family of techniques that seed small faults into programs and ask whether the tests notice. This file is a classroom cousin, not a paper reproduction. Three hand-written mutants. No AST rewrite. No claim about industrial coverage.

Prerequisites are boring on purpose. Python 3.10 or newer. Standard library unittest only. Save one file. Do not pip-install anything for the frozen path.

Implementation

Put this in waitlist_mutants.py. Read it once before you run it. The tests call target, and target is a mailbox we can swap.

"""waitlist_mutants.py — classroom-scale mutation check, stdlib only."""
from __future__ import annotations

import unittest


def buggy_extract_rank(msg: str) -> int:
    """Looks parsed. Always returns 1 when a hash exists."""
    if "#" not in msg:
        raise ValueError("no rank marker")
    return 1


CURRENT = {"fn": buggy_extract_rank}


def target(msg: str) -> int:
    return CURRENT["fn"](msg)


class FrozenModelTests(unittest.TestCase):
    """Frozen from a typical generated suite. Not an official spec."""

    def test_hash_one(self):
        self.assertEqual(target("You are #1 of 10"), 1)

    def test_returns_int(self):
        self.assertIsInstance(target("You are #1 of 10"), int)


def always_one(msg: str) -> int:
    return 1


def always_zero(msg: str) -> int:
    return 0


def first_integer(msg: str) -> int:
    parts = "".join(ch if ch.isdigit() else " " for ch in msg).split()
    if not parts:
        raise ValueError("no integer")
    return int(parts[0])


MUTANTS = {
    "always_one": always_one,
    "always_zero": always_zero,
    "first_integer": first_integer,
}


def run_suite() -> tuple[bool, int]:
    loader = unittest.TestLoader()
    suite = loader.loadTestsFromTestCase(FrozenModelTests)
    result = unittest.TextTestRunner(verbosity=0, buffer=True).run(suite)
    failed = len(result.failures) + len(result.errors)
    return result.wasSuccessful(), result.testsRun if result.testsRun else failed


def main() -> None:
    CURRENT["fn"] = buggy_extract_rank
    ok, n = run_suite()
    print(f"baseline: {'PASS' if ok else 'FAIL'} ({n} tests)")
    if not ok:
        print("verdict: fix the baseline before mutating")
        return

    survivors = 0
    for name, fn in MUTANTS.items():
        CURRENT["fn"] = fn
        survived, _ = run_suite()
        if survived:
            survivors += 1
            print(f"mutant {name}: SURVIVED")
        else:
            print(f"mutant {name}: KILLED")

    CURRENT["fn"] = buggy_extract_rank
    total = len(MUTANTS)
    print(f"survivors: {survivors}/{total}")
    if survivors:
        print("verdict: suite is too weak to grade the function")
    else:
        print("verdict: these mutants died; try a nastier one")


if __name__ == "__main__":
    main()

Run it like this.

python3 --version
python3 waitlist_mutants.py

I wrote the file against Python 3.11. On 3.10+ you should see the same four status lines. If python3 is missing, try python.

Results

always_zero dies, and that is the trap. A suite that can kill a constant zero still looks strict. It is not. Both remaining mutants return 1 on the only sentence the tests bother to read.

always_one is the broken student function with a different name. first_integer is sneakier. On You are #1 of 10 the first digit cluster is 1, so the weak oracle nods. The model did not test rank twelve. It tested a slogan.

Want an error input? Feed the real function a line with no hash.

print(buggy_extract_rank("see you in the hallway"))

You should get ValueError: no rank marker. The frozen suite never sends that sentence, so a mutant that returns 1 on garbage can still look healthy. Silence is not coverage.

Now the diagnosis invite. Keep the expected rank at 12 and add one assertion in your head — do not paste yet. Which of these strings should kill first_integer?

You are #12 of 40 on the CSCI waitlist.
CSCI 2201: you are #12 of 40
Opened 2026, you are #12 of 40

The first string still yields 12 for first_integer, so that mutant survives even after you "upgrade" the example. The course code and the year push an earlier integer in front of the rank. Those two should fail assertEqual(..., 12). If you predicted the first line was enough, you just met the same optimism the model had.

A stronger test looks like this.

def test_rank_not_the_year_or_catalog(self):
    msg = "Opened 2026, CSCI 2201, you are #12 of 40"
    self.assertEqual(target(msg), 12)

After you drop that method into FrozenModelTests, rerun. always_one should die. first_integer should die. always_zero stays dead. That is the lesson hiding in the printout: one ugly fixture teaches more than three polite ones.

What I actually learned

Green means the suite is consistent with the function you plugged in. It does not mean the function matches the English in the assignment. A model will happily test its own example. If that example is rank 1, almost every lazy implementation bows.

I also learned to keep the generated tests in git as a suspect artifact, not as truth. The interesting object is the survivor list. If I cannot name a mutant that should die, I am not ready to trust the dots.

Common mistake one: mutating the tests instead of the function, then celebrating a red bar. Common mistake two: printing inside tests and calling that an oracle. Common mistake three: adding more happy paths that still start with #1. Volume is not malice. The year-prefixed string is malice, and you need some.

Limitations

This harness is tiny on purpose. Three mutants will not certify a parser. It will not catch a correct rank extracted with a brittle regex that explodes next semester. It will not replace a TA reading your code. Free model access is handy for drafting a weak suite you then attack, not for outsourcing judgment.

Who should not use this as a workflow? Anyone whose syllabus forbids model-written tests on graded work — follow the course rules, not this lab. Anyone shipping production parsers. Anyone who wants a leaderboard of "mutation scores" from three lambdas. That number would be fan fiction.

If you extend the lab, add one mutant that returns the last integer (40 in the waitlist line) and one fixture where the rank is zero. Predict the survivor list before you run. If your prediction is wrong, keep the fixture. That mismatch is the note.

I ran the longer generate-then-mutate loop on MonkeyCode's free server option so the fan on my laptop stayed out of the story. The part worth keeping is still the frozen file. If the survivor count is not zero, the chat transcript does not get a vote.