The Bug My Unit Tests Didn't Catch: An Infinite Loop in Mutation Testing

Who the hell writes a unit test for an infinite loop? Well, now that's me 😇

When I started working on gooze, my goal was simple: a mutation testing tool for Go that is easy to shard in CI, so a big project can split the work across many machines. Before gooze, I tried to add this to an existing Golang mutation testing tool: I refactored it in a private fork, and failed miserably. So I started from scratch.

If you are not familiar with the concept of mutation testing, it checks whether your tests have holes that bugs can slip through (what is mutation testing). It does this by adding small, controlled bugs to your code on purpose. It turns < into <=, || into &&, true into false, then runs your tests. If the tests still pass, that change "survived", and your tests missed something. This makes it a very powerful extension to your vibe coding harness, next to code coverage. Coverage tells you which lines your tests run. Mutation testing tells you whether your tests actually check what those lines do. When an AI agent writes both the code and the tests, that difference matters a lot.

And, like in a movie where everyone expected it except the main character (me), things started to go wrong. Like everyone who tries mutation testing, I ran into equivalent mutants: changes that look different but behave exactly the same, so no test can ever catch them. This is an old, well-known problem. Researchers have studied it for decades (a systematic review by Madeyski et al.), and the people who built mutation testing at Google ran into it too (Mutation Testing, Google Testing Blog).

I thought this was the hardest case. It was not. There was one more thing nobody told me about: unintended infinite loops. Sounds simple, right? The last time I hit one was at university.

Recently, I was working on those same equivalent mutants. In PR #92 I added a way to ignore single mutants, like only the != change on one line. Then CI started failing in a way I did not expect.

Only "timeout"

Some context first. gooze tests itself: on every pull request, CI runs gooze on gooze's own code. It mutates only the lines the pull request changed, and it has a strict rule. Every mutant on those lines must be killed, so the score must be 100%. If one change survives, the pull request can't be merged.

For each mutant, gooze applies the change, runs the tests, and records one result:

• killed: a test failed, so the tests caught the change. Good 👍
• survived: all tests passed, so the tests missed the change. Bad 👎
• timeout: the tests did not finish within the time limit. gooze stops them and counts the mutant as not killed. Also bad 😐
• not covered: no test runs that line at all, so the change can't be caught. gooze knows it without running the tests. Bad 🕳️
• skipped: the change was not tested. It counts as not killed. 🤷
• error: the changed code did not compile, so there was nothing to test. It is left out of the score. 🚧

The score is killed divided by everything except errors.

Here is the hero of the evening, the function the mutants changed. Line 201 of the file is the if inside the loop:

func replacementOf(original, mutated []byte) string {
    offset := firstDifference(original, mutated)

    fset := token.NewFileSet()
    file := fset.AddFile("", fset.Base(), len(mutated))

    var s scanner.Scanner

    s.Init(file, mutated, nil, 0)

    found := ""

    for {
        pos, tok, lit := s.Scan()
        if tok == token.EOF || file.Offset(pos) > offset {
            return found
        }

        found = lit
        if !tok.IsLiteral() {
            found = tok.String()
        }
    }
}

My pull request failed. No test failed. The only output was this report from gooze, and as with any timeout, the error message is not enough. It tells you that something did not finish, not why:

internal/domain/mutagen.go:201: timeout (comparison): if tok == token.EOF || file.Offset(pos) > offset {
    -> if tok < token.EOF || file.Offset(pos) > offset {
internal/domain/mutagen.go:201: timeout (logical): if tok == token.EOF || file.Offset(pos) > offset {
    -> if tok == token.EOF && file.Offset(pos) > offset {
internal/domain/ignore.go:277: timeout (comparison): ... line <= last ... -> ... line >= last ...
Mutation score: 98.58% (209 killed, 0 survived, 3 timeout, 1 error)
Error: mutation score 98.58% is below threshold.total 100%

How to read it: each result shows the file and line, the result, the kind of change in brackets, then the original code and, after the arrow, the changed code. The first one means: in mutagen.go on line 201, gooze changed tok == token.EOF to tok < token.EOF, and with that change the tests never finished.

So: zero survivors, but three timeouts. Three changes made the tests run until gooze stopped them. 209 of 212 is 98.58%, not 100%, so the build was red.

My first thought was that the CI machine was slow. That was wrong.

Step 1: Is it slow, or does it never end?

I applied one mutant by hand, changing the operator myself, and ran the tests with a time limit:

baseline:              ok   2.08 s
mutant  tok < EOF:     still running after 300 s (killed)
mutant  || -> &&:      still running after 300 s (killed)

A slow test finishes later. These never finished. So it is not about speed: it is a loop that never ends.

Step 2: Read the loop slowly

Look again at the loop in replacementOf above. It reads the code one token at a time, and it has two ways to stop:

1. tok == token.EOF: it reached the end of the file.
2. file.Offset(pos) > offset: it passed the position where the mutant changed the code.

Both mutants break the first way. tok < token.EOF is never true at the end of the file. With && instead of ||, the loop stops only when both checks are true at the same time.

For a change in the middle of the file, both mutants still end. With tok < token.EOF, the second check stops the loop as before. With &&, the loop runs to the end of the file and returns the wrong token, and a test catches that. But when the change is at the very last character, offset is the length of the file. The scanner reports EOF at exactly that position, not after it, so file.Offset(pos) > offset is never true. The scanner keeps returning EOF, and nothing stops the loop.

And one existing test did reach that edge. It compared a <= with a <, so the change was at the very end. With the original code it passed. With the mutant it did not fail. It hung. A test can only fail if it finishes, so gooze saw a timeout, not a killed mutant.

Step 3: Make the code testable

The test was there. The problem was that the code could hang, so the test could never fail. And the loop was hidden inside replacementOf, which first compares two whole files and then scans one of them, so the edge was hard to test directly.

So I did three things.

1. Moved the loop into its own small function, with a fixed number of steps. This is the code after the fix:

func replacementOf(original, mutated []byte) string {
    return tokenAt(mutated, firstDifference(original, mutated))
}

func tokenAt(src []byte, offset int) string {
    fset := token.NewFileSet()
    file := fset.AddFile("", fset.Base(), len(src))

    var s scanner.Scanner

    s.Init(file, src, nil, 0)

    found := ""

    for range len(src) + 1 {
        pos, tok, lit := s.Scan()
        if tok == token.EOF || file.Offset(pos) > offset {
            break
        }

        found = tokenText(tok, lit)
    }

    return found
}

func tokenText(tok token.Token, lit string) string {
    if tok.IsLiteral() {
        return lit
    }

    return tok.String()
}

replacementOf now only finds where the two files differ and asks tokenAt for the token there. tokenAt runs at most len(src) + 1 steps. Every token takes at least one byte, except EOF and the ; that Go adds at the end, so it never needs more. Even if someone breaks the condition again, the loop stops.

2. Tested the small function directly at the edge. Now it is one line in a table test:

{"inserted semicolon at the end", "a<b", 3, ";"},

The input is three characters, a<b, and the position is 3, the very end. Every byte is a token, and Go adds a ; at the end that takes no space, so this input needs every one of the len(src) + 1 steps. If someone shortens the limit by mistake, this test fails too.

3. Removed the other loop completely. The third timeout was a loop like for line := first; line <= last; line++. Changed to >=, it never ends for a one-line function, where first equals last. It now stores each function's first and last line and checks the range, so there is no loop left to break.

The result after the fix:

mutant  tok < EOF:   FAIL in under a second
mutant  || -> &&:    FAIL in under a second
Mutation score: 100.00% (221 killed, 0 survived, 1 error)

What about mocks?

I also tried another idea. I let the loop take its "read the next token" step as a function, so a test can pass its own. The test version counts the calls and stops the test on the first extra one:

calls := 0
next := func() (int, token.Token, string) {
    calls++
    if calls > 5 {
        t.Fatalf("scanned %d tokens of a 5-token input: the loop didn't stop at EOF", calls)
    }

    pos, tok, lit := s.Scan()

    return file.Offset(pos), tok, lit
}

It works: the broken loop failed in 0.006 s with a clear message. But it only protects the one test that uses it. Other tests that reach the same code through the real scanner still hung, until they were killed after 120 s. The step limit protects every caller, so I kept the step limit and did not keep this change.

What about benchmarks and fuzzing?

Forget mutation testing for a moment. Say this infinite loop was a real bug in my code. Would a benchmark or a fuzz test catch it? I tried both on the broken function, and the answer is no.

A benchmark hangs, it does not fail. A benchmark measures time. It does not check the result. With normal input, x := a <= b, the broken function ran fine: 552 ns per call. With the bad input, a<b and the position at the very end, the benchmark never finished. It printed nothing, not even a progress line. I set Go's own limit, -timeout 15s, and it did not stop it either. After 2 minutes I killed it from outside. And most benchmarks use normal, realistic input, so they would never reach this edge in the first place.

A fuzz test gets stuck and still says PASS. I ran go test -fuzz for 60 seconds with random inputs. After 307 inputs it got stuck: execs: 307 (0/sec), for the rest of the minute. No failure, no crash report, no saved input. At the end it printed PASS. Without -fuzz, plain go test runs only the seed inputs from f.Add(...), so it catches the bug only if a seed already has the change at the very end. Then it is just a unit test with a different name.

So both tools turn an infinite loop into the same thing my CI showed: a hang, and at best a timeout, with no clear failure and no logs. Both tools are good at what they do. But to catch an infinite loop, you need a test that hits the exact edge, and code that cannot hang.

What I learned

• Sometimes mutation testing finds a weak point, not a hole. My original code was correct. It never looped forever, and if I had not changed it, it would still work the same today. The tests were not missing anything either: one of them reached the edge. What mutation testing showed me was a weak point in testability. If someone breaks that condition in the future, by mistake or in a refactor, the tests will not fail. They will hang, CI will show only "timeout", and someone will spend an evening finding out why. Mutation testing makes that future bug happen today, while the code is still fresh in my head.

The last time I fought an infinite loop, I was at university, and the loop was the bug. This time the code was fine, and the loop was only a possibility. Mutation testing found it anyway. Now I have a test for it, too.

The full change is in gooze PR #92.