Who watches the watchmen? Mutation testing in action

Test effectiveness measurement. Credits for the title go to Yoni Goldberg. I love it because it gets the homonym and amazing graphic novel come to my mind.

So, you're developing a new function and it needs a lot of unit tests (like my Bitcoin address regex validation). Tell me: how could you judge the effectiveness of your tests? Are you sure that you're testing every feature with all the necessary corner cases? What makes you so confident?

I know the answer: Code Coverage and experience.

Two considerations:

• code coverage does not work for measuring the effectiveness of the tests. Code coverage helps you finding untested cases and it's pretty good at that. But it can't tell you anything about the quality of your code. You can leave a huge number of bugs into the source code and code coverage helps you only to pass through all of these bugs, not to solve them. You can argue that bugs must be discovered by tests but...
• testing requires experience, you do not write your best test right when you start doing it. The more experience you gain, the more doubts come to your mind asking yourself "Am I sure that these tests are really testing my function?"

Do not worry, it's a road already traveled. The more you gain experience, the more you know that your code is not perfect, that's why you started testing after all... And the more you test, the more you want to be sure that your tests are effective.

Quoting a colleague of mine (Hi Mirko 👋):

Sometimes I introduce mutations to be confident about my tests

Mutation Testing

Checking that your tests are effective is pretty straightforward: place some bugs into your code and launch the tests. If the tests do not fail, they are useless to catch the bug. This is mutation testing! You mutate the source code (this newly and bugged source code is called "a mutant") and you launch the test suite against it.

In fact, the main goals of Mutation Testing are:

• identify weakly tested pieces of code (those for which mutants are not killed)
• identify weak tests (those that never kill mutants)

[Mutation Testing

If you think that we're speaking about some testing freaks... Go ahead 😊

An example

I chose to play with Mutation Testing with an old package of mine: Typescript Is Type. The whole package is pretty small, below the source code

module.exports = {
  /**
   * Perform a runtime check about an instance type. It returns true if all the specified keys are
   * different from undefined
   * @param {*} instance The instance to be checked
   * @param {string|string[]} keys The keys that must be defined while checking the instance type
   */
  is(instance, keys) {
    if (instance === undefined || instance === null) {
      return false;
    }
    if(!Array.isArray(keys)) {
      keys = [keys];
    }

    for(let i = 0, n = keys.length; i < n; i++) {
      if(instance[keys[0]] === undefined) {
        return false;
      }
    }
    return true;
  }
}

Do you see the giant bug?

I installed Stryker that manages all the process of mutating my source code and launching the tests suite automatically.

Please, note: the Stryker installation is pretty straightforward, then you only need to add the files you want it to mutate through the Stryker configuration. In my case, it was just mutate: ["index.js"].

Once launched Styker with npx styker run I found a surprise, a mutant was survived

Stryker was telling me that if it mutated i < n with i <= n nothing changed from a testing point of view. The tests did not kill this mutant (the test passed both with i < n or the i <= n mutation). How could it be possible? How could the loop condition not influence on the tests? 🤔 Well, because of a giant bug! Take a look at the instance[keys[0]]! It did not use the i variable!!!

Ok, shit happens, let's analyze it:

• code coverage did not help to find the bug because, as I told at the beginning of the article, it can not check the quality of the code, it checks just that the tests pass through all the lines of code
• my tests were not so well, after all. Bugs can happen, both in the source code and in the tests

How I fixed it

The subject of mutation testing is not the source code but the tests themselves. And if my tests allowed a giant bug like the discovered one, I need to improve them.

I added some more tests and expectations that really test the missing feature (checking all the keys, not only the first one). Without changing the source code, I had now some failing tests, as expected! Then, I fixed the bug and the tests passed, perfect! I run Stryker again and everything is now fine

Long life to Mutation Testing 😊

Conclusions

The more you test, the more you need to know if your tests are effective. Stryker is not perfect yet (for example: you can not ignore some lines at the moment, like some various console.log) but it's a good tool to catch bugs, improve your testing confidence, and making you code better tested.

I discovered the Mutation Testing topic while following Yoni Goldberg and watching his Advanced Node.js testing: beyond unit & integration tests (2019) video, take a look at him contents 😉

Comments

[JoeSchr]

Hi, nice topic! Do you know of any way to do mutation testing combined with cypress?

[Stefano Magni]

Not at the moment but Mutation Testing has a problem: tests must be fast, really fast! Because they are going to be run hundreds of times, good for unit testing, not so good for UI tests...

[JoeSchr]

True, didn't think of that. Still would be cool for most important routes and components...