Measure the Quality of Your Tests with Mutation Testing

Unit testing is an integral part of the current development process. Especially with the introduction of Continuous Delivery and Continuous Deployment methods as well as the business need of delivering as quickly as possible, we need a way to deliver software quickly with the best possible quality.

• Less time performing integration tests
• Protection against regression
• Executable documentation
• Less coupled code

Since starting implementing unit tests, one question that first rises is how can we measure the quality of our unit tests. So actually it’s more of a philosophical question.

If tests are the guardians, then who guards the guardians?

In other words, what metrics should I use to measure the quality of my tests and ideally how can I do it in an automated way?

Mutation Testing

The most traditional way to do something like that is Code Coverage. This is the most commonly used method and its main purpose is to find how many lines and branches of our code our unit tests have executed. And here starts the problem.

It does not check that your tests are actually able to detect faults in the executed code. It is therefore only able to identify code that is definitely not tested.

For example, tests that are assertless (meaning that they only perform actions in SUT but no assertion, so they are not actual tests) can not be identified with code coverage tools. These are extreme cases (although some would disagree with that statement) but still there are other issues that might exist.

Let’s take a look at the examples below:

Here are some unit tests that will check it:

In the above example although we check that calls to collaborator classes are made we do NOT check the return type. However code coverage tools will show 100% coverage because tests pass from the code but they do not actually test it! Even if it seems a straight forward example, in more complex cases, it could create issues.

So, mutation testing comes to the rescue. It’s not a modern technique, it’s actually older than JUnit, but it was mainly into academia. Now it has started gaining popularity in the industry with some appropriate changes that have been done.

Mutation testing is a simple technique to evaluate the quality of your unit tests. It is part of failure injection testing. Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lives.

PITest

PITest is the most well known mutation testing tool for Java based applications. The main reasons to choose it are:

• PIT is fast — can analyse in minutes what would take earlier systems days
• PIT is easy to use — works with ant, maven, gradle and others
• PIT is actively developed
• PIT is actively supported

Usage

We should add PITest maven plugin in our pom.xml like this:

In excludedClasses we add the classes that we don’t want to mutate. Also in excludedTestClasses we exclude the test that we don’t want to run (e.g. functional tests).

In order to run mutation coverage, run mvn org.pitest:pitest maven:mutationCoverage

Stryker

In JS and C# world the most famous tool is Stryker.

• Stryker is easy to use
• Stryker is actively developed
• Stryker is actively supported

Concluding automated tests are a great tool that can help our development team to deliver faster and with higher quality. But these tests should also be treated in the same way as production code and have clear quantitative metrics that show us their quality. Mutation testing techniques are the best ones we currently have.