Introduction
Have you worked on a project with high test coverage? Ever wondered why, despite extensive coverage, bugs still manage to slip into production? Have you or your teammates begun questioning the effectiveness of unit testing? Have you questioned whether there’s a better way to check your test’s effectiveness?
The solution to all the questions we described here lies in mutation testing. This system allows you to check the real strength of your test suit. From there you can come up with new scenarios, check if a test is checking the proper behavior, and the potential is unlimited. In this blog post, I’ll show you how!
What is mutation testing?
Mutation tests are simple, they introduce faults in your code. It removes some lines, inverts conditions, changes return statements, etc. Each of these changes that it does is called a mutation. It’s like the sonar coverage, but for measuring test strength.
It will run your test suit and the goal here is that your tests fail. Good tests will fail because they are checking the correct behavior.
We can run mutation tests in our code with the use of PIT. According to PIT’s home page: ”PIT is a state of the art mutation testing system, providing gold standard test coverage for Java and the JVM. It's fast, scalable, and integrates with modern test and build tooling.”
PIT will generate a report of test strength, it’s similar to what Jacoco does with line coverage.
Let’s start implementing.
Implementing Mutation Testing with PIT: Step-by-Step Guide
Start by adding PIT on your pom.xml :
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.16.0</version>
</dependency>
Also to run PIT we need a Maven plugin to run the mutation tests:
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>LATEST</version>
</plugin>
This way PIT will mutate all your code. However, if you need PIT to run only in some target classes and tests, you will need to change the Maven plugin configuration:
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>LATEST</version>
<configuration>
<targetClasses>
<param>com.your.package.root.want.to.mutate*</param>
</targetClasses>
<targetTests>
<param>com.your.package.root*</param>
</targetTests>
</configuration>
</plugin>
If you are using JUnit 5 will not be supported right away. But don’t worry, there’s a plugin to make it compatible. The plugin configuration should look like this:
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.16.0</version>
<dependencies>
<dependency>
<groupId>org.pitest</groupId>
<artifactId>pitest-junit5-plugin</artifactId>
<version>1.2.1</version>
</dependency>
</dependencies>
<configuration>
<targetClasses>
<param>com.springmastery.mutation.*</param>
</targetClasses>
<targetTests>
<param>com.springmastery.mutation.*</param>
</targetTests>
</configuration>
</plugin>
Now you can run the mutation coverage goal:
mvn org.pitest:pitest-maven:mutationCoverage
Warning: this goal can take several minutes to execute if your classes and applications are big. To speed up this code analysis, look at PIT’s documentation and see how to use history to track what was already analyzed. So the next time you run it, PIT will only look at what has changed speeding up the process.
Mutation Testing in Action: Practical Example
Now that we know how to add PIT to our project, and how to set the maven goal. Let’s apply this password validator code:
package com.springmastery.mutation;
public class PasswordValidator {
public static boolean isValidPassword(String password) {
if (password == null || password.isBlank()) {
return false;
}
int minLength = 8;
String specialChars = "@#!_-";
return password.length() >= minLength &&
hasUpperCase(password) &&
hasSpecialChar(password, specialChars) &&
hasNumber(password) &&
!hasWhitespace(password);
}
private static boolean hasUpperCase(String password) {
return password.chars().anyMatch(Character::isUpperCase);
}
private static boolean hasSpecialChar(String password, String specialChars) {
return password.chars().anyMatch(c -> specialChars.contains(String.valueOf((char) c)));
}
private static boolean hasNumber(String password) {
return password.chars().anyMatch(Character::isDigit);
}
private static boolean hasWhitespace(String password) {
return password.chars().anyMatch(Character::isWhitespace);
}
}
And let’s say that I have these tests for that particular code:
public class PasswordValidatorTest {
@Test
public void testValidPassword() {
String password = "StrongPassword123!";
assertTrue(PasswordValidator.isValidPassword(password));
}
@Test
public void testNullPassword() {
assertFalse(PasswordValidator.isValidPassword(null));
}
@Test
public void testBlankPassword() {
assertFalse(PasswordValidator.isValidPassword(""));
}
@Test
public void testShortPassword() {
String password = "short123!";
assertFalse(PasswordValidator.isValidPassword(password));
}
@Test
public void testNoUpperCasePassword() {
String password = "lowercase123!";
assertFalse(PasswordValidator.isValidPassword(password));
}
@Test
public void testNoSpecialCharPassword() {
String password = "Password1234";
assertFalse(PasswordValidator.isValidPassword(password));
}
@Test
public void testNoNumberPassword() {
String password = "StrongPassword!";
assertFalse(PasswordValidator.isValidPassword(password));
}
@Test
public void testWhitespacePassword() {
String password = "Password 123!";
assertFalse(PasswordValidator.isValidPassword(password));
}
}
Let’s execute the mutation testing maven goal and check the report to see the results.
Just run: mvn org.pitest:pitest-maven:mutationCoverage
You can check the report in your target folder:
target>pit-reports>index.html
It will look like this:
It has only one class because the package contains only this PasswordValidator class. If your package has many classes in it, it’ll run for every class inside of it.
Looking at the report, we found that our test strength is good (95%) but indicates that we need to include some scenarios. Or that some of our tests aren’t checking the proper behavior.
We can open the class to understand what happened.
It seems that one scenario was overlooked during testing. We can identify it by hovering the mouse over the red line.
Or we can see the mutations report that will tell you the same.
Looking at the code we can assume that PIT changed the boundary condition >= to >. This means we forgot to test with values on the boundary, where the password size will be exactly 8.
So we add this scenario to check the boundary by adding a test that contains a password that is exactly 8 characters long:
@Test
public void testBoundaryCasePassword() {
String password = "Short!12";
assertTrue(PasswordValidator.isValidPassword(password));
}
Now we can execute again and check the report:
Great!! Now we achieved 100% of test strength. All mutations were killed.
As we saw, mutation tests helped us to catch some missing scenarios that we might forget and helped to improve our test suite strength.
Conclusion
In this article, you learned how to use and apply mutation testing in your code so you can find the real strength of your tests. It helps you to identify test scenarios that you might missed or some tests that are sloppy and are not checking the right behavior of your code.
Now it’s your turn, add PIT plugin to your project. Pick a package and test it out. Analyze the results. Does your code need some more scenarios? Do your tests are not catching all the mutations? Experiment, create a separate branch in your project, and test it out!
Share your experiences and discoveries with the community by commenting below.
Don’t forget to follow me on social media to be the first to read when my next article comes out!
Willian Moya (@WillianFMoya) / X (twitter.com)
Willian Ferreira Moya | LinkedIn
Follow me here in dev.to!
Top comments (0)