Use Coverage as a guide, not a goal

Introduction

If you are a developer, you probably know what code coverage is. It is a metric that tells you how much of your code is executed by your tests. It is often used as a way to measure the quality of your tests and your code.

But is coverage a reliable indicator of quality? Does it guarantee that your tests are effective and that your code is bug-free? The answer is no.

In this article, I will explain why coverage is not enough to ensure the quality of your tests and your code, how coverage can be misleading and give you a false sense of security, and how to use coverage as a guide to identifying what to test.

What some developers think coverage is good for:

Code coverage is a metric that shows how much of your code is exercised by tests and it’s often used by tools like Sonar as a quality gate.

Maybe you are confident that you can do a refactoring in your code. Because of the high coverage, you think there’s no chance you going to miss a possible bug, and assuming that high coverage means low risk.

However, this is a big mistake!

Code coverage doesn’t guarantee good test quality or good code quality. It is only one aspect of testing, and it should not be used as the sole criterion for evaluating your tests.

How coverage can hide problems that show up in production:

Coverage can be misleading. You can make your tests exercise the code without testing anything. This will make you miss some bugs that can cause you trouble at a later time.

When you write tests and the assertions are not meaningful enough, and they are too generic, or even worse you don’t have any assertions. You are just invoking the method that is going to be tested, and not checking the behavior and results. The test coverage will not check the quality of your test suite. It will only measure if you covered all the code.

Let’s see this example:

So you have this Calculator class:

public class Calculator {

    // A method that adds two numbers
    public int add(int a, int b) {
        return a + b;
    }

    // A method that subtracts two numbers
    public int subtract(int a, int b) {
        return a - b;
    }

    // A method that multiplies two numbers
    public int multiply(int a, int b) {
        return a * b;
    }

    // A method that divides two numbers
    public int divide(int a, int b) {
        if (b == 0) {
            throw new ArithmeticException("Cannot divide by zero");
        }
        return a / b;
    }
}

Writing tests to have 100% coverage in this code can be something like this:

@Test
    public void testCalculator() {
        Calculator calculator = new Calculator();

        calculator.add(2, 3);

        calculator.subtract(5, 2);

        calculator.multiply(3, 4);

        calculator.divide(10, 2);

        try {
            calculator.divide(5, 0);
            fail("Expected an ArithmeticException to be thrown");
        } catch (ArithmeticException e) {
            // Do nothing
        }
    }

The result:

This simple test gives the calculator class 100% of coverage! However, it's not checking anything. It only calls the methods with valid parameters. So the coverage will be high because you are running all the class code. But it won’t detect anything if someone adds a bug in the code. This bug can break the business rules.

If you change de add method for something like this:

  public int add(int a, int b) {
        return a - b;
    }

Your test will not detect any problem, and you still will have 100% line coverage:

A bug was introduced, and your tests are not catching it!

Another example would be tests that have no meaningful assertions. They are too generic not checking if your code is behaving properly. For example:

    @Test
    public void testCalculator() {
        Calculator calculator = new Calculator();

        int sum = calculator.add(2, 3);
        // The test asserts that the sum is not null, which is always true for int
        assertNotNull(sum);

        int difference = calculator.subtract(5, 2);
        // The test asserts that the difference is not equal to zero, which is irrelevant
        assertNotEquals(0, difference);

        int product = calculator.multiply(3, 4);
        // The test asserts that the product is positive, which is not always true
        assertTrue(product > 0);

        int quotient = calculator.divide(10, 2);
        // The test asserts that the quotient is less than the dividend, which is not always true
        assertTrue(quotient < 10);

        try {
            calculator.divide(5, 0);
            fail("Expected an ArithmeticException to be thrown");
        } catch (Exception e) {
            // Do nothing
        }
        // The test does not assert anything meaningful because the try-catch are looking for all exceptions, but it should be looking for ArithmeticException
    }
}

In the example, our assertions are not meaningful enough. It exercises the code, but will not help much when it comes to detecting problems.

If a developer goes into the code and makes the same change as mentioned before in the add method:

  public int add(int a, int b) {
        return a - b;
  }

The tests still pass, and the coverage still will be 100%!

How you can use coverage in your favor:

So maybe you’re thinking, if I can't trust coverage to ensure my project is well tested, how can I use coverage in my favor?

Coverage is a great tool to see visually which parts of the code (lines, conditions, functions, etc) are being exercised or not. That’s an awesome tool for that because you can see which partitions of your code need some attention when writing your tests! Also, it helps you come up with more test scenarios for your code. Adding more scenarios you will be a step closer to finding bugs before they go to your production code.

Look at this code:

public class EvenOddChecker {

    public boolean isEven(int number) {
        if (number % 2 == 0) {
            return true;
        }
        return false;
    }
}

If we decided to test this simple method and only add a test for the even part:

@Test
public void testIsEvenTrue() {
    EvenOddChecker checker = new EvenOddChecker();
    assertTrue(checker.isEven(4));
}

When you run coverage you’ll see that you only tested part of the method, not all lines of the method were exercised:

Look at line 9. It is colored differently because it was not covered yet. Coverage shows you visually what your test scenarios missed. Therefore, you can add tests and make sure the scenarios cover all branches of your code.

Then we can go and fix it by adding an odd scenario to cover the missing lines:

@Test
public void testIsEvenFalse() {
   EvenOddChecker checker = new EvenOddChecker();
   assertFalse(checker.isEven(5));
}

That will give us 100% coverage:

But remember, you still need to add effective tests to make this worth it.

That’s how to use coverage well. It helps you to focus on the parts that need more attention in your test suite. Now you can go there and write tests with meaningful assertions to cover these forgotten branches.

Conclusion

In this article, you discovered why coverage is not enough to measure the quality of your tests, how to dodge common pitfalls that can trick the coverage metric, and how to write better tests that exercise the code meaningfully.

I hope you found this article useful and enlightening. If you did, please share it with your friends and coworkers who might benefit from it. And if you want to learn more about effective testing, please follow me on dev.to and social media. Thank you for reading and happy testing!

Willian Moya (@WillianFMoya) / X (twitter.com)

Willian Ferreira Moya | LinkedIn