Different code coverage types and the importance of a good report

You probably already know that code coverage tells us what percentage of our code is covered by tests. But did you know there are different coverage types and that the expressiveness of the coverage results depends on which type you choose?

In this article, we’ll take a deeper look at what code coverage is, which types of coverage exist, and the importance of having a good coverage report.

What is code coverage?

Your code is never going to be perfect. Even if you’re an experienced developer, it’s only natural that you’ll make mistakes at some point or forget particular scenarios. That’s why testing your code is such a critical part of the development lifecycle.

In a nutshell, code coverage is a metric that specifies how much of your codebase is covered by tests. It is an essential metric because your tests might have good results, but if they only cover 20% of your codebase, it’s hard to be confident about the overall quality of the product.

The different types of code coverage

We’ll briefly explain line, branch, condition, and MC/DC coverage by looking at a simple example. Let’s assume you only go to the beach if it’s sunny and the weather forecast is good - meaning the temperature is between a specific range of degrees (those values are entirely up to you!).

boolean isBeachTime(boolean isSunny, boolean isGoodTemperature) {
     boolean isBeachWeather = false;
     if (isSunny && isGoodTemperature) {
          isBeachWeather = true;
     }
     return isBeachWeather;
}

Yes, we could’ve simplified the code above to only return (isSunny && isGoodTemperature)! However, we chose this version to illustrate the different coverage types.

There are several ways to determine how well tests cover your code. Various metrics offer a different perspective on your code quality, and it’s helpful to know the basics of each of them. So now that we have our running example let’s dive in!

Line coverage

Line coverage is probably the oldest and most common way of calculating code coverage. It tracks which lines of code your tests executed and gives you a percentage based on the total number of code lines.

For our example, a single test case is enough to get full line coverage:

isBeachTime(true, true) == true;

But what happens if we have the following slightly modified code?

void showBeach(boolean isSunny, boolean isGoodTemperature) {
     Beach ourFavouriteBeach;
     if (isSunny && isGoodTemperature) {
          ourFavouriteBeach = new Beach();
     }
     return ourFavouriteBeach.show();
}

We get full coverage with showBeach(true, true), but if we call showBeach(false, false), we’ll miss a NullPointerException, even though we have full line coverage. As such, line coverage can give us false confidence about the code’s quality.

Branch coverage (or decision coverage)

The problem with line coverage in the showBeach example is that it doesn’t force us to write tests so that every possible branch is visited. For that, there’s branch coverage, also known as decision coverage.

A branch can occur due to if and switch-case statements, while loops, catch blocks, and other boolean expressions. Branch coverage sees the conditional logic branches in the code and ensures that tests cover all of them.

To get full branch coverage for our example, we must consider every outcome of the boolean expression (isSunny && isGoodTemperature):

isBeachTime(true, true) == true;
isBeachTime(false, false) == false;

Thus, full branch coverage allows us to detect the NullPointerException in our modified example. However, there are more complicated code constructs where even branch coverage is insufficient.

Condition coverage

Condition coverage analyzes code statements that include conditions, such as if-statements. It checks if there are tests for the conditions to be at least once true and once false. 

Our example contains a single condition: if (isSunny && isGoodTemperature)

The tests need to ensure that both isSunny and isGoodTemperature were at least once true and once false. We can achieve full condition coverage with the same tests for branch coverage:

isBeachTime(true, true) == true;
isBeachTime(false, false) == false;

However, condition coverage does not imply branch coverage. We reach full condition coverage with the following two tests, but not full branch coverage:

isBeachTime(true, false) == false;
isBeachTime(false, true) == false;

Condition coverage doesn't imply branch coverage as well.

Modified condition / decision coverage (or MC/DC)

Modified condition / decision coverage, also known as MC/DC, can be seen as a combination of branch and condition coverage. As such, it’s stronger than both of them. MC/DC is used in safety-critical systems and is required in international standards like ISO 26262 and DO 178C.

To achieve full MC/DC coverage, we have to make sure that:

• every boolean expression in a control statement takes all possible outcomes at least once (branch coverage);
• every condition within a boolean expression takes all possible outcomes at least once (condition coverage); and
• each condition has been shown to independently affect the outcome of the boolean expression (by varying just that condition while holding all other possible conditions).

MC/DC coverage has semantic requirements for tests. In our example, the tests have to reflect that the absence of either sun or good temperature can keep us from going to the beach. Look again at the two tests in branch coverage:

isBeachTime(true, true) == true;
isBeachTime(false, false) == false;

These tests satisfy the first two MC/DC criteria requirements: the boolean expression (isSunny && isGoodTemperature) takes every possible outcome, and every predicate within that expression is at least once true and once false. However, the tests don’t satisfy the third requirement since they don’t capture that either isSunny = false or isGoodTemperature = false would be enough to make the boolean expression false.

So let’s look at the second set of tests in condition coverage:

isBeachTime(true, false) == false;
isBeachTime(false, true) == false;

These tests satisfy the second and third requirements of the MC/DC criteria. However, in both tests, the boolean expression (isSunny && isGoodTemperature) does not hold, so the first requirement is not satisfied.

We need a combination to achieve full MC/DC coverage:

isBeachTime(true, true) == true;
isBeachTime(true, false) == false;
isBeachTime(false, true) == false;

These tests cover each requirement of the MC/DC coverage.

Other types of code coverage

There are others types of code coverage to keep in mind when thinking about your testing. We won’t dive into them in detail, but it’s still good to know they exist. Some examples include:

• Statement coverage: measures the number of code statements executed when the code runs. It helps uncover unused statements and branches, missing statements, and dead code.
• Function coverage: measures the number of declared functions covered by tests. It’s important in software that relies on a large number of functions.
• Path coverage: measures the number of linearly independent paths (given by McCabe’s cyclomatic complexity) covered by tests to identify broken, redundant, or inefficient paths.

The importance of a good coverage report

If your test suite is automated (as it should!), you can run all your tests and have a tool like Codacy to check the executed pieces of code. The ultimate power of code coverage is in the report.

First of all, you'll get an overall coverage percentage. So, for example, the Codacy repository dashboard might tell you that your tests cover 48% of your code, which means you should add more tests.

Codacy dashboard, with coverage

The overall percentage isn't the only exciting information. Codacy also shows you a detailed view of code coverage per file. That way, you can focus first on areas with the lowest code coverage.

Codacy files view, with coverage

You can define thresholds where pull requests are not accepted if code coverage is below a specific percentage. This encourages your team to add tests to their code. Finally, when your code coverage increases, you should also increase the threshold. Vevo followed this strategy!

PR not up to standards

The code coverage report is important because it can tell us where we need to focus first. Tools like Codacy can also assist with prioritization by detecting high-risk code.

Get started for free

<!-- /wp:button -->

Conclusion

A well-tested codebase is usually (but not always) a well-structured codebase. This means it will be easier for developers to modify the code or add new features. The developers will also feel safer, as a good (automated) test suite provides a safety net for changes.

For most software, 80%-90% code coverage is an excellent score. Tools like Codacy can help you visualize your score and point out files and lines of code that need more tests, allowing you to be more confident about your code quality.

However, remember that, as Edsger Dijkstra, an early contributor to software engineering development, eloquently stated: Testing can only show the presence of errors, not their absence.