Code coverage. You keep using that word. I do not think it means what you think it means

Most companies use the code coverage metric to have some indication of how well their code has been tested. However, as many have pointed out, code coverage is a very poor metric for indicating whether or not your code has been tested well.

100% code coverage = 0% code tested

Indeed, observe the following test:

int AddOne(int in)
{
  return in - 1;
}

TEST(MyTest, TestsAbsolutelyNothing)
{
  const auto val = AddOne(1);
}

This example is a bit extreme, but in a large legacy code base, situations like this can easily occur. This test will have 100% code coverage of AddOne, but due to the missing expectation, it will not test anything.

There is no direct correlation between code coverage and the amount of code that has been tested.

This doesn’t mean that code coverage is a useless metric. Code with a high code coverage tends to be better tested than code with a very low code coverage metric. But relying on high code coverage can lull us into a false sense of security.

Inconcievable!

The problem is that code coverage is a poor term. The issue stems from the term coverage. When you have in your head that something is “covered”, you immediately think that it has been taken care of (testing wise). With code coverage, as we have seen, this is not the truth.

Let’s look at a different example:

int Add(int in, int value)
{
    return in + value;
}

int Subtract(int in, int value)
{
  return in - 1;
}

TEST(MyTest, TestAddAndSubtract)
{
  EXPECT_EQ(2, Add(1, 1));
  EXPECT_EQ(0, Subtract(1, 1));
}

In this test we have added expectations, and still have 100% code coverage. This will suggest that the code is tested, but there is an obvious error in the code still.

Inverting the term

Instead, we should invert the metric, and use another term. In lack of a better name, I will use the term “Legacy Code”. This term, I have borrowed from Dean Martins excellent book “Working effectve with legacy code”, wherein he describes legacy code as code with no tests. “Legacy code%” is then equal to “100% - code coverage%”. And this metric accurately tells us how many percent of the code has not been tested at all.

So, where previously your CI tool would report 80% code coverage, now should now report 20% legacy code.

Legacy code% = 100% - code coverage%

Now, having 0% legacy code still does not mean that code has been tested well. It could still be tested poorly as in the second example. So the metric is still fairly flawed. In the end, tests can never prove the absense of errors.

However, I believe that getting rid of the word “coverage” will impove peoples understanding of what this metric actually means.