How do you measure your test coverage?

Corey Johnson on April 16, 2019

In some places and teams I've worked with, I've been asked what % of test coverage we want to shoot for and I've never had a good answer. I perso... [Read Full]
markdown guide
 

However it seems odd to me that you'd have a test for an immutable model that tests all fields.

We ignore files that don't necessarily need to be tested at the moment, even if they don't fit the above criteria. Code coverage is inherently pretty arbitrary so chasing a number without purpose is probably not ideal.

If 100% indicates success, set yourself up for 100% by only measuring the important parts. Once you get near 100% you could expand for other nice to haves.

I'll add by saying that code coverage is a pretty "bad" metric, but no metrics are perfect, and I do think it can serve a purpose when used correctly.

 

If 100% indicates success, set yourself up for 100% by only measuring the important parts. Once you get near 100% you could expand for other nice to haves.

Yes! Aim for 100% on the important things, but just as important, do not sacrifice simplicity for the 100%. If you have a pretty high coverage already, and the core rules are covered, 80~90+ should be fine.

The two rules I try to follow when programming are:

  • Don't make me think.
  • Don't make me work.

If I have to do any of those two while maintaining the code, then something might be off.

If I have to combine 10 components to complete a behavior it's too much work (it might cover the 100% though). If It takes me more than 2 minutes to understand the design, I'm being set up for failure (I'll probably introduce a bug).

The domain and the amount of time you have to manually test things will determine what number your percentage may be (I work on an open source project and since I didn't design it with tests, and I don't have the time any more to test it manually (testing on games take a lot of time), I release almost zero features per year...)

As Woody Zuill would say "Maximize the good". Figure out what works and then maximize it for the best result. No hard or fast rules here: as long as people care for the code base is what matters most.

 

Yea I’m with Ben on the imperfection of code coverage as a metric. I set up Istanbul/NYC to ignore files that are just wrappers on API calls, files with constants, and any file where the type system is doing all of he heavy lifting. But honestly, I don’t use code coverage as anything more than a fun thing to watch on the build. Instead, I use the number of bugs that pop up in a given sprint as my measure of quality. And if a bug occurs then I know that my code coverage was low and/or that our tests were flaky. So I immediately go beef up the tests. :)

 

I agree, I think having some idea of coverage is better than none. I tend to use acceptance criteria to write tests and then break out unit tests where it makes sense. This gives me a pretty good spread of tests as I work.

 

In some places and teams I've worked with, I've been asked what % of test coverage we want to shoot for and I've never had a good answer.

There is no good answer, it's a vanity metric.

Test coverage tools are a good way of understanding what parts of your code are untested, that's all.

Dont get me wrong, I love tests! But coverage doesn't tell you the value of your tests in respect to giving you the ability to change and refactor your system with confidence. Unfortunately there's not really a number that can be assigned to that.

 

Each of my tests enforces its own independent coverage rules for what code it needs to cover, and it needs to cover those things 100% or the test fails. This allows for strict coverage where you want it, but leaves all of that up to the judgement of the test writer. So some tests are not required to cover anything, while others might be required to independently achieve 100% coverage of multiple classes under test.

 

I use Mutation Testing as my metric. Thats far better than code coverage. For .NET I‘m using stryker-mutator.io/

code of conduct - report abuse