Using code coverage to debug large Go application

Code coverage is typically used as a code quality metric.

One can run

go test -coverpkg ./... -coverprofile test.cov ./...

and receive a test.cov file containing information on how often particular parts of code were triggered during execution.

This is a great tool to keep your tests relevant.

However, code coverage can also help you to debug large codebases!

Imagine a situation when a seemingly innocent change in code leads to broken tests. Getting to the root cause might be non-trivial if there is a lot of code involved in execution.

In smaller cases, you could walk through all the statements with a debugger and compare state and conditions, but when you have thousands of statements, this approach is not really feasible.

Code coverage can help you to narrow down the scope of debugging.

Here is the recipe. The prerequisite is that you have a test that passes in original code, and fails in new.

Collect coverage

Collect coverages of both failure and pass.

It is important to collect coverage across all packages to have a wholistic picture. You can use ./... or something like my-module/..., github.com/foo/my-module/... for -coverpkg.

Apply the change and run the test.

go test -cover -coverpkg ./... -coverprofile new.cover -run ^Test_Foo$ ./path/to/tested/package

Then revert the change and collect coverage again.

go test -cover -coverpkg ./... -coverprofile orig.cover -run ^Test_Foo$ ./path/to/tested/package

After that, you will end up having two files: orig.cover and new.cover.

They might be big, but hopefully not very different.

Coverage file may look like this.

mode: set
github.com/swaggest/jsonschema-go/camelcase.go:14.31,21.22 6 1
github.com/swaggest/jsonschema-go/camelcase.go:21.22,22.27 1 1
github.com/swaggest/jsonschema-go/camelcase.go:22.27,24.4 1 1
github.com/swaggest/jsonschema-go/camelcase.go:26.3,26.27 1 1
github.com/swaggest/jsonschema-go/camelcase.go:26.27,28.4 1 1
github.com/swaggest/jsonschema-go/camelcase.go:30.3,30.27 1 1
.....

The first line says which mode of coverage collection was used.

All the other lines describe a span of code (file:start_line.start_col,end_line.end_col) with a total number of statements in that span and a number of statements executed.

Create coverage diff

Coverage files are ordered alphabetically, so they are friendly to diff.

diff orig.cover new.cover > diff.cover

The resulting file may look somewhat similar to this.

305c305
< github.com/swaggest/jsonschema-go/helper.go:75.46,78.16 2 1
---
> github.com/swaggest/jsonschema-go/helper.go:75.46,78.16 2 0
307c307
< github.com/swaggest/jsonschema-go/helper.go:82.2,84.46 2 1
---
> github.com/swaggest/jsonschema-go/helper.go:82.2,84.46 2 0
309c309
< github.com/swaggest/jsonschema-go/helper.go:88.2,88.15 1 1

You can already spot that the lines here are mostly different in the number of executed statements. This is the clue for us that those places in code are the best candidates to have a deeper look with a debugger.

Let's make this diff more convenient to follow and check.

Coverage reporting

Lines starting with < represent the original code, lines starting with > are about the new code.

We can build filtered coverage reports from the diff.

For that, we need to restore the mode: set and remove "< " or "> " from the lines.

echo "mode: set" > orig_flt.cover && grep "< " diff.cover | sed 's/< //' >> orig_flt.cover

echo "mode: set" > new_flt.cover && grep "> " diff.cover | sed 's/> //' >> new_flt.cover

Now, there are two coverage files that only contain differences between two runs.

We can conveniently inspect them with standard go tool cover.

go tool cover -html=orig_flt.cover -o orig_flt.html
go tool cover -html=new_flt.cover -o new_flt.html

You can see which parts of code were executed differently, this can lead to an idea of what could be the root cause or at least can give a hint where debugger break point should be set.

I hope you've enjoyed reading this as much as I enjoyed digging into a huuuuge code base with this approach. :)