False-positive Go code coverage

Go offers tools for code coverage directly in the main installation package, no need for addons or any other tool. But with the initial settings, the coverage results could be much higher than a real coverage.

🇨🇿 V češtině si lze článek přečíst na kutac.cz

---

To understand why the code coverage can be much higher is mandatory to understand, how the coverage is measured. Go has some internal tools, which modify the code before executing tests and then counting occurrences in each path. But is counting occurrences only in the currently tested package.

Let's say you have a package A and B. Package A is fully covered by tests and is using some functions from package B, which has zero tests. The result will be that 100% of statements are covered. But if you add a test into package B, overall coverage can drop.

Example

For example purposes I created repo arxeiss/go-false-positive-coverage, which contains 2 completely separated packages unary and binary.

Step 1 - False hope I have 100% coverage

Clone the repo mentioned above and try to run the following commands. In the result, you can see a total coverage of 100%. It is only because there are no tests in unary package. Then there are no records in coverage.out file about other packages, so go tool cover knows nothing about them.

go test -covermode=count -coverprofile=coverage.out ./...
# ?     github.com/arxeiss/go-false-positive-coverage   [no test files]
# ok    github.com/arxeiss/go-false-positive-coverage/binary    0.002s  coverage: 100.0% of statements
# ?     github.com/arxeiss/go-false-positive-coverage/unary [no test files]

# Count total coverage and coverage per function.
go tool cover -func coverage.out
# github.com/arxeiss/go-false-positive-coverage/binary/operations.go:3: Add     100.0%
# github.com/arxeiss/go-false-positive-coverage/binary/operations.go:7: Sub     100.0%
# github.com/arxeiss/go-false-positive-coverage/binary/operations.go:11:    Mul     100.0%
# github.com/arxeiss/go-false-positive-coverage/binary/operations.go:15:    Div     100.0%
# total:                                    (statements)    100.0%

Step 2 - Adding more tests drops overall coverage

There is a prepared test in the unary package. Just rename the file operations_test.go.bak and remove .bak suffix. Now run the same commands as above and total coverage will be 71.4%!.

Step 3 - Switch -coverpkg will come to help

To get the real overall score, it is mandatory to use the -coverpkg switch and define all packages to test. This switch has 2 effects.

1. Will insert records about packages without tests into coverage.out.
2. Will measure the coverage across packages. So package without a single test can be fully covered if it is used by other packages.

go test -covermode=count -coverprofile=coverage.out -coverpkg=github.com/arxeiss/go-false-positive-coverage/... ./...
# ?     github.com/arxeiss/go-false-positive-coverage   [no test files]
# ok    github.com/arxeiss/go-false-positive-coverage/binary    0.002s  coverage: 44.4% of statements in github.com/arxeiss/go-false-positive-coverage/...
# ok    github.com/arxeiss/go-false-positive-coverage/unary 0.002s  coverage: 11.1% of statements in github.com/arxeiss/go-false-positive-coverage/...

go tool cover -func coverage.out
# github.com/arxeiss/go-false-positive-coverage/binary/operations.go:3: Add     100.0%
# github.com/arxeiss/go-false-positive-coverage/binary/operations.go:7: Sub     100.0%
# github.com/arxeiss/go-false-positive-coverage/binary/operations.go:11:    Mul     100.0%
# github.com/arxeiss/go-false-positive-coverage/binary/operations.go:15:    Div     100.0%
# github.com/arxeiss/go-false-positive-coverage/main.go:10:     main        0.0%
# github.com/arxeiss/go-false-positive-coverage/unary/operations.go:5:  Abs     100.0%
# github.com/arxeiss/go-false-positive-coverage/unary/operations.go:9:  Sin     0.0%
# github.com/arxeiss/go-false-positive-coverage/unary/operations.go:13: Cos     0.0%
# total:                                    (statements)    55.6%

Step 4 - Problems with generated code

The main problem of the -coverpkg switch is, it will include also generated code, like Proto files, etc. Those are mostly not tested explicitly, because they are generated. To ignore them, you have to filter out those records from coverage.out file with some shell magic. See more in StackOverflow.

cat coverage.out | grep -v ".pb.go" > coverage.filtered.out
go tool cover -func coverage.filtered.out