DEV Community

Paul J. Lucas
Paul J. Lucas

Posted on

Code Coverage Testing in Autotools

#c #cpp #autotools

Introduction

The previous article in this series showed how you can write and run and end-to-end test suite in Autotools. But how do you know your test suite tests enough? One way is to add code coverage so that after you run your test suite, you can view a report that shows, for each file, what percentage of lines of code executed at least once and see which ones did not execute at all. You can then add more tests to test the lines that didn’t execute.

Note that even if you have 100% code coverage, that does not mean your program is bug-free. Subtle bugs occur for many reasons, e.g., when either statements are executed in a particular order or when your program is in a particular state.

Another caveat of code coverage is that just because, for example, line 101 in foo.c and line 247 in bar.c were executed during a run of the entire test suite and so were covered, that doesn’t necessarily mean they were both executed in the same run of your program for which there might be a bug.

Having a high code coverage (≥ 90%) is certainly good in that it helps find bugs, but don’t be fooled into thinking your program is bug-free.

Background

Part of the gcc compiler tools is gcov, the GNU code coverage tool. This can be integrated into your build to provide code coverage reports.

To implement code coverage, the compiler has to generate additional object code per source code line to increment a counter for how many times that line has been executed during not only the lifetime of a run of your program, but across running all the tests comprising your test suite.

Because gcov is tied to gcc, you must use gcc and not clang or some other compiler. Other compilers may provide their own auxiliary tools for code coverage, but this article will use gcov. Despite that, the techniques shown on how to integrate it with Autotools will likely be tweakable for other coverage tools.

Enabling Coverage

Because programs have to be compiled differently to enable coverage, you don’t want coverage enabled all the time. Additionally, it’s generally the case that when coverage is enabled, optimization should be disabled so you get an accurate source-to-object-code mapping.

A previous article in this series showed how to enable optional features. For coverage, the following can be added to configure.ac:

AC_ARG_ENABLE([coverage],
  AS_HELP_STRING([--enable-coverage],
                 [enable code coverage])
)
Enter fullscreen mode Exit fullscreen mode

That will set the shell variable enable_coverage to yes if the option is given to configure.

To disable optimization, the following needs to be added immediately after:

AS_IF([test "x$enable_coverage" = xyes], [: ${CFLAGS=""}])
Enter fullscreen mode Exit fullscreen mode

What that does is explained by the AC_PROG_CC documentation:

If output variable CFLAGS was not already set, set it to -g -O2 for the GNU C compiler .... If your package does not like this default, then it is acceptable to insert the line : ${CFLAGS=""} after AC_INIT and before AC_PROG_CC to select an empty default instead.

Hence, you need to put the AC_ARG_ENABLE and the AS_IF early in your configure.ac, specifically before AC_PROG_CC.

Checking for Coverage Tools

Just because the user enabled coverage doesn’t mean the coverage tools are present on the user’s computer. Later in configure.ac, you need to add this:

AS_IF([test "x$enable_coverage" = xyes], [
  AS_IF([test "x$GCC" != xyes], [
    AC_MSG_ERROR([gcc is required for code coverage])
  ])
  AC_CHECK_TOOL([GCOV], [gcov], [gcov])
  AS_IF([test "x$GCOV" = "x:"], [
    AC_MSG_ERROR([required program "gcov" for code coverage not found])
  ])
  AC_CHECK_PROG([LCOV], [lcov], [lcov])
  AS_IF([test "x$LCOV" = x], [
    AC_MSG_ERROR([required program "lcov" for code coverage not found])
  ])
  AC_CHECK_PROG([GENHTML], [genhtml], [genhtml])
  AS_IF([test "x$GENHTML" = x], [
    AC_MSG_ERROR([required program "genhtml" for code coverage not found])
  ])
  AC_DEFINE([ENABLE_COVERAGE], [1], [Define to 1 if code coverage is enabled.])
])
Enter fullscreen mode Exit fullscreen mode

The first inner AS_IF ensures the compiler is actually gcc via the GCC variable that is automatically set to yes only if the C compiler is gcc.

AC_CHECK_TOOL and the following AS_IF checks for and ensures the existence of gcov itself.

AC_CHECK_PROG and the following AS_IF checks for an ensures the existence of lcov, a related tool that uses the output of gcov to generate nice coverage reports in HTML easily viewed in a browser. Simiarly, genhtml, part of lcov, is also checked for.

Finally, if all requisite programs are present, AC_DEFINE defines ENABLE_COVERAGE in config.h.

Additionally, it’s also needed to add a makefile conditional via:

AM_CONDITIONAL([ENABLE_COVERAGE], [test "x$enable_coverage" = xyes])
Enter fullscreen mode Exit fullscreen mode

One last thing that’s handy is to add the following early, right after AM_INIT_AUTOMAKE:

AM_EXTRA_RECURSIVE_TARGETS([clean-coverage distclean-coverage])
Enter fullscreen mode Exit fullscreen mode

AM_EXTRA_RECURSIVE_TARGETS adds those additional targets so that if you type make target at the top level, make will make that target in every subdirectory, in particular src. (More on why those targets later.)

That’s everything that’s needed in configure.ac. Now for src/Makefile.am.

Compiling with Code Coverage Enabled

In src/Makefile.am, you need to add a few things so that your program is compiled with code coverage enabled, specifically:

if ENABLE_COVERAGE
AM_CFLAGS +=    --coverage -g -O0
AM_LDFLAGS =    --coverage
endif
Enter fullscreen mode Exit fullscreen mode

When compiled with code coverage enabled, your program will generate additional files to store line-execution-count information that lcov will used to generate the HTML report. For example, for foo.c, the files foo.gcda and foo.gcno will be generated.

If you type make clean or make distclean, you want those files to be deleted. To do that, add the following to src/Makefile.am:

clean-coverage-local:
        rm -f *.gcda

distclean-coverage-local: clean-coverage-local
        rm -f *.gcno
Enter fullscreen mode Exit fullscreen mode

Additionally, if you re-run your test suite via make check, you want to clear out line-execution-count information from the previous run. To do that, add:

check-local: clean-coverage-local
Enter fullscreen mode Exit fullscreen mode

Lastly, you should add the following to src/.gitignore:

/*.gcda
/*.gcno
Enter fullscreen mode Exit fullscreen mode

so git will ignore those files.

Generating Code Coverage Reports

At this point, your code has been compiled with code coverage enabled and running make check will run your test suite collecting coverage data. But to generate the reports, you need to add some things to the top-level Makefile.am:

COVERAGE_INFO = $(top_builddir)/coverage.info
COVERAGE_DIR =  coverage

LCOV_FLAGS =    --capture \
                --config-file "$(top_srcdir)/lcovrc" \
                --directory "$(abs_top_builddir)" \
                --output-file $(COVERAGE_INFO)

GENHTML_FLAGS = --legend \
                --title "$(PACKAGE_NAME)-$(PACKAGE_VERSION) Code Coverage" \
                --output-directory $(top_builddir)/$(COVERAGE_DIR)
Enter fullscreen mode Exit fullscreen mode

where:

  • COVERAGE_INFO specifies the file into which generated code coverage information is to be placed.
  • COVERAGE_DIR specifies a subdirectory to create that will contain the generated HTML files.
  • LCOV_FLAGS specifies options for lcov (more later).
  • GENHTML_FLAGS specifies options for genhtml.

To actually collect code coverage information and generate a report, you also need to add the following to the top-level Makefile.am:

check-coverage:
if ENABLE_COVERAGE
        $(MAKE) $(AM_MAKEFLAGS) check
        $(LCOV) $(LCOV_FLAGS)
        $(GENHTML) $(GENHTML_FLAGS) $(COVERAGE_INFO)
        @echo "file://$(abs_builddir)/$(COVERAGE_DIR)/index.html"
else
        @echo "Code coverage not enabled; to enable:"
        @echo
        @echo "    ./configure --enable-coverage"
endif

clean-local: clean-coverage-local

clean-coverage-local:
        rm -f $(COVERAGE_INFO)

distclean-local: distclean-coverage-local

distclean-coverage-local:
        rm -fr $(COVERAGE_DIR)
Enter fullscreen mode Exit fullscreen mode

The check-coverage target is a new variant of the built-in check target that runs make check and calls lcov and genhtml to collect the coverage information and generate the HTML — but only if ENABLE_COVERAGE is true.

The clean and distclean targets are just to do proper clean-up upon either a make clean or a make distclean.

Lastly, you should add the following to .gitignore:

/coverage*
Enter fullscreen mode Exit fullscreen mode

so git will ignore those files.

lcov Configuration

The lcovrc file is lcov’s configuration file. It has many configuration options and I’m not going to cover them here. The one used for cdecl contains:

geninfo_external = 0
lcov_excl_line   = LCOV_EXCL_LINE|unreachable
Enter fullscreen mode Exit fullscreen mode

where:

  • geninfo_external enables or disables whether coverage data is captured by “external” source files, those not within the directory given by --directory option, e.g., the files in the lib subdirectory from Gnulib. I disable this because I’m not testing that code, so I don’t care about its coverage.
  • lcov_excl_line specifies a pipe (|) separated list of identifiers that, if any one is present on a line of code, that line is excluded from being counted.

Excluding Lines from Coverage

Why would you want to exclude lines from coverage? Sometimes you have lines of code that are just there to silence a warning or handle a degenerate case. Hence, there’s no point in writing a test to ensure that line is executed. Nevertheless, you don’t want your lack of testing that line to detract from a file’s coverage percentage. For such lines, you can add a comment containing LCOV_EXCL_LINE. For example, the following function frees the name of a p_param_t data structure, but you want it to do nothing if param is NULL:

void p_param_free( p_param_t *param ) {
  if ( param == NULL )
    return;                     // LCOV_EXCL_LINE
  free( param->name );
  free( param );
}
Enter fullscreen mode Exit fullscreen mode

You could argue that you want to test such lines also, at least in unit tests. If you do, fine: remove the LCOV_EXCL_LINE; but for general testing, you don’t really care.

You’d also want to exclude unreachable since such lines are never executed by definition.

lcov has a built-in way to exclude multiple lines of code by using LCOV_EXCL_START and LCOV_EXCL_STOP. For example, your program’s -v option prints its current version, but that changes for every version that would require updating expected output of the test. If you don’t want to bother testing that, you could exclude those lines of code:

// ...
if ( opt_version > 0 ) {
  // LCOV_EXCL_START -- since the version changes
  print_version();
  exit( EX_OK );
  // LCOV_EXCL_STOP
}
Enter fullscreen mode Exit fullscreen mode

Example Reports

Now that it’s all set-up, you can just type make check-coverage and it will run your test suite and generate the HTML report. For example, here’s an excerpt from cdecl’s report:

cdecl top-level code coverage

and here’s an excerpt from a particular file that you get when you click on a file’s name:

cdecl file code coverage

Everything should be fairly self-explanatory. Lines that are not covered are highlighted in orange. Given this information, you can devise new tests that test those lines and add them to your test suite to make it more comprehensive.

Conclusion

While code coverage can’t eliminate all bugs, it can certainly help reduce them by ensuring your test suite tests all of what you think it’s testing. Integrating code coverage into your project via Autotools does require several parts, but none is particularly complicated.

More parts to come!

Top comments (0)