My Autoconf Primer

Since a young age I've learned that build systems and dependency management range from difficult to mind-bogglingly frustrating. My best experiences have been with Autotools, which might surprise many. However, I had never made anything serious from scratch using Autotools myself, but rather only modified things setup by others. Lately I had some positive experiences building C programs linking libguile with Autoconf and this prompted me to try something bigger. I setup the build toolchain for a C++ program using Autotools. In this blog post I attempt to document what I learned along the way.

Autotools, Automake and Autoconf

The first question I wanted answered is if I should be using Autotools, Autoconf or even Automake. The answer is that Autotools is not a binary or similar, but rather it's an umbrella term for Autoconf, Automake and everything related. On the other hand, automake is a binary that converts a Makefile.am into Makefile.in.

In my brand new program, I needed to learn very little about Automake. I created a top-level Makefile.am containing a pointer to the related documentation and an instruction to recurse into the subdirectory src:

$ cat Makefile.am
SUBDIRS = src
dist_doc_DATA = README.md

The file in src/Makefile.am contains information of the sources to compile into objects and the resulting binaries:

$ cat src/Makefile.am 
bin_PROGRAMS = test_algorithm
test_algorithm_SOURCES = \
    algorithm.cpp \
    exception.cpp \
    job.cpp \
    warning.cpp \
    test_algorithm.cpp

You can run automake as a standalone program. However, I found it far more convenient to call it using autoreconf. autoreconf is a program that runs automake and autoconf (and more) conveniently for you. Typically, I run autoreconf with the following switches: -i to generate missing auxiliary files, -v for verbose output and -f to force regeneration of all configuration files.

$ autoreconf -vif

But now I'm getting ahead of myself. Before we can run autoreconf, we need to write the input for autoconf, namely, configure.ac.

Introduction to Autoconf

The role of Autoconf is to generate the configure to ship with a software distribution. The input file configure.ac is formatted in the macro language M4.

Now, this might be a bit confusing, but configure (and hence autoconf) is not meant to work with software dependency management. Instead, the main design principle of Autoconf is to work with features and capabilities of the installed system where the distributed software is compiled.

config.h

Although I didn't use it myself, I need to mention config.h. (for what it is worth, I should probably try it out and update this part of the blog post later)

As a result of running configure, a header file config.h is typically generated. This header file contains detected capabilities of the system we are running the compilation on. For example, if we look for pthread, config.h will contain suitable #define's to indicate if pthread is found and linkable or not.

As stated previously, I did not utilize config.h and therefore no such file is generated in my case.

Structure of configure.ac

The first statement of configure.ac must be AC_INIT and the last must be AC_OUTPUT. The rest of the statements can be placed anywhere, except for when they depend on each other. This dynamic seems to have created a de facto standard structure for configure.ac files. Even the Autoconf manual has a section Standard configure.ac Layout. However, I find it quite confusing and instead I follow a mental picture, which is as follows:

• Initialization
• Compiler settings
• Configuration Files
• Dependencies
• Output

As previously stated, AC_INIT is the first statement. It accepts parameters such as program name, program version, maintainer e-mail address etc. In my case it is:

AC_INIT([algorithm], [1.0])

As part of the initialization phase, I also put a statement to initialize automake with AM_INIT_AUTOMAKE. I found this in the automake manual. Note that statements starting with AC_ are for Autoconf and statements starting with AM_ are macros for Automake. However, AM_INIT_AUTOMAKE is the only statement for Automake in my case.

Now, this is really important: We give some parameters that look just like compiler flags, but they are for Automake when it generates Makefile.in etc. I used -Wall, -Werror and foreign. I needed foreign, because otherwise Automake requires all the files in a standard GNU package, i.e. NEWS, README, AUTHORS and ChangeLog. My AM_INIT_AUTOMAKE is as follows:

AM_INIT_AUTOMAKE([-Wall -Werror foreign])

Compiler settings

Next we need to select a compiler. When compiling a standard C program, we use AC_PROG_CC. In my case, I'm compiling C++ so I use AC_PROG_CXX instead. Note that the macro AC_PROG_CPP is related to the C preprocessor and has nothing to do with C++ (for the C++ preprocessor command, use the macro AC_PROG_CXXCPP).

The only parameter that AC_PROG_CC and AC_PROG_CXX accept are a compiler search list, but no compiler flags. Instead, if you want to specify compiler flags, you need to set CFLAGS or CXXFLAGS before calling the macro. As per the Autoconf manual, by default AC_PROG_{CC,CXX} sets {C,CXX}FLAGS} to -g -O2 (or -g if the detected compiler is not a GNU compiler). But if {C,CXX}FLAGS was already set, it's kept unchanged.

In my case, I call the macro AC_PROG_CXX with the following construct:

: ${CXXFLAGS="-g -O2 -Wall -Werror"}
AC_PROG_CXX

Furthermore, as my source code uses C++17 features, I use an extra macro for that. But it's a little special and I will come back to it later in the section Autoconf Archive.

Configuration files

As explained earlier, Automake compiles an Makefile.in from Makefile.am. However, this Makefile.in is not yet a Makefile. To specify which makefiles need to be compiled, we use the macro AC_CONFIG_FILES. In my case:

AC_CONFIG_FILES([
    Makefile
    src/Makefile
])

Note that we will still distribute the Makefile.in and the Makefile will be generated only on site. The instructions to build the file are contained in config.status while the configure will use.

Furthermore, the utility of AC_CONFIG_FILES is not retricted to makefiles, but other files can be specified, too. However, I had no such use.

Dependencies

As explained earlier, Autoconf doesn't work with software dependency managers in the conventional sense. I'll call this section Dependencies anyhow, as that's how I think about it. There are various things you can do here, and in my case I was happy I needed 3 different ways to search for libraries and headers, as that really made me learn.

Dependencies: AC_CHECK_LIB

The most typical thing you would do is AC_CHECK_LIB. To AC_CHECK_LIB you give the library name, a symbol in the library, and, optionally, a code segment to execute on success and a code segment to execute on failure. For example, in my scenario I ended up doing

AC_CHECK_LIB([pthread], [pthread_create], [], [AC_MSG_ERROR([pthread is not installed.])])

My first surprise was that a symbol from the library is needed. In fact, the macro will expand into a C program that will be compiled. The source of the C program will contain the symbol we indicated, i.e. pthread_create and the program will be linked against the library, i.e. with the flag -lpthread. If compilation succeeds, the third parameter will be executed, but since its [] the default action will be taken. Note that to disable the default action you can do [ ]. If compilation fails, we run the fourth parameter which is a macro to signal an error and halt operation.

The default action is to add the specified library to LIBS, i.e. in my case LIBS="-lpthread $LIBS" (yes, I double-checked from the generatedconfigure). Furthermore, the defineHAVE_LIBlibrarywill be set. In my case, it's#define HAVE_LIBPTHREAD 1`.

I think there's at least two takeaways here. First of all, we don't match libraries with any kind of versioning. Instead, we look for a symbol. I would assume this can cause headache with missing or deprecated features. Furthermore, the macro AC_CHECK_LIB uses always C linking. This means that C++ libraries cannot be added using this construct, unless the library also has some symbols using C linking (i.e. extern "C").

Dependencies: Linking C++ libraries

Since we can't use AC_CHECK_LIB for linking C++ code, what other options do we have? There's a macro AX_CXX_CHECK_LIB, but that is not distributed as part of Autoconf and not even the Autoconf Archive, so I have decided against using it.

Instead, I found a blog post that suggests writing your own check. The basic idea is to use the macro AC_LINK_IFELSE (this is the same macro as AC_CHECK_LIB uses). To AC_LINK_IFELSE you give a source to compile and a segment to run on success and a segment to run on failure. In my case:

AC_LINK_IFELSE(
    <source here>,
    [HAVE_<library>=1],
    [AC_MSG_ERROR([<library> is not installed.])])

To generate the source code that is used to verify linking, the blog post suggests using the macro AC_LANG_PROGRAM. The macro takes to parameters, a prologue and a body. In my case:

AC_LANG_PROGRAM([#include <gtest/gtest.h>], [EXPECT_EQ(1,1)])

We are still missing two pieces here. First, even though we set AC_PROG_CXX, we have never set the language to use when running compilation tests. This is done using the macro AC_LANG. By setting AC_LANG(C++), AC_LINK_IFELSE will use a C++ compiler (and linker) rather than the default choice of a C compiler.

Furthermore, on success, AC_CHECK_LIB will add the specified library to the variable LIBS. Therefore, we have to do it manually. If we would want to be really fancy, we could unset the library from LIBS, if the test fails. Since we exit on failure, we don't unset anything.

Here's the whole block of code:

AC_LANG(C++)
LIBS="-lgtest $LIBS"
AC_LINK_IFELSE(
    [AC_LANG_PROGRAM([#include <gtest/gtest.h>], [EXPECT_EQ(1,1)])],
    [HAVE_GTEST=1],
    [AC_MSG_ERROR([gtest is not installed.])])

Dependencies: Headers

Some dependencies are merely headers. To check the availability of a header, there's the macro AC_CHECK_HEADER. This macro takes the header as it would be used in a #include preprocessor directive. Furthermore, you can specify an action to take on success and on failure, respectively. In my case, I'm using the header-only library nlohmann-json and only headers from gmock, and thus I'm using the following two statements:

AC_CHECK_HEADER([nlohmann/json.hpp], [], [AC_MSG_ERROR([nlohmann-json is not installed.])])
AC_CHECK_HEADER([gmock/gmock.h], [], [AC_MSG_ERROR([gmock is not installed.])])

Dependencies: pkg-config

The idea behind pkg-config seems simple enough: it's a program that can be used to query for installed software and their versions, respectively. pkg-config requires software to be distributed with package metadata contained in a .pc file. The metadata file will contain various information of the package, such as CFLAGS and LIBS. Sounds like a replacement for AC_CHECK_LIB, right?

The Autoconf Archives provides the macro AX_PKG_CHECK_MODULES. However, on various online forums and blogs, I found warnings about how Autoconf and pkg-config have a fundamentally different approach. Therefore, I decided against using pkg-config in my setup.

As I understand it, one problem is that merely the success or failure of running pkg-config does not adequately indicate if linking the package works. For example, if pkg-config is not installed, the test may fail, even though the dependency we are looking for exists and is linkable. Likewise, there can be a discrepancy between how pkg-config and our compiler/linker of choice works. Lastly, the information in the .pc file can be wrong. The last point is probably relevant when chrooting or similar.

All of configure.ac

In the previous sections, I've explained all of my configure.ac, with the exception of AC_OUTPUT. There's not much to it: we just have to call it in the end of configure.ac, as it will generate the files needed for Automake.

My whole configure.ac is as follows (comments included):

$ cat configure.ac
AC_INIT([algorithm], [1.0])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])

: ${CXXFLAGS="-g -O2 -Wall -Werror"}
AC_PROG_CXX

# check that compiler supports c++17 and set -std=c++17
AX_CXX_COMPILE_STDCXX_17([noext], [mandatory])

AC_CONFIG_FILES([
    Makefile
    src/Makefile
])

AC_CHECK_LIB([pthread], [pthread_create], [], [AC_MSG_ERROR([pthread is not installed.])])

# AC_CHECK_LIB doesnt work well with C++ linking. Instead, we use a test program to see if linking of gtest works.
AC_LANG(C++)
LIBS="-lgtest $LIBS"
AC_LINK_IFELSE(
    [AC_LANG_PROGRAM([#include <gtest/gtest.h>], [EXPECT_EQ(1,1)])],
    [HAVE_GTEST=1],
    [AC_MSG_ERROR([gtest is not installed.])])

# nlohmann-json is a header only library.
AC_CHECK_HEADER([nlohmann/json.hpp], [], [AC_MSG_ERROR([nlohmann-json is not installed.])])

# we need gmock headers even though we don't link it
AC_CHECK_HEADER([gmock/gmock.h], [], [AC_MSG_ERROR([gmock is not installed.])])

AC_OUTPUT

Recap on Usage

Alright, so you've created all the needed source files and now you wonder what commands to use? First, we do
autoreconf to generate configure, Makefile.in and many more files:

$ autoreconf -vif

Note: This step is only done by package maintainers.

To the end-user, we ship all of configure, Makefile.in's etc. The first command the end-user does is to run the
configure script:

$ ./configure

If we, as the package maintainers, did everything correctly, running configure at the end-user will either fail, if
the program (or library) would never be able to compile or run properly. Likewise, assuming we did everything correctly,
if configure succeeds, then also compiling and running the program (or library) will work.

To build the source:

$ make

Note that if we did everything correctly, overriding compiler flags with CXXFLAGS (or CFLAGS if C program), linkable
libraries with LIBS etc will work properly.

The Autoconf Archives

The Autoconf Archives is a separate software distribution from Autoconf itself. It contains a multitude of macros that can be used when generating the configure. Note that it's not an issue to depend on the Autoconf Archives, as we don't distribute the configure.ac, but instead we distribute configure. Hence, the end-user needs neither Autoconf nor the Autoconf Archives.

Conclusion

In this post I went through what I learned as I setup a C++ project which is built with Autoconf (and Automake). Autoconf is quite different from CMake and other build systems. For this reason it's probably off-putting for newcomers as many things seem unnecessarily complicated. However, in my experience, well-written configure files tend to work far better on various systems compared to many other solutions. I doubt I've reached that level of understanding, but everyone needs to start somewhere :-)

While it was definitely frustrating to figure out many things about Autoconf, I'm happy I did it. I feel I understand the world of package management and software distribution a tad better.