How to Support Multiple OSes with One Mac

Making operating-system specific packages of your project can be a great service to your users. But it's also a pain. Linux distributions differ in which package-management system they use, and you need to take the time to learn how to do it correctly. In the end, the scripts you write to make these packages are really just more pieces of your project, and they should be covered by automated tests along with the rest of it.

In an enterprise setting (or an open-source project with funding), we would have a Jenkins server for each OS that we support, and building and testing packages would be part of our Continuous Integration routine. Unfortunately, this is not an option for most open-source projects.

In this article, I present a scheme (really, a bunch of Make files and a certain directory structure) that can test these packaging scripts for an arbitrary number of different (Unix) OSes on your own dev box. I came up with it while working on making packages for my project Jobber.

An important benefit of this scheme is that it can easily incorporate any automated system tests you may have, making it very easy for you to ensure that your program works on all the OSes you claim to support.

I have made a toy project that uses this system: polly, named after a cat I had who was happy to pack up and travel with me regularly. I'll use it as an example, showing how this scheme can be used to add support for CentOS 7 and Debian 9.

Packaging: Out of Scope

This article will not go into the details of how to make packages for different OSes. However, the toy project does provide a good starting point if you need to make RPMs or Debian packages for non-daemon programs. If you'd like an example of how to do it for a daemon, take a look at Jobber.

Prereqs

Here are the tools you'll need:

• GNU Make
• Vagrant
• VirtualBox

VirtualBox is open-source virtualization software.

Vagrant is the key to this scheme. It is a tool that makes it easy to automate creation, booting, shutdown, etc. of VMs. It's like Docker for VMs.

The toy project is written in Go, but you don't need the Go compiler on your system, as we'll do all the compilation on VMs.

What It Does

To get started, please clone polly and then check out tag initial.

$ git clone https://github.com/dshearer/polly.git
$ cd polly
$ git checkout initial

This wonderful Go project initially looks like this:

|- src/github.com/dshearer/polly
    |- main.go
    |- meow.go
    |- meow_test.go

If you have Go installed, you can play around with it:

$ go test
PASS
ok      .../polly   0.006s
$ go build
$ ./polly
meow! meow! meow! meow! meow!

Let's now take a look at how our scheme adds support for CentOS 7 and Debian 9. Please check out the tip of master (git checkout master).

The project now looks like this:

|- src/github.com/dshearer/polly
    |- Makefile
    |- main.go
    |- meow.go
    |- meow_test.go
    |- packaging/
        |- Makefile
        |- centos_7/
            |- Makefile
            |- Vagrantfile
            |- polly.spec
            |- sources.mk
        |- debian_9/
            |- Makefile
            |- Vagrantfile
            |- debian-pkg/
                ...
            |- sources.mk
        |- head.mk
        |- sources.mk
        |- tail.mk
    |- system_test/
        |- meow_test.sh
    |- sources.mk

Yes, there's a lot of new crap, but making Linux packages isn't exactly simple. Most of the new files are in the directory packaging, in which we have one subdirectory for each of the OSes we wish to support --- centos_7 and debian_9. packaging/centos_7/polly.spec is our RPM spec file that we'll use to make the CentOS 7 package, and packaging/debian_9/debian-pkg contains all the standard files needed for making a Debian package.

We also have a new file at system_test/meow_test.sh. This script contains any system tests that should be done on the program after it is installed.

So what does this give us? If you have installed Vagrant and VirtualBox, try this:

$ make -C packaging -j test-vm

(The "-j" option causes this to be done for each OS in parallel. Occasionally, I have seen this brick the VMs, which will cause the command to hang for a while. If this happens to you, try the command without "-j".)

When this command is done, you will find a shiny new RPM at packaging/results/centos_9/polly-1.0-1.el7.centos.x86_64.rpm and a shiny new Debian package at packaging/results/debian_9/polly_1.0-1_amd64.deb. Also, those packages will have been tested to ensure that they install polly correctly, and polly will have been tested to ensure it works on each of those OSes, using system_test/meow_test.sh. You can see the results of the tests thus:

$ tail results/centos_7/test-vm.log
Installed:
  polly.x86_64 0:1.0-1.el7.centos                                               

Complete!
# run test
vagrant ssh --no-tty -c 'make -C polly-1.0/packaging/centos_7 test-local'
make: Entering directory `/home/vagrant/polly-1.0/packaging/centos_7'
"/home/vagrant/polly-1.0/system_test/meow_test.sh"
PASS
make: Leaving directory `/home/vagrant/polly-1.0/packaging/centos_7'

$ tail results/debian_9/test-vm.log 
(Reading database ... 41645 files and directories currently installed.)
Preparing to unpack polly_1.0-1_amd64.deb ...
Unpacking polly (1.0-1) ...
Setting up polly (1.0-1) ...
# run test
vagrant ssh --no-tty -c 'make -C polly-1.0/packaging/debian_9 test-local'
make: Entering directory '/home/vagrant/polly-1.0/packaging/debian_9'
"/home/vagrant/polly-1.0/system_test/meow_test.sh"
PASS
make: Leaving directory '/home/vagrant/polly-1.0/packaging/debian_9'

How It Does It (Overview)

This whole process is orchestrated by Make files. I know Make isn't used as much anymore, but it really does work well. Moreover, it makes it much easier to build packages if your project can be built and installed with Make.

Making a package on both CentOS and Debian involves several OS-specific steps and OS-specific tools (namely, rpmbuild for CentOS and dpkg-buildpackage for Debian). We of course need to automate those steps. We also need to automate the steps that will be executed on our host machine --- for example, creating and starting the VMs. We might expect to be able to break our automation code into the following mutually exclusive categories:

• Automation code that is to be run on the host
• Automation code that is to be run on a CentOS 7 VM
• Automation code that is to be run on a Debian 9 VM

But it turns out that there's some overlap --- specifically, we need to run some of the code from the first category on the VMs. So our solution takes this approach:

1. Make one system of Make files that does everything we need for every platform --- builds the program, builds the packages, runs unit tests, etc. --- ignoring the fact that not all these commands can actually be run on the same OS
2. Add logic that "magically" switches from the host to, say, the Debian VM, and then resumes execution on the VM

How It Does It (Details)

At the root of the project is the main Make file --- Makefile. Its important targets are

• build: build the program (actually, it calls go install)
• install: install the program to the appropriate place (for example, /usr/local/bin)
• check: run unit tests
• dist: make a source tarball

(The dist target is the reason for all those sources.mk files: those files list the source files in their respective directories, and Makefile imports them all to make the final list of all source files to be included in the tarball.)

Importantly, the main Make file does not concern itself with making packages or any other OS-specific activities. That stuff is covered by packaging/centos_7/Makefile and packaging/debian_9/Makefile. Both of these contain the following targets:

• pkg-local: build the OS-specific package (assuming we are on a VM)
• test-local: Run system_test/meow_test.sh (assuming we are on a VM)
• pkg-vm: "Magically" run pkg-local on a VM with the appropriate OS (assuming we are on the host)
• test-vm: Run pkg-vm, then install the package on a VM with the appropriate OS, and finally "magically" run test-local on the VM (assuming we are on the host)

I want to be clear about something. packaging/centos_7/Makefile and packaging/debian_9/Makefile contain test-local, pkg-vm, and test-vm, but these targets are defined in packaging/tail.mk, which those two Make files import. In general, packaging/<some_os>/Makefile should define only OS-specific stuff.

For your convenience, here is the only target in packaging/debian_9/Makefile:

.PHONY : pkg-local
pkg-local : ${WORK_DIR}/${SRC_TARBALL}
    cp "${WORK_DIR}/${SRC_TARBALL}" \
        "${SRC_ROOT}/../polly_${VERSION}.orig.tar.gz"
    cp -R debian-pkg "${SRC_ROOT}/debian"
    cd "${SRC_ROOT}" && dpkg-buildpackage -us -uc
    mkdir -p "${DESTDIR}/"
    mv "${SRC_ROOT}"/../*.deb "${DESTDIR}/"

Lastly, packaging/Makefile also has targets pkg-vm and test-vm, but they just recursively call the same targets in each of the OS-specific subdirectories' Make files.

The "Magic" Parts

The implementation of the "magic" parts is actually quite straightforward. When make -C packaging/<some_os> pkg-vm is called, the Make file

1. Uses the main Make file's dist target to make a source tarball
2. Makes (or starts) a VM with the needed OS
3. Copies the tarball to the VM
4. Expands the tarball on the VM
5. Runs make -C packaging/<some_os> pkg-local on the VM

test-vm calls test-local on the VM in a similar way.

Vagrant comes into play in steps 2-5. packaging/centos_7/Vagrantfile and packaging/debian_9/Vagrantfile specify the VM image we wish to start with and the steps needed to set it up. For your convenience, here's the CentOS one:

Vagrant.configure("2") do |config|
    config.vm.box = "centos/7"

    config.vm.network :forwarded_port, guest: 22, host: 2223, id: "ssh"

    config.vm.provision "shell", inline: <<-SHELL
        yum install -y epel-release rpm-build
        yum install -y golang
    SHELL
end

When the Make file uses the vagrant command to create, start, etc. the VM, Vagrant looks to Vagrantfile for the details.

When you do all this for the first time, it takes a while for Vagrant to download the VM images and create the VMs. Happily, Vagrant will cache the images so that subsequent runs will not download them again.

In addition, the Make file takes a snapshot of each VM just after it is made. If you run make -C packaging/<some_os> pkg-vm or make -C packaging/<some_os> test-vm multiple times, they will reuse the VM, and any files from previous runs will still be on the VM. To revert to the pristine snapshot, do make -C packaging/<some_os> clean before calling the other targets.

The Next Step

I mentioned in the intro that this scheme naturally supports running system tests on each of these OSes. In polly, system_test/meow_test.sh is a tiny toy system test for a tiny toy program. In a real project, this opportunity to make system testing a part of your development process should not be ignored.

Jobber has a good example, containing tests using Robot Framework of every major feature. When I'm working on Jobber, I just need to do make -C packaging -j test-vm and watch the tests run on every supported OS, in parallel. When they are done, there will be a beautiful Robot test report for each OS waiting for me in packaging/results.

Conclusion

Supporting multiple operating systems is hard, especially if you don't have infrastructure that automates away a lot of the tedious parts. The scheme presented in this article provides such infrastructure, and I hope it has some ideas that would be useful in your own projects.