A Tale of Two Build Systems: Launchpad and Copr

Whoa, a blog post! Haven’t written one of those in a while. I’ll make another one to go over what’s happened since my last post, but right now, let’s focus on something very specific: package managers, and the build systems that support them. It’s Ubuntu vs Fedora, RPM vs DEB, apt vs dnf (or yum), Launchpad vs Copr.

Fedora and Copr

Let’s start off at the beginning. I use Fedora on my desktop and personal laptop. It’s great! I like how cutting edge yet stable everything is. But before that I used Ubuntu. And that’s probably the case with a lot of other people. Ubuntu is very popular and easy to get started with, and because of that there’s a lot of guides for how to use it and how to install things. Its popularity has basically grown because of its popularity, in short. There’s so many flavors and distros based on Ubuntu too. Ubuntu and Fedora both have something in common as well: the groups behind them both have a platform that lets you build and host packages for them. Ubuntu has Launchpad, and the PPA, or Personal Package Archive, and Fedora has Copr.

Let’s start chronologically for me. I first used Copr… Four years ago? It’s been a bit, and I’m decently familiar with it. There are a handful of tools you can use to build an RPM package. RPM is the packaging format that Fedora and related distros use. To start, you have to write a spec file. This file can be pretty simple, or very large, but what it’s responsible for is describing what your package is, how to build it, and where all the files to install are located. Here’s an example of one I made a while back.

%define name minsh
%define build_timestamp %{lua: print(os.date("%Y%m%d"))}
%define version 1.9.1
Name: %{name}       
Version: %{version} 
Release: %{build_timestamp}%{?dist}
Summary: A very simple shell    
Source0: https://github.com/dvdmuckle/minsh/archive/master.tar.gz#/%{name}-%{version}-%{release}.tar.gz
License: GPLv3
Packager: David Muckle <[e-mail]>
BuildRequires: gcc make

%description

A very simple shell that supports running commands and output redirection.
%global debug_package %{nil}
%prep
%autosetup -n %{name}-master

%build
make 

%install
mkdir %{buildroot}%{_bindir} -p
install -s minsh.o %{buildroot}%{_bindir}/minsh

%clean
rm -rf %{buildroot}

%files
%{_bindir}/minsh
%doc

%changelog

This is a spec for a small shell in C I wrote in college. I got the actual class project itself done in an afternoon, and had some time to package it up before the thing was due. Thus, this is pretty simple. The spec describes where you can get the code to build this under Source0, what prep work needs to be done to build it, what building it looks like, what installing it looks like, and where the other files are. For this, there’s only one file, the binary. This also describes the tools requires to build the package. Note this doesn’t include any runtime dependencies. There technically are some, but they’re all libraries included in libc which is a library package that’s just about always installed on any system.

In order to use this spec file to build your RPM, there are a few ways. Back then, I used spectool and rpmbuild, and builddep. spectool lets you lint the spec file and get the sources, rpmbuild does the actual building, and builddep lets you make sure all your build dependencies are installed. One of the cool things you can build is something called an SRPM, which is basically this spec file and your sources wrapped into a fancy archive file. It’s an RPM waiting to happen, just add water! Or a build system! Or, you can take another route and use mock, which is very nice as it will build your package in a chroot and manage all your build dependencies for you. Instead of having to remember two tools and the flags needed to do what you want, you can use one tool. And you can build for multiple OSes! So long as they use RPM, that is. This tool is more or less the same as what Copr does, so it’s a great way to make sure your build will work on Copr.

Copr is the first build system I used. You can feed it either a spec or an SRPM, either uploaded directly or cloned from a git repo. It’ll get your source (or use the one in the SRPM you uploaded), get all the dependencies in order, execute your instructions, and upload the resulting package to a repo anyone can add to their system. The most important detail is building an RPM is pretty flexible. Sure, there are some standards, but if you break the rules things usually won’t fail outright. For example, let’s say you want to build a binary written in Golang. It’s 2021, you probably have a go.mod file, right? One that specifies all your dependencies? And a go.sum file that specifies the versions of those dependencies, right? So why manage those dependencies in the spec file as well? With Copr you can run a build with internet access, which allows for you to pop a go mod vendor into your build instructions, so your dependencies are satisfied. This is technically against the packaging “code” for Fedora packages, but, well…

It’s important to note that both Fedora and Debian/Ubuntu have standards for packaging. These standards are more for if your package is going to be published in one of their repositories, but it’s still useful to keep the standards in mind.

One final thing before we move on from Copr is that with Copr, you can specify the versions of Fedora (and friends) as well as the architectures you want your package to be built for with just a check of a checkbox. And you only need one spec written! Generally speaking Fedora packages don’t change names or anything between versions, so this shouldn’t break any of your build and runtime dependencies.

… Okay one last last thing and then we move on to Debian packaging. Spec files also allow you to put dynamic stuff into the file. Notice how the build_timestamp uses some Lua to get the current time? What a pain it would be to specify that manually every time! There’s also probably a macro for that too! There’s a number of packages that Fedora supplies that include macros, which can be useful for building things such as Golang-based packages. (Wow all this talk of Golang, you’d think I had some kind of Golang project I’ve been working on packaging…). These macros can do a whole slew of things including specifying how your package is built and where any binaries are installed. The downside to that is unless you know what precisely the macro does, it can be hard to debug when something goes wrong, and to an outside observer it can be hard to figure out what’s going on.

Ubuntu/Debian and Launchpad

Okay! Now on to Debian packaging. This is going to be a lot of me saying “I don’t know what this is exactly,” because I literally figured this out a day ago. I am going to figure out a lot of that though, and probably publish an addendum post to this, so a lot of my opinions will probably also change, but for now this is a lot of “This is what I had to do to get this working.”

As previously stated, there’s a handful of tools to build a DEB package. Which is a lot compared to the maybe four or so tools for RPM. You can even use dpkg, the package manager for DEB, to build packages! This is largely a bad idea because it literally creates an archive you can install, with nothing else involved, and assumes a lot like that your binary will run on other systems with potentially different library versions. Cool you can do it, though. As an aside, while dpkg is the actual package manager used by Debian based systems, and thus Ubuntu, usually it’s used via apt or apt-get. Debian based systems also have their own version of mock system, called pbuilder… and also cowbuilder??? One uses the other under the hood, I think, but I couldn’t get either of them working, but either way, they both build in a chroot and are in theory a good way to build for multiple distro versions.

For the package I built, I ended up using a combination of debuild and dpkg-buildpackage. Technically speaking they can both generate the same files you need for Launchpad, so I ended up with debuild since it requires fewer command line arguments. debuild also uses dpkg-buildpackage and the like under the hood along with some other linting stuff. To generate all the files to actually build this package, I used dh-make-golang. This is related to dh-make, which basically allows you to provide some things like your package’s name, what kind it is, and some other arguments to generate all of those files you need to build your package. And there’s, uh, more than just one, compared to RPM. At least they all go into the same debian directory…

Let’s start alphabetically. You’ve got your changelog file that has some very specific formatting to do a couple things. It lists your current and past versions for your package, as well as, well, your changelog, and what distro you’re building for. I think the distro part can be templated out so when the time comes to send generate the files to send to Launchpad, you can have one changelog for multiple distros. For now I’m just building for Ubuntu 20.04 since it’s the latest LTS release. The formatting here is so specific there’s a command just for updating the changelog, dch, that will do things like put the date in there correctly, bump the version number, and put in your email.

Your control file will list all of the metadata about your package. What it’s called, the maintainer, any build dependencies. One weird oddity I ran into is the package that enables shell completion in Ubuntu, bash-completion, also provides the files required to implement shell completion when building a package. Which is a little weird, initially I thought the package to allow for this would be some kind of package specific to building packages, much like the macros packages in Fedora.

The last major file (There are many others but they’re things like copyright which includes licensing stuff, and a couple other things I don’t even remember generating.) is the rules file. If you’re familiar with make, this will probably look familiar. It’s a Makefile! And it’s executable, but it gets executed by some other helper programs, so maybe don’t try executing it yourself if you’re new to this. This file is basically where you define how to go from source to binary for your package, and will probably also utilize an existing Makefile you already have. My rules file… does not do that, it uses a dh command with some other Golang-specific flags to build my binary, so I don’t know entirely what it does. What I do know is I had to add the --with=bash-completion flag and then add a spc.bash-completion file with all the shell completion stuff. Remember how I said macros can complicate things and make it hard to figure out what’s happening? Well the shoe’s on the other foot now. I clearly have a lot of learning to do in terms of how Debian packaging works, but what I do want to share are a couple gripes I’ve run into thus far.

Miscellaneous Headaches

At first I ran into some issues with building the files necessary for Launchpad because it didn’t want to allow uncommited files in my repository when creating the source archive. Well, tough cookies, I need to include my vendored Golang libraries, because according to my research Launchpad doesn’t allow for building with internet access, so it can’t go off and get the necessary files for me to build. Copr can do builds with internet access, so while it would be cool for Launchpad to do that, I get why they may want to disallow it. At the time I was using this guide to dh-make-golang in order get it working, which uses a couple other different tools later on in order to do the actual package build. I don’t need to do a full package build, however, I just need to generate the Debian equivalent of a SRPM, so I checked out a couple other tools. Eventually I settled on debuild, which lets me build just my source and other files, most of which are variants on the above files, sign things with my GPG key, and also ignore that I don’t have my build dependencies installed, since those include library packages I don’t need.

This was another pain point. Launchpad has a heavy reliance on GPG keys. You have to upload the public key to Ubuntu’s keyserver, then import it to Launchpad. The whole process can take anywhere from thirty minutes to an hour. It’s slow. Granted we’re talking about Ubuntu infrastructure, infrastructure supporting one of if not the most popular Linux distro on the planet, so I get why it might be slow. Still, it’s hard to not be annoyed. On top of that, once you get the build going, it can take a few minutes after it’s completed for the DEB package to be available in the PPA. With Copr, it’s available nearly immediately, and there’s no need to use GPG keys for authentication. For both systems, keys will be generated for the package repository itself, so the GPG key you use with Launchpad, as far as I can tell, is only use to authenticate you and your files when you use dput to upload to Launchpad. This operation is apparently straight-up FTP. Also, if you screw up the GPG stuff from earlier, you can end up in a position where your Launchpad build fails silently and you can’t reuse that version for a build, and will be forced to increment the build number and rebuild.

Conclusion

That all being said… I did it! I solved for Launchpad! Now I can focus on automating all of this with GitHub actions, something I’m not completely looking forward to. Overall, these two build systems allow you to do some neat stuff and make it very user friendly to install a package that isn’t in the main repositories. The way it gets to there, however, is very different. Debian package builds have a lot of rules and certain ways you have to do things, in addition to having many tools that do the same thing in ever so slightly different ways. Debian packaging also has documentation of mixed quality. The basics are well documented, but trying to do something a little more exotic like using dh-make-golang isn’t as documented as I would have liked. Fedora packaging has a handful of useful tools, with mostly up to date documentation to boot. The vibe I get from these build systems is they tend to reflect the OS you can build for. Fedora is cutting edge while being solid and pretty user friendly, whereas Ubuntu is incredibly well supported but, at least for LTS releases, you may not be getting the latest thing. Again, this is all my opinion as someone new to Launchpad and only decently knowledgeable with RPM packages. I’ll do more research into how to do a lot of these things, but this has been my journey thus far.

The post A Tale of Two Build Systems: Launchpad and Copr appeared first on dvdmuckle.