Avoiding Ansible `apt_key` on Debian

Originally published on peateasea.de.

The apt_key Ansible module has been deprecated and there isn’t a drop-in replacement. There is a workaround available which achieves the same result but requires more steps. This post describes the issue and presents my riff on the recommended solution.

This is a retropost: it describes a problem I had at my previous job while upgrading servers from Debian buster to bullseye. Although the information presented here is a bit outdated, I hope it helps someone who finds themselves in a similar situation.

---

Ansible is great in the way that it encapsulates the details of common sysadmin tasks in well-maintained modules. A change to how Debian handles GPG keys for third-party APT repositories means that we can no longer use the apt_key module; we now have to do more manual work to achieve the same result. This is my variation on the recommended workaround.

If you’ve upgraded from Debian buster to bullseye, and you use third-party APT repositories for some software packages, you’ll have probably noticed a warning like this when upgrading them:

W: https://apt.releases.hashicorp.com/dists/buster/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

What this means is that one can’t use the /etc/apt/trusted.gpg file as a global keyring for third-party packages anymore. Also, the apt-key command is now deprecated with bullseye being the last Debian version to use it.

We can get more information about the deprecation by looking at the DEPRECATION section of the apt-key(8) man page:

If your existing use of apt-key add looks like this:

wget -qO- https://myrepo.example/myrepo.asc | sudo apt-key add -

Then you can directly replace this with (though note the recommendation below):

wget -qO- https://myrepo.example/myrepo.asc | sudo tee /etc/apt/trusted.gpg.d/myrepo.asc

Make sure to use the “asc” extension for ASCII armored keys and the “gpg” extension for the binary OpenPGP format (also known as “GPG key public ring”). The binary OpenPGP format works for all apt versions, while the ASCII armored format works for apt version >= 1.4.

This is all well and good, but as Ansible users we have to manage the keyring files ourselves now and we can’t use the apt_key module anymore. And it did its job so well! Cue “sysadmin-sad-face”. 🙁

Oh well, let’s rock on and find a solution.

Cobbling together a workable solution

From the notes section of the Ansible apt_keymodule we have:

The apt-key command has been deprecated and suggests to ‘manage keyring files in trusted.gpg.d instead’.

So each third-party repository has to have its own keyring file in /etc/apt/trusted.gpg.d. Good, that’s one piece of the puzzle. How should we do that as an Ansible task now? Fortunately, the Ansible apt_key module documentation has a suggestion:

- name: One way to avoid apt_key once it is removed from your distro, armored keys should use .asc extension, binary should use .gpg
  block:
    - name: somerepo |no apt key
      ansible.builtin.get_url:
        url: https://download.example.com/linux/ubuntu/gpg
        dest: /etc/apt/trusted.gpg.d/somerepo.asc

    - name: somerepo | apt source
      ansible.builtin.apt_repository:
        repo: "deb [arch=amd64 signed-by=/etc/apt/trusted.gpg.d/myrepo.asc] https://download.example.com/linux/ubuntu stable"
        state: present

That’s cool! So, the new solution still verifies the keys when downloading and installing them like apt_key did, right? Right?

Unfortunately, that’s not the case and how to do this is left as an exercise for the reader. Can we glean any ideas from other locations to come up with a more complete solution?

Looking around the web, it seems that most advice from projects or businesses providing third-party APT repositories is to perform steps like these:

• download the project’s APT repository key
• verify its fingerprint by checking against a known value (not all sites mention this though)
• convert the key into the binary GPG format via gpg --dearmor
• save the file to /etc/apt/trusted.gpg.d (or /usr/share/keyrings)
• set the signed-by attribute in the APT repository sources configuration to point to the binary keyring file (note that this was implicitly mentioned by the Ansible apt_key workaround recommendation).

I think what worries me the most here is the amount of work to do by hand to import a third-party package’s key. In particular, I’d like to verify the key’s fingerprint automatically as part of an Ansible task (this is something that the apt_key module did). But, as far as I could tell, there aren’t any Ansible modules doing this as an optional, single step.

I did find ansible-gpg-key, but to verify the key with its fingerprint, one has to import the key, and this is something I’d like to avoid. ansible-gpg-key seems to want to install the keys into a central keyring, which I also want to avoid. In other words, I had to come up with my own workaround for key verification.

After staring at this a while I realised that I could download the key to my development machine and check the fingerprint there. Then I could create a checksum for this file which I could use in get_url to check that the file that Ansible downloads is correct. At least then I can be very sure that an Ansible step using get_url downloads a copy of the file that I’ve verified by hand.

Also, because the manual fingerprint verification step is only necessary every few years, this is an acceptable amount of effort. It’s just something to keep in mind when handling new keys in the future.

Plan of attack

The plan of attack for the workaround goes like this:

Initial manual preparation on a development machine

• download the key file
• verify the file’s fingerprint using gpg --show-keys
• if the fingerprint matches, use sha256sum to create a checksum for the get_url module to use

Ansible steps as part of a playbook task

• download the key file with the get_url module and use the checksum generated above
• convert this to the binary GPG key format (an optional, but common, step); this will require the shell module and gpg --dearmor
• put the key file into the right location; we can do this as part of the previous step (use the -o option to gpg --dearmor)
• configure the APT source file appropriately; we can use apt_repository for this

Let’s see this in action.

Action stations!

At my previous job,¹ we installed HashiCorp Vault by using the HashiCorp Linux Repository as a third-party APT repository. The HashiCorp Official Packaging Guide documents how to set this up.

Let’s set up the HashiCorp Linux Repository to illustrate the process I recommend.

Initial preparation

First, we need to get the key. Download it from the upstream source into the /tmp directory on our local system, calling it hashicorp-key.asc:²

$ wget -O /tmp/hashicorp-key.asc https://apt.releases.hashicorp.com/gpg

After downloading the GPG key, we need to check that the fingerprint matches that mentioned on the HashiCorp security page. To do this, we need to use the --show-keys option to the gpg command:

$ gpg --show-keys /tmp/hashicorp-key.asc
pub rsa4096 2023-01-10 [SC] [expires: 2028-01-09]
      798AEC654E5C15428C8E42EEAA16FCBCA621E701
uid HashiCorp Security (HashiCorp Package Signing)
<[e-mail]>
sub rsa4096 2023-01-10 [S] [expires: 2028-01-09]

We then compare the string 798AEC654E5C15428C8E42EEAA16FCBCA621E701 (which appears under the pub part of the --show-keys output) with the fingerprint mentioned on the HashiCorp security page.³

If the fingerprint matches (and in the current example, it does) we generate the SHA256 checksum of the file via the sha256sum program:

$ sha256sum /tmp/hashicorp-key.asc | cut -d' ' -f1
cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029

I’ve used the cut shell command here to extract the checksum value from the output. Otherwise, we’d have the filename appear in the output, which we don’t need.

To use this checksum in an Ansible step, we need to prepend the string sha256: to it so that Ansible knows what kind of checksum to generate. Thus Ansible will be able to correctly compare the downloaded file’s checksum with the expected checksum. In other words, we specify the following string in the relevant Ansible step:

sha256:cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029

With this information verified and available, we can define the Ansible steps to download the key and configure the third-party APT repository. Then it will be possible to install HashiCorp Vault via apt.

Ansible steps

The Ansible steps are simply generalisations of the commands one would enter at the command line. There are three steps:

• download the ASC file and check that its contents are what we expect
• “dearmor” the ASC file (turning it into a binary GPG file)
• configure the HashiCorp Linux APT repository

You’ll notice that the HashiCorp Official Packaging Guide recommends downloading the ASC file and “dearmoring” it in one step. It’s not possible to join these steps when implementing this process in Ansible. This is because the get_url module needs to download the ASC keyring file and verify it before continuing. Thus, we can’t bundle downloading and “dearmoring” into one step. For one-off situations, this is probably fine. But we want to do this automatically and potentially across several hosts, hence it’s best to split downloading and “dearmoring” into their component steps.

Download

To download the upstream GPG file, we use the get_url module.

- name: download the package signing key for the HashiCorp Debian repository
  get_url:
      url: https://apt.releases.hashicorp.com/gpg
      dest: /tmp/hashicorp-archive-keyring.asc
      checksum: "sha256:cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029"

The step specifies the URL of the file we want to download, the checksum to check it with, and that its download destination is in the /tmp/ directory. I’ve put the file in /tmp because we’ll be creating a binary version of it in the next section and hence we don’t need to keep the ASC file.

Dearmor

An optional, yet rather common, step is to “dearmor” the ASC file, converting it into a binary GPG file. One does this via the --dearmor option to the gpg command.⁴ For instance:

$ gpg --dearmor -o /etc/apt/trusted.gpg.d/hashicorp-key.gpg /tmp/hashicorp-key.asc

This command will create the file hashicorp-key.gpg in the /etc/apt/trusted.gpg.d/ directory.

To convert this shell command into an Ansible step, we use the shell module.

- name: dearmor the downloaded signing key file
  shell: gpg --batch --dearmor -o /etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg /tmp/hashicorp-keyring.asc
  args:
    creates: /etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg

Here we’ve told Ansible the full path of the file that this step creates. Thus, Ansible can skip this step if the file already exists the next time it runs the step.

Note that we’ve also used the --batch option to gpg which tells gpg to never ask for any confirmations and to disallow interactive commands. This way the gpg command won’t hang and indefinitely wait for input if something goes wrong when we run the step.

The -o option places the binary GPG output file in /etc/apt/trusted.gpg.d/. Note that this is not the directory suggested by the HashiCorp documentation! I’ve chosen instead to use the Debian standard location for such files. As mentioned in the Debian secure apt docs:

… the keyrings are stored in specific files all located in the /etc/apt/trusted.gpg.d directory

I think that if one is running Debian servers it makes sense to use the standard Debian locations rather than the locations recommended by third party software. This way, any sysadmin familiar with secure apt on Debian will know where to look for the keyring files.

APT repository configuration

To configure the third-party APT repository, we use the apt_repository module:

- name: add HashiCorp Linux repository
  apt_repository:
      repo: deb [arch=amd64, signed-by=/etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com main
      state: present
      filename: hashicorp-linux

This is much the same as what one would normally enter into /etc/apt/sources.list. The main differences to the standard case are:

• the keyring file to verify the upstream repository is specified explicitly via the signed-by attribute, and
• the configuration is placed in its own file via the filename parameter.

There are some subtleties here with the filename parameter: Ansible will implicitly place the file in the /etc/apt/sources.list.d directory and will append the extension .list to the filename. Thus, although we used hashicorp-linux for the filename parameter, the file will actually be called hashicorp-linux.list on the filesystem. In other words, don’t use hashicorp-linux.list for the filename parameter because then the file will be called hashicorp-linux.list.list. That would be silly.

All together

Putting all the steps together, we arrive at this final Ansible configuration:

- name: download the package signing key for the HashiCorp Debian repository
  get_url:
      url: https://apt.releases.hashicorp.com/gpg
      dest: /tmp/hashicorp-archive-keyring.asc
      checksum: "sha256:cafb01beac341bf2a9ba89793e6dd2468110291adfbb6c62ed11a0cde6c09029"

- name: dearmor the downloaded signing key file
  shell: gpg --batch --dearmor -o /etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg /tmp/hashicorp-keyring.asc
  args:
    creates: /etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg

- name: add HashiCorp Linux repository
  apt_repository:
      repo: deb [arch=amd64, signed-by=/etc/apt/trusted.gpg.d/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com main
      state: present
      filename: hashicorp-linux

And that’s it! All set up 😄 Now it’s just a matter of running these steps in an Ansible playbook to roll out the configuration to our servers.

Wrapping up

I would have liked to verify the GPG key automatically by matching its fingerprint with a known value. However, there doesn’t seem to be an Ansible module able to do this. I did consider writing one but decided not to dive down that particular rabbit hole.

Since such keys change only every few years, the effort to download a key to a development system, verify its fingerprint, and then create a checksum of the file (so that get_url can verify the file’s contents) is sufficient for our purposes here.

I would have liked things to be more automated, but you can’t always get what you want. Maybe in the future, there’ll be an Ansible module which automatically downloads and verifies a GPG key according to a given fingerprint value. Who knows? Time will tell.

Addendum: handling Grafana

If you want to use Grafana and also want to install it via apt, then you’ll want to use the get_url and apt_repository pattern discussed here.

Unfortunately, it’s hard to find the fingerprint for the Grafana repository’s GPG key. For instance, the Grafana Debian installation docs don’t mention the keys’ fingerprint. In investigating this issue myself, I managed to stumble across an entry in the Grafana community forum mentioning the Grafana package repository. The package repository web page mentions the fingerprint, so they do publish the fingerprint, just not anywhere obvious.

Note that Grafana refers to its public GPG file with the extension .key. Thus, if you want to be consistent with the Debian documentation, you should use .asc because this is the extension for an ASCII armored key.

Dearmoring the key, adding it to the system, and configuring the third-party APT repository is then the same as in with the HashiCorp Linux APT repository discussed in this post.

1. I’m now a freelance software developer doing Python/Perl backend development and maintenance work. Contact me at [e-mail] and let’s discuss how I can help solve your business’ hairiest problems. ↩

2. Otherwise, its name would be gpg. This name isn’t helpful because it’s not clear where it came from and why. ↩

3. At the time of writing, this was under the “Linux package checksum verification” section. ↩

4. As a native speaker of a descendant variant of British English, I find it nice that the GPG developers also implemented this option as --dearmour. ↩