DEV Community: Ervin Barta

From Zero to Encyrpted Secrets in 2 Minutes with SOPS and GPG

Ervin Barta — Thu, 10 Dec 2020 20:37:43 +0000

You probably heard about mozilla/sops, but even if the readme is amazingly detailed, a from-scratch example is always nice to have.

sops, in a nutshell, bridges the gap between various key management services (PGP, AWS KMS, GCP KMS, Azure Key Vault) and you.
This post will attempt to get you on your feet as fast as possible, in 3 simple steps: from "I have no idea what to do with my hands" to "No way it's that easy!".

Install the dependencies:

$ brew install sops gnupg

And run these 3-ish commands to convince yourself:

# Clone the example repository
$ git clone https://github.com/ervinb/sops-gpg-example.git
$ cd sops-gpg-example

# Import the encryption key
## The path is Keybase specific and it will work on any platform - no need to use your local filesystem path
$ gpg --import <(keybase fs read /keybase/team/sopsgpgexample/pgp/key.asc)
## Or if you don't have Keybase set up yet
$ gpg --import <(curl -L https://gist.githubusercontent.com/ervinb/288c44a45cf2614a0684bea333b3aa36/raw/sops-gpg-example.asc)

# Decrypt and open the file
$ sops secrets/mysecrets.dev.enc.yaml

Your day-to-day interaction with this would be only the last line.

gpg --import has to be executed only once, after which the key will be part of the local keychain (persists reboots as well).
That's literally all there is to it, after following the below steps.

Do it yourself

Start the stopwatch - we have 2 minutes.

Generate a PGP key

   $  gpg --batch --generate-key <<EOF
     %no-protection
     Key-Type: default
     Subkey-Type: default
     Name-Real: Foo Bar
     Expire-Date: 0
   EOF

The key is created without a passphrase because of the %no-protection option. Otherwise a Passphrase: <pass> would be required.

Create a sops configuration file with the key's fingeprint. This is the ✨ magic ✨ ingredient, which makes the onboarding so frictionless.

   $ gpg --list-keys
   pub   rsa2048 2020-12-06 [SC]
         7E6DC556C66C43D928A95EA3715A56B718EAF0B6
   uid           [ultimate] Foo Bar
   sub   rsa2048 2020-12-06 [E]

   $ cat .sops.yaml
   creation_rules:
     - path_regex: secrets/.*\.dev\.enc\.yaml$
       pgp: 7E6DC556C66C43D928A95EA3715A56B718EAF0B6

This is also perfect if you want more control over the secrets, like using different keys for different environments.
For example secrets/*.dev.enc.yaml could use one key, and secrets/*.prod.enc.yaml another one. More details on this here.

Use sops to edit and create new secrets

$ sops secrets/mysecrets.dev.enc.yaml

Then it just a question of distributing the keys to the right people and/or environment.

Which brings us to Keybase.

Note for Linux users

I've found that both on Fedora and Ubuntu, for whatever reason, creating a new file with sops throws the following cryptic error:

$ sops secrets/new.dev.enc.yaml
<save the file in the editor>
File has not changed, exiting.

The solution is to create the file first and encrypt it in-place afterwards:

$ vi secrets/new.dev.enc.yaml
$ sops -i -e secrets/new.dev.enc.yaml

Distributing the key to firends and family

To extract the PGP key from your local keychain, use:

$ gpg --list-keys
-------------------------------
pub   rsa2048 2020-12-06 [SC]
      7E6DC556C66C43D928A95EA3715A56B718EAF0B6
uid           [ultimate] Foo Bar
sub   rsa2048 2020-12-06 [E]

$ gpg --armor --export-secret-keys 7E6DC556C66C43D928A95EA3715A56B718EAF0B6 > key.asc

--armor makes it so that the output is ASCII (.asc) formatted, and not in binary (default).

One of the most seamless ways to distribute keys and other sensitive files is Keybase. It has a low barrier of entry, and you can control the granularity of access with "teams".
It also integrates nicely with the filesystem.

Install Keybase

   $ brew install keybase

Create an account
Store the secret key under a team's folder

After that, you grab the universal path and import the key to anywhere with gpg installed.

Use it in your applications

To use the decrypted values in your application, you can just add a line to your setup scripts to run:

sops -d secrets/mysecret.dev.enc.yaml > configuration.yaml

(make sure to add the decrypted files to .gitignore)

For Terraform projects use terraform-sops, and if you're into Terragrunt,
it has a built-in sops_decrypt_file function.

You will be running sops only to create or edit secrets, otherwise, it will be invisible (and incredible).

Casually removing root files

Ervin Barta — Tue, 28 Apr 2020 20:36:37 +0000

You're walking at $HOME, minding your own business.

$ whoami
> user

$ pwd
> /home/user

But something is bothering your feet. It's like if a little rock has fallen into your shoe. You take it off, to see what's going on.

$ ls -lah ./left-shoe
---------- 1 root root 4 May 30 13:20 little-rock

That's odd. It's there, but it doesn't seem to be yours. It's left there by root, the Rock Tamer, and only he can decide its fate.

# bash -c "echo 'You stay here' > /home/user/left-shoe/little-rock"
# chmod 0000 /home/user/left-shoe/little-rock

You reach into your pocket for your phone, to speed dial him with sudo. Suddenly, you feel powerful (from watching Gladiator last night), and decide to put back the phone, and try your luck.

$ rm -f ./left-shoe/little-rock
$ ls -lah ./left-shoe/little-rock
ls: cannot access little-rock: No such file or directory

You look down at your shaking hands, trying to figure out if this is the real world. It is. You did it. Without the Rock Tamer. But how?

The little rock in your shoe had absolutely no idea what's coming. As seen from it's incarnation, nobody had any permissions on it (--- --- ---). No reads, no writes, no throwing by anyone (owner, group, others).

The catch

What happened is, the Rock Tamer forgot that you are even more powerful
than him, when you're at $HOME. Let's see why.

To be able to do anything with a file, the first step is to look it up in its
directory. Listing a directory's contents is controlled by the execute flag. If
a user has execute permissions on a directory, he can see what's inside it. Also,
the execute flag on the directory gives access to its files' inodes, which is
crucial in this context, as the removal process unlinks the file.

Next, the removing part. Renaming or removing a file doesn't involve the write() system call. Practically, we don't need any permissions to remove the file, nor do we care about its owner. The only requirement is to have write permissions on the parent directory (and the execute flag on the parent directory).

The $HOME directory naturally fulfills both of these requirements from the user's perspective.

The contra-catch

If the Rock Tamer, really didn't want anyone to mess around with his rocks, he would've done:

# chattr +i /home/user/left-shoe/little-rock

This operation makes the file immutable, which among other things, prevents its removal. Excerpt from the man page:

A file with the 'i' attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file and no data can be written to the file. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.

Moonwalks away.

Fixing Fedora black screen on boot (AMDGPU)

Ervin Barta — Tue, 28 Apr 2020 20:28:09 +0000

Having your OS hang on you with a black screen, is not the best start of a day.
The main suspects are usually a failing mount in /etc/fstab, the GPU driver, a kernel upgrade or all three.
In my case, the kernel was upgraded and it wasn't booting. Starting the previous one did work however.

To quickly weed out the GPU, you can try booting with nomodeset. While on the GRUB screen, press 'e' (for edit) and add nomodeset to the boot options.
This disables KMS (Kernel mode setting), which moved the loading of the GPU driver from user space to kernel space. This is why it's stuck at boot when the driver doesn't load properly.

Having nomodeset in the boot options will prevent the amdgpu kernel module from being loaded and hopefully you can boot into the OS to find the root cause.

By being in the OS now (rescue mode works too), we can investigate what went wrong, by looking into the previous boot log.

$ sudo journalctl --system --list-boot
...
-12 db5d180a75ea46b98bd5abfd6ddf2de3
-11 36ba4e53a6864702bdc72b243fb64a80
-10 964a1285dd5545b699bae81282077fa3 # << offending boot!
-9 932f400f4d1c4078b8c972cb841a1c57
-8 8d0abb5acb284d7d90b7ff5814c63389
-7 444fe6eb8ebe43e98373523414a9be05
...

# show the previous boot by providing its ID
$ sudo journalctl --boot 964a1285dd5545b699bae81282077fa3
...
kernel: amdgpu 0000:03:00: Direct firmware load for amdgpu/navi10_gpu_info.bin failed with error -2
kernel: amdgpu 0000:03:00: Failed to load gpu_info firmware "amdgpu/navi10_gpu_info.bin"
kernel: amdgpu 0000:03:00: Fatal error during GPU init
kernel: [drm] amdgpu: finishing device.
kernel: ------------[ cut here  ]------------
kernel: sysfs group 'fw_version' not found for kobject '0000:03:00.0'
kernel: WARNING: CPU: 1 PID: 422 at fs/sysfs/group.c:278 sysfs_remove_group+0x74/0x80
kernel: Modules linked in: amdgpu(+) amd_iommu_v2 gpu_sched i2c_algo_bit ttm crc32c_intel drm_kms_helper e1000e(+) n>
kernel: CPU: 1 PID: 422 Comm: systemd-udevd Not tainted 5.5.10-100.fc30.x86_64 #1
kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z270 Extreme4, BIOS P1.20 11/03/2016
kernel: RIP: 0010:sysfs_remove_group+0x74/0x80
kernel: Code: ff 5b 48 89 ef 5d 41 5c e9 39 be ff ff 48 89 ef e8 c1 b9 ff ff eb cc 49 8b 14 24 48 8b 33 48 c7 c7 e8 >
kernel: RSP: 0018:ffffb941003d7a20 EFLAGS: 00010282
kernel: RAX: 0000000000000000 RBX: ffffffffc0884a00 RCX: 0000000000000007
kernel: RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff91f15ec99cc0
kernel: RBP: 0000000000000000 R08: 00000000000003c9 R09: 0000000000000003
kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff91f1599020b0
kernel: R13: ffff91f150b34d98 R14: ffff91f159da5bc0 R15: 0000000000000000
kernel: FS:  00007f63a53f9940(0000) GS:ffff91f15ec80000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 000055c153964000 CR3: 0000000452912004 CR4: 00000000003606e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: Call Trace:
kernel:  amdgpu_device_fini+0x43d/0x471 [amdgpu]
kernel:  amdgpu_driver_unload_kms+0x4a/0x90 [amdgpu]
kernel:  amdgpu_driver_load_kms.cold+0x39/0x5b [amdgpu]
...

Zooming in:

Direct firmware load for amdgpu/navi10_gpu_info.bin failed with error -2

The AMDGPU is not loading after the kernel upgrade.

Naturally, the first step is to see if the amdgpu/navi10_gpu_info.bin file is there. In my case it was there, but if it's missing for you, download it.

As you probably already now by all the purple links in your Google search results, this error can be caused by almost anything. However, before dwelling into re-installing the driver and whatnot, give this simple solution a go.

Solution

After digging around a lot and searching for everything with amd in its name,
I've found that there are some amdgpu-pro related files in the /etc/dracut.conf.d/ directory.

Dracut is generating the initial ramdisk which boots the system (more on this), so it looked fishy. One of the files had the previous kernel version in its name. The natural instinct in this case is remove it into oblivion.

After restarting, sure enough, the new kernel was able to boot and the amdgpu firmware loaded properly. At one point I did install the amdgpu-pro driver, but removed it soon after. amdgpu-uninstall didn't do a proper job in cleaning up after itself.

After the next kernel update, this happened again. The amdgpu-pro driver was long removed but there were some remnants still.

Putting together kernel upgrade + new device related file appearing results in DKMS. DKMS rebuilds third-party kernel modules with each new version:

Dynamic Kernel Module Support (DKMS) is a program/framework that enables generating Linux kernel modules whose sources generally reside outside the kernel source tree. The concept is to have DKMS modules automatically rebuilt when a new kernel is installed.

This means that a user does not have to wait for a company, project, or package maintainer to release a new version of the module.

Checking DKMS status confirms this:

$ dkms status
amdgpu-pro, 16.50-362463.el7: added

Searching the installed packages revealed:

$ dnf list installed | grep amd
amdgpu-pro-dkms.noarch                             16.50-362463.el7                     @amdgpu-pro-local

$ sudo dnf remove amdgpu-pro-dkms.noarch

Uninstalling the amdgpu-pro-dkms.noarch package will not remove the generated files, it has to be don manually:

$ sudo rm -rf /etc/dracut.conf.d/amdgpu*

Lastly, the current initramfs is regenerated to exclude the old module.

$ sudo dracut --force

From this point on, every new kernel worked out of the box.