Introduction
When we are using Ansible, we often need passwords or tokens or any secrets that we don't want to store as plain text and definitely don't want to commit to a git repository. At least not as plain text. Personally, I don't like to commit secrets even if they are encrypted, unless it is required.
I used Ansible Vault for years, and I was suggested to replace it with GPG since we used that too anyway, but somehow I just didn't like it so much, so I kept using Ansible Vault. My problem with Ansible Vault was that when I encrypted a yaml file which contained secrets, the whole file was encrypted including parameter names. Of course there was a solution for that. Creating an unencrypted file without the values and another which was completely encrypted, so you could manage two files instead of one. Or you could encrypt individual variables manually.
Maybe about a year ago, I started to use Secrets Operations (SOPS) and it made everything easier. Not to mention that I can use SOPS without Ansible. In this post I will use SOPS to finally encrypt the sudo password (aka become pass) for the user on the remote server and I will use the Nix package manager to install the required to packages, sops and age.
If you want to be notified about new videos, you can subscribe to my YouTube channel: https://www.youtube.com/@akos.takacs
Table of contents
- Before you begin
- Why Nix again?
- Create a new config file
- Generate the age key pair
- Create a wrapper script for sops and detect the public key automatically
- Encrypt a file
- Edit an already encrypted file
- Use the secret in the inventory file
- Use a dedicated user for Ansible
- Run Ansible in a Nix shell
- Test if Ansible could read the secret
- Don't commit secrets to a git repository
- What more you should know
- Conclusion
Before you begin
Requirements
- The project requires Nix which we discussed in Install Ansible 8 on Ubuntu 20.04 LTS using Nix
- You will also need an Ubuntu remote server. I recommend an Ubuntu 22.04 virtual machine.
Download the already written code of the previous episode
If you started the tutorial with this episode, clone the project from GitHub:
git clone https://github.com/rimelek/homelab.git
cd homelab
If you cloned the project now, or you want to make sure you are using the exact same code I did, switch to the previous episode in a new branch
git checkout -b tutorial.episode.6b tutorial.episode.6
Have the inventory file
Copy the inventory template
cp inventory-example.yml inventory.yml
- Change
ansible_host
to the IP address of your Ubuntu server that you use for this tutorial, - and change
ansible_user
to the username on the remote server that Ansible can use to log in. - If you still don't have an SSH private key, read the Generate an SSH key part of Ansible playbook and SSH keys
- If you want to run the playbook called
playbook-lxd-install.yml
, you will need to configure a physical or virtual disk which I wrote about in The simplest way to install LXD using Ansible. If you don't have a usable physical disk, Look fortruncate -s 50G <PATH>/lxd-default.img
to create a virtual disk.
Activate the Python virtual environment
How you activate the virtual environment, depends on how you created it. In the episode of The first Ansible playbook describes the way to create and activate the virtual environment using the "venv" Python module and in the episode of The first Ansible role we created helper scripts as well, so if you haven't created it yet, you can create the environment by running
./create-nix-env.sh venv
Optionally start an ssh agent:
ssh-agent $SHELL
and activate the environment with
source homelab-env.sh
Why Nix again?
There are multiple ways which you can find in the git repository of sops to install it, but I will show you one that isn't there. The reason is that I want to support this project on Linux and macOS and I also want you to use the same version that I use. So if I follow the guide on GitHub, I need to either download a specific binary from GitHub or use different package managers on different platforms. Both would require automatically detecting the platform. Since downloading binaries would also mean that I would need to manually verify checksums and install dependencies and I like to use general solutions, I rather choose Nix. We already use Nix to create a Python virtual environment, so why not?
Create a new config file
SOPS supports multiple tools for encryption like PGP or age, and we will use age. The first step is to open a Nix shell with pre-installed age
so we can use the age-keygen
command to generate a keypair. Eventually, we need to run a command like this:
age-keygen -o age/private-key
SOPS will look for this file in specific folder which is different on macOS and on Linux, so we will override that, and regardless of the OS you are using, you can run the same commands, and I can use the same project on macOS and in a Linux virtual machine where I mount the project from the host. In order to use the same key in the terminal and in Ansible, we will need two environment variables:
SOPS_AGE_KEY_FILE
ANSIBLE_SOPS_AGE_KEYFILE
Since I will have multiple scripts using the same config, I will create a config file which will be a simple shell script. Let's save it in the project root and call it config.sh
.
# HOMELAB variables
export HOMELAB_VAR_DIR="${HOMELAB_PROJECT_ROOT:-.}/var"
# SOPS variables
export SOPS_AGE_KEY_FILE="$HOMELAB_VAR_DIR/age/key"
# Ansible variables
export ANSIBLE_SOPS_AGE_KEYFILE="$SOPS_AGE_KEY_FILE"
We have a "HOMELAB variables" section for general variables, an "SOPS variables" section for variables that are used by the "sops" executable directly and an "Ansible variables" section for Ansible of course. We also use the HOMELAB_PROJECT_ROOT
variable, which is just a simple dot by default, but can be set in each script. We do this only because this way we don't have to support sourcing the config file in different shells and use different ways to determine the project root.
Generate the age key pair
Now we need a script that runs age-keygen
and stores the private key file based on the config file. Let's call it sops-keygen.sh
#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p age
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz
current_dir="$(cd "$(dirname "$0")" && pwd)"
export HOMELAB_PROJECT_ROOT="$current_dir"
source "$HOMELAB_PROJECT_ROOT/config.sh"
parent_dir="$(dirname "$SOPS_AGE_KEY_FILE")"
if [[ ! -e "$parent_dir" ]]; then
mkdir -p "$parent_dir"
fi
if [[ ! -e "$SOPS_AGE_KEY_FILE" ]]; then
age-keygen -o "$SOPS_AGE_KEY_FILE"
fi
echo "The private key file is at $SOPS_AGE_KEY_FILE"
It should be executable of course:
chmod +x sops-keygen.sh
And when you execute it,
./sops-keygen.sh
the script will generate the private key, save it and show the location. If you are wondering where the public key is, it is in the same file.
source config.sh
cat $SOPS_AGE_KEY_FILE
The output will be something like this:
# created: 2023-11-12T11:36:39+01:00
# public key: age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e
AGE-SECRET-KEY-1YCKA5MJ44V2YDA8RZ9MKV0YWTDLG8MXFSDEFT9AQ76GQ7005JFQQN0WFN4
Create a wrapper script for sops and detect the public key automatically
You can copy the public key manually when you need it, but sometimes you want to get it in a script like now. We need a wrapper script for the sops
command. In this script we can get the public key and load it into a variable. We don't set this variable in the config file, because we need it only when we run sops. Even then, we would need it only when we want to encrypt a file, not when we decrypt it. I won't overcomplicate this script even more, so I will let the script read the public key every time. The filename will be sops.sh
.
#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p sops
#! nix-shell -p age
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz
current_dir="$(cd "$(dirname "$0")" && pwd)"
export HOMELAB_PROJECT_ROOT="$current_dir"
source "$HOMELAB_PROJECT_ROOT/config.sh"
export SOPS_AGE_RECIPIENTS="$(age-keygen -y < $SOPS_AGE_KEY_FILE)"
sops "$@"
And of course we make it executable:
chmod +x ./sops.sh
You can test it with a simple command that shows its version number:
./sops.sh --version
Encrypt a file
What we need now is a file to encrypt. Let's create secrets.plain.yml
in the project root.
become_pass:
And encrypt it:
./sops.sh --encrypt --output secrets.yml secrets.plain.yml
We will get a secrets.yml
file like this:
become_pass: null
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNZjVEOTJJZWkwQytML1F6
QVFJNHNmcVhHbThMZWI2SGJWalArd2c0bkVvCng3QjgvQms4SE9LT1o3Z21DN1Yw
MVBWcFdnNVV3UGVKNFB4YS85UU9ScWsKLS0tIFNOTkdodkt6aEZMT1pvdGxxVjNm
M29mL2JmMDI4N1FmbUMxVmpsRW1vMWcKRIxPc0dZv9JcEn2NyZ9OJ6QDerh9VIcw
rvD0Tyvrbzoc32cxUZMEUGH+tCwFi5eQ212Fehw1jlLh/YYmYPYBEA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2023-11-12T20:21:57Z"
mac: ENC[AES256_GCM,data:QsdpRH36FdY3Gq01U/4NvuiXt/OAkxQv+GURksOmlJCpEzyjKDXrqJOk6D/ZiuGrwPn48803MFSvojNtvdrLr0k8+sJ58/Kv9u1vf7/fxbZczvW5ohVhH1bfou6ZgqsJyMLu8yp3EbXukQ8hJe9N159HgqIdCkyycSFwOGko5O4=,iv:0Y4cP2mzq9wedRLxxYSBM4GAS/VvvbohKWGICO+IWw0=,tag:dfoEDyC0736bL+aXHEmAcA==,type:str]
pgp: []
unencrypted_suffix: _unencrypted
version: 3.8.1
The important part for us now is the first line where we can see the become_pass
with a null value. Now I like to edit the values from a terminal and I can simply use
Edit an already encrypted file
./sops.sh secrets.yml
It will open the file in the default text editor like nano
or vim
and show you only the decrypted content without the rest of the YAML keys. My default editor currently is vim, but I'm not afraid to tell you that I usually prefer nano, although I don't care enough to change it. If I want, I can change it just for this specific command:
EDITOR=nano ./sops.sh secrets.yml
I will use the following new value in the encrypted file:
become_pass: HomeLab2023
After saving the file, the first line in the encrypted view will be like this:
become_pass: ENC[AES256_GCM,data:degiXqRIpLEVdTY=,iv:Loyf/XbEcTq6Z9enDq1ccCFzOYurL4kUa1Js9dpQzgo=,tag:xmhhExQREc6wkY373eW5+g==,type:str]
We can remove the plaintext file:
unlink secrets.plain.yml
Use the secret in the inventory file
We need to pass this file somehow to Ansible. We could do it multiple ways, but what I prefer is loading the content
in the inventory file. For that we need a lookup plugin called community.sops.sops
which is the part ot the collection called community.sops
. We can set a variable called sops
and load all the secrets from yaml as an Ansible "dict".
all:
vars:
sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
We also need to use the values somehow like this:
all:
vars:
sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
hosts:
ta-lxlt:
ansible_become_pass: "{{ sops.become_pass }}"
Now as a reminder, this is my current full inventory.yml
file:
all:
vars:
ansible_user: ansible-homelab
config_lxd_zfs_pool_disks:
- /dev/disk/by-id/scsi-1ATA_Samsung_SSD_850_EVO_500GB_S2RBNX0J103301N-part6
sops: "{{ lookup('community.sops.sops', 'secrets.yml') | ansible.builtin.from_yaml }}"
hosts:
ta-lxlt:
ansible_host: 192.168.4.58
ansible_ssh_private_key_file: ~/.ssh/ansible
ansible_become_pass: "{{ sops.become_pass }}"
Use a dedicated user for Ansible
You might have noticed that I changed the value of ansible_user
from ta
to ansible-homelab
. I did it, so I don't need to change my user's password and I can still show you the secrets. It is also useful to have a dedicated user for Ansible when you have a CI/CD pipeline, and you don't want to share your password with others who can read the secrets. I created the user with the following command:
pass="HomeLab2023"
salt="sugar"
useradd \
--shell /bin/bash \
--create-home \
--groups sudo \
--password "$(openssl passwd -6 -salt "$salt" "$pass" )" \
ansible-homelab
If you don't have openssl or just prefer to create a user manually, that's fine too, just make sure you add the user to the sudo
group on Ubuntu, so the user can become root. After all, this is why we store the become pass (sudo password) in a secret.
Run Ansible in a Nix shell
Now there is one script that we already have, but needs to be changed. We used Nix to download sops
, but that means Ansible needs to run in a Nix shell. We had this in our original run.sh
in the project root:
ansible-playbook \
-i inventory.yml \
--ask-become-pass \
"$@"
We don't want that script to force Ansible to ask for the become pass so that line must be removed, and we need the usual shebang lines for Nix and also our config parameters. The new run.sh
will be this:
#!/usr/bin/env nix-shell
#! nix-shell -i bash
#! nix-shell -p sops
#! nix-shell -I https://github.com/NixOS/nixpkgs/archive/refs/tags/23.05.tar.gz
source config.sh
ansible-playbook \
-i inventory.yml \
"$@"
Test if Ansible could read the secret
Now let's run the hello playbook which requires root privileges, but we need to change the original destination path of the hello-world.txt
. Otherwise, Ansible would not need to copy again, and it could work even if the sudo password is incorrect.
./run.sh playbook-hello.yml -e hello_world_dest=/opt/hello-world-$(date +%s).txt
As you can see, we can override a role parameter from terminal. Unless this command failed, our secret worked.
Don't commit secrets to a git repository
We have one more job before we say goodbye. As I stated at the beginning of this post, I prefer not to commit these encrypted secrets either, so I add secrets.yml
to gitignore. The other file I definitely don't want to commit is the private key so let's add the following two lines to gitignore:
/var
/secrets.yml
My full gitignore looks like this now:
/venv
/venv-linux
/inventory.yml
/var
/secrets.yml
# for macOS
/.DS_Store
What more you should know
In this tutorial I used a bit different way than what you can find in documentations. That made the commands easier, but also more complicated at the same time. sops
supports passing the public key as an argument like sops --encrypt --age age1lh5qpf04dq0xcvgg63wf3qha32d8mxfslm97nh0utfl9rv784dts5zpl8e
, but I used the environment variable instead. Why? Because that way you can still have the option to override it from terminal, although this tutorial is not for production systems but for a private home lab, a playground, so you will probably not need to change it or add multiple public keys which means multiple recipients which would support multiple private keys to decrypt the secret file. You can find everything in the documentations and I recommend reading it and playing with sops and age, so you can discover more like how you can use age
with a hardware key like YubiKey to decrypt files. For using YubiKey, you need a plugin called "age-plugin-yubikey".
Conclusion
We can now use secrets, but that is completely optional. If you want to pass test passwords like "abc" or "password" as plaintext on a machine where you have nothing to protect like a virtual machine which you just created for playing with Ansible or test some commands, that's fine. Then you don't need to use the lookup plugin in your inventory file.
I have to note again that this series is for creating a home lab, a local environment to develop and learn and not for a production environment. In a production environment you should always encrypt secrets. Don't forget that SOPS is just one tool so be prepared for other options too when you have to use a keyserver for example in a team.
The final source code of this episode can be found on GitHub:
https://github.com/rimelek/homelab/tree/tutorial.episode.7
README
This project was created to help you build your own home lab where you can test your applications and configurations without breaking your workstation, so you can learn on cheap devices without paying for more expensive cloud services.
The project contains code written for the tutorial, but you can also use parts of it if you refer to this repository.
Tutorial on YouTube in English: https://www.youtube.com/watch?v=K9grKS335Mo&list=PLzMwEMzC_9o7VN1qlfh-avKsgmiU8Jofv
Tutorial on YouTube in Hungarian: https://www.youtube.com/watch?v=dmg7lYsj374&list=PLUHwLCacitP4DU2v_DEHQI0U2tQg0a421
Note: The inventory.yml file is not shared since that depends on the actual environment so it will be different for everyone. If you want to learn more about the inventory file watch the videos on YouTube or read the written version on https://dev.to. Links in the video descriptions on YouTube.
You can also find an example inventory file in the project root. You can copy that and change the content, so you will use your IP…
Top comments (0)