Manage your secrets in Git with SOPS & GitLab CI 🦊

In this series, we saw how to create and manage our secrets in git and restrict their access to team members.

What if I want do use my secret from my CI for Continuous Deployment? This is exactly what we will see in this article, based on GitLab CI 🦊.

DISCLAIMER: Every CI system should be compatible, you just need to have access to CI secrets and be able to fetch them during execution.

🤖 Creation of the CI identity

To be able to decrypt secrets within Continuous Integration, we should create an identity for our CI. Alice, Bobby and Devon will have a new friend named CI 🤖 !

Devon, in charge of the CI, will create a new key-pair for this new special user. He will use the gpg --full-gen-key command and answer questions like if it was for a real human.

Then, he will extract a backup of the private/public key with the command gpg -o ci.key --armor --export-secret-keys [e-mail]. This key is in a textual format, which is easy to use as an environment variable.

NOTE: This key and the passphrase should be stored securely, in a keypass, vault or something else, because they represent the identity of your CI. By design, it will be able to decrypt all secrets from the repository.

We also need to extract the public key separately to distribute it to every team members. Without it, they won't be able to encrypt a secret and include the CI key in the process.

To do so, Devon will use the command gpg -o ci.public.key --armor --export.

🔑 Import public key of CI

Each team member should import the public key extracted by Devon in the previous step. Here, Booby will use the gpg --import ci.public.key command to perform this.

At the end, he, and every other team members should have in their key list (gpg --list-keys) the public key of CI.

🔏 Update keys of every secret

CI is a new team member, as any other human. The team has to let this new "person" access and decrypt every secret. So we will use the sops updatekeys <secret> on every secret.

This requires a correctly configured .sops.yaml file. For more information about this, you can read the second part of this series.

🦊 GitLab CI Configuration

Now, the CI is ready to be configured. Devon, in charge of this, will import into GitLab secret pane (in Settings > CI/CD > Variables) the content of the ci.key created previously and the passphrase defined for it.

NOTE: The variable Type is defined to File for the KEY. For more information about this, look at the official GitLab documentation about file variable

Then, Devon will define a GitLab CI job able to read the value from the encrypted secret. For our example, this will just do a cat on the standard output. Of course, you can do what you want with those values in your CI pipeline.

We need to take care of the image used in our Continuous Integration. We should be able to access the gpg command. In this example, we will install it.

deploy int:
  image: google/cloud-sdk # <1>
  before_script:
  - apt-get update && apt-get install -y curl gnupg # <2>
  - curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /usr/local/bin/sops # <3>
  - chmod +x /usr/local/bin/sops
  - cat $KEY | gpg --batch --import # <4> 
  - echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp) # <5>
  script: 
  - sops -d int.encrypted.env > int.env # <6>
  - cat int.env

1. Define a job image based required for our deploy phase... it should fit team needs, this is just for example here.
2. We install gpg and curl thanks to apt on debian distribution. Should be adapted if you are using another distribution.
3. We use curl to download sops binary from GitHub. Take care to get the latest version 😉.
4. Import the ci.key into our gpg toolchain.
5. Provide the PASSPHRASE to the gpg toolchain to be totally "not interactive".
6. Then, the CI can decrypt the int.encrypted.env file and use it where we want in our deploy phase.

Running with gitlab-runner 13.0.0 (c127439c)
   on docker-auto-scale 0277ea0f
Preparing the "docker+machine" executor
 Using Docker executor with image google/cloud-sdk ...
 Pulling docker image google/cloud-sdk ...
 Using docker image sha256:5d096c48b3b4bab72df240dcf94940be3e0397606bb4cc94143f2a12189dba1f for google/cloud-sdk ...
Preparing environment
 Running on runner-0277ea0f-project-19225532-concurrent-0 via runner-0277ea0f-srm-1591443365-4e9ad7c3...
Getting source from Git repository
 $ eval "$CI_PRE_CLONE_SCRIPT"
 Fetching changes with git depth set to 50...
 Initialized empty Git repository in /builds/davinkevin/sops-blog-post-repository/.git/
 Created fresh repository.
 From https://gitlab.com/davinkevin/sops-blog-post-repository
  * [new ref]         refs/pipelines/153546622 -> refs/pipelines/153546622
  * [new branch]      master                   -> origin/master
 Checking out 84edc518 as master...
 Skipping Git submodules setup
Restoring cache
00:02
Downloading artifacts
00:01
Running before_script and script
 $ apt-get update && apt-get install -y curl gnupg
 Hit:1 http://deb.debian.org/debian buster InRelease
 Get:2 http://deb.debian.org/debian buster-updates InRelease [49.3 kB]
 Get:3 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
 Hit:4 https://packages.cloud.google.com/apt cloud-sdk-buster InRelease
 Get:5 http://deb.debian.org/debian sid InRelease [146 kB]
 Get:6 http://security.debian.org/debian-security buster/updates/main amd64 Packages [201 kB]
 Get:7 http://deb.debian.org/debian sid/main amd64 Packages.diff/Index [27.9 kB]
 Get:8 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-05-2003.49.pdiff [23.8 kB]
 Get:9 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0208.52.pdiff [16.2 kB]
 Get:10 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0803.34.pdiff [12.5 kB]
 Get:10 http://deb.debian.org/debian sid/main amd64 Packages 2020-06-06-0803.34.pdiff [12.5 kB]
 Fetched 543 kB in 2s (243 kB/s)
 Reading package lists...
 Reading package lists...
 Building dependency tree...
 Reading state information...
 The following packages were automatically installed and are no longer required:
   dh-python libperl5.28 libpython3.7 libpython3.7-minimal libpython3.7-stdlib
   perl-modules-5.28 python3.7-minimal
 Use 'apt autoremove' to remove them.
 The following additional packages will be installed:
   dirmngr gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server
   gpgconf gpgsm gpgv libbrotli1 libcurl4
 Suggested packages:
   dbus-user-session libpam-systemd pinentry-gnome3 tor parcimonie xloadimage
   scdaemon
 The following NEW packages will be installed:
   libbrotli1
 The following packages will be upgraded:
   curl dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client
   gpg-wks-server gpgconf gpgsm gpgv libcurl4
 13 upgraded, 1 newly installed, 0 to remove and 132 not upgraded.
 Need to get 8559 kB of archives.
 After this operation, 1241 kB of additional disk space will be used.
 Get:1 http://deb.debian.org/debian sid/main amd64 gpg-wks-client amd64 2.2.20-1 [507 kB]
 Get:2 http://deb.debian.org/debian sid/main amd64 dirmngr amd64 2.2.20-1 [740 kB]
 Get:3 http://deb.debian.org/debian sid/main amd64 gnupg-utils amd64 2.2.20-1 [889 kB]
 Get:4 http://deb.debian.org/debian sid/main amd64 gpg-wks-server amd64 2.2.20-1 [500 kB]
 Get:5 http://deb.debian.org/debian sid/main amd64 gpg-agent amd64 2.2.20-1 [641 kB]
 Get:6 http://deb.debian.org/debian sid/main amd64 gpg amd64 2.2.20-1 [894 kB]
 Get:7 http://deb.debian.org/debian sid/main amd64 gpgconf amd64 2.2.20-1 [532 kB]
 Get:8 http://deb.debian.org/debian sid/main amd64 gnupg-l10n all 2.2.20-1 [1035 kB]
 Get:9 http://deb.debian.org/debian sid/main amd64 gnupg all 2.2.20-1 [749 kB]
 Get:10 http://deb.debian.org/debian sid/main amd64 gpgsm amd64 2.2.20-1 [627 kB]
 Get:11 http://deb.debian.org/debian sid/main amd64 gpgv amd64 2.2.20-1 [608 kB]
 Get:12 http://deb.debian.org/debian sid/main amd64 libbrotli1 amd64 1.0.7-6.1 [267 kB]
 Get:13 http://deb.debian.org/debian sid/main amd64 curl amd64 7.68.0-1 [249 kB]
 Get:14 http://deb.debian.org/debian sid/main amd64 libcurl4 amd64 7.68.0-1 [321 kB]
 debconf: delaying package configuration, since apt-utils is not installed
 Fetched 8559 kB in 0s (22.3 MB/s)
 (Reading database ... 85863 files and directories currently installed.)
 Preparing to unpack .../00-gpg-wks-client_2.2.20-1_amd64.deb ...
 Unpacking gpg-wks-client (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../01-dirmngr_2.2.20-1_amd64.deb ...
 Unpacking dirmngr (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../02-gnupg-utils_2.2.20-1_amd64.deb ...
 Unpacking gnupg-utils (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../03-gpg-wks-server_2.2.20-1_amd64.deb ...
 Unpacking gpg-wks-server (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../04-gpg-agent_2.2.20-1_amd64.deb ...
 Unpacking gpg-agent (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../05-gpg_2.2.20-1_amd64.deb ...
 Unpacking gpg (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../06-gpgconf_2.2.20-1_amd64.deb ...
 Unpacking gpgconf (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../07-gnupg-l10n_2.2.20-1_all.deb ...
 Unpacking gnupg-l10n (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../08-gnupg_2.2.20-1_all.deb ...
 Unpacking gnupg (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../09-gpgsm_2.2.20-1_amd64.deb ...
 Unpacking gpgsm (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Preparing to unpack .../10-gpgv_2.2.20-1_amd64.deb ...
 Unpacking gpgv (2.2.20-1) over (2.2.12-1+deb10u1) ...
 Setting up gpgv (2.2.20-1) ...
 Selecting previously unselected package libbrotli1:amd64.
 (Reading database ... 85881 files and directories currently installed.)
 Preparing to unpack .../libbrotli1_1.0.7-6.1_amd64.deb ...
 Unpacking libbrotli1:amd64 (1.0.7-6.1) ...
 Preparing to unpack .../curl_7.68.0-1_amd64.deb ...
 Unpacking curl (7.68.0-1) over (7.64.0-4+deb10u1) ...
 Preparing to unpack .../libcurl4_7.68.0-1_amd64.deb ...
 Unpacking libcurl4:amd64 (7.68.0-1) over (7.64.0-4+deb10u1) ...
 Setting up libbrotli1:amd64 (1.0.7-6.1) ...
 Setting up gnupg-l10n (2.2.20-1) ...
 Setting up gpgconf (2.2.20-1) ...
 Setting up libcurl4:amd64 (7.68.0-1) ...
 Setting up curl (7.68.0-1) ...
 Setting up gpg (2.2.20-1) ...
 Setting up gnupg-utils (2.2.20-1) ...
 Setting up gpg-agent (2.2.20-1) ...
 Installing new version of config file /etc/logcheck/ignore.d.server/gpg-agent ...
 Setting up gpgsm (2.2.20-1) ...
 Setting up dirmngr (2.2.20-1) ...
 Setting up gpg-wks-server (2.2.20-1) ...
 Setting up gpg-wks-client (2.2.20-1) ...
 Setting up gnupg (2.2.20-1) ...
 Processing triggers for libc-bin (2.30-8) ...
 $ curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /usr/local/bin/sops
 $ chmod +x /usr/local/bin/sops
 $ cat $KEY | gpg --batch --import
 gpg: directory '/root/.gnupg' created
 gpg: keybox '/root/.gnupg/pubring.kbx' created
 gpg: /root/.gnupg/trustdb.gpg: trustdb created
 gpg: key FFEFFE9451381735: public key "continuous-integration <[e-mail]>" imported
 gpg: key FFEFFE9451381735: secret key imported
 gpg: Total number processed: 1
 gpg:               imported: 1
 gpg:       secret keys read: 1
 gpg:   secret keys imported: 1
 $ echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
 $ sops -d int.encrypted.env > int.env
 $ cat int.env
 secret=5Tz2QNxki789YFDa
Running after_script
00:01
Saving cache
00:02
Uploading artifacts for successful job
00:01
 Job succeeded

Like we see here (and in the GitLab job view), every time the pipeline is triggered, it will re-install every software, from gpg to sops, which can be time-consuming and source of error.

🦊 GitLab CI Optimization

Devon will create another project just for CI tooling image, and he will prepare the image of the CI to be able to have gpg and sops installed. Here is for example, the Dockerfile:

FROM tutum/curl AS downloader

RUN curl -qsL https://github.com/mozilla/sops/releases/download/v3.5.0/sops-v3.5.0.linux -o /opt/sops && \
    chmod +x /opt/sops

FROM google/cloud-sdk as final

COPY --from=downloader /opt/sops /usr/local/bin/sops

RUN apt-get update && apt-get install -y gnupg --no-install-recommends

Then, he will activate a schedule to build this image every day and publish it in their own docker registry. Thanks to this, he can improve the previous .gitlab-ci.yml:

deploy int:
  image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops # Image created by the previous Dockerfile 🐳
  before_script:
  - cat $KEY | gpg --batch --import 
  - echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
  script: 
  - sops -d int.encrypted.env > int.env
  - cat int.env

You can see, the job execution is much straightforward now:

Running with gitlab-runner 13.0.0 (c127439c)
   on docker-auto-scale ed2dce3a
Preparing the "docker+machine" executor
 Using Docker executor with image registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
 Authenticating with credentials from job payload (GitLab Registry)
 Pulling docker image registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
 Using docker image sha256:ec7e941ae9f66656c9c0b7b4ce61b229eeb215492c65ebfe21c69026aa166013 for registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops ...
Preparing environment
00:05
 Running on runner-ed2dce3a-project-19225532-concurrent-0 via runner-ed2dce3a-srm-1591446538-6f33846d...
Getting source from Git repository
00:03
 $ eval "$CI_PRE_CLONE_SCRIPT"
 Fetching changes with git depth set to 50...
 Initialized empty Git repository in /builds/davinkevin/sops-blog-post-repository/.git/
 Created fresh repository.
 From https://gitlab.com/davinkevin/sops-blog-post-repository
  * [new ref]         refs/pipelines/153552640 -> refs/pipelines/153552640
  * [new branch]      master                   -> origin/master
 Checking out 625b7479 as master...
 Skipping Git submodules setup
Restoring cache
00:01
Downloading artifacts
00:02
Running before_script and script
00:04
 Authenticating with credentials from job payload (GitLab Registry)
 $ cat $KEY | gpg --batch --import
 gpg: directory '/root/.gnupg' created
 gpg: keybox '/root/.gnupg/pubring.kbx' created
 gpg: /root/.gnupg/trustdb.gpg: trustdb created
 gpg: key FFEFFE9451381735: public key "continuous-integration <[e-mail]>" imported
 gpg: key FFEFFE9451381735: secret key imported
 gpg: Total number processed: 1
 gpg:               imported: 1
 gpg:       secret keys read: 1
 gpg:   secret keys imported: 1
 $ echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp)
 $ sops -d int.encrypted.env > int.env
 $ cat int.env
 secret=5Tz2QNxki789YFDa
Running after_script
00:02
Saving cache
00:01
Uploading artifacts for successful job
00:02
 Job succeeded

NOTE: You can use this method with any docker registry, not only the one from GitLab 🦊.

Of course, the before_script should be copy/paste for every job requiring access to secrets. Devon will refactor this to optimize the Don't Repeat Yourself aspect:

.auth: &auth | # <1>
  function gpg_auth() {
    cat $KEY | gpg --batch --import > /dev/null
    echo $PASSPHRASE | gpg --batch --always-trust --yes --passphrase-fd 0 --pinentry-mode=loopback -s $(mktemp) > /dev/null
  }

  gpg_auth

deploy alice:
  image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops
  before_script:
  - *auth # <2>
  script: 
  - sops -d dev_a.encrypted.env > dev_a.env
  - cat dev_a.env

deploy int:
  image: registry.gitlab.com/davinkevin/sops-blog-post-repository/gcloud-sops
  before_script:
  - *auth # <3>
  script: 
  - sops -d int.encrypted.env > int.env
  - cat int.env

1. A YAML anchor used to set a custom shell function and calling it right away.
2. Usage of the anchor as a before_script step in the job.
3. Re-use of the auth anchor in another job.

You can see deploy alice and deploy int logs in the GitLab UI.

Of course, this is just an extract of the real .gitlab-ci.yaml, which includes linting, test, build & cie 👍.

Conclusion

Here, we learn how to configure our CI identity and configure the GitLab CI 🦊 to be able to decrypt secrets. We also see how to optimize it in a GitLab environment leveraging usage of home made docker image.

You can find the source code of this article, files and scripts in this GitLab repository. The CI is triggered in this GitLab repository