At Camptocamp, we've been using Argo CD for some time. It does a wonderful job of deploying Helm charts to Kubernetes clusters. But we lacked a way to manage our secrets so that we could safely commit them to our Git repositories.
Helm Secrets plugin
We knew about Helm Secrets, a Helm plugin which uses Sops under the hood to manage encrypted value files. We intended to use it with Argo CD but we faced several issues:
- To render an Helm chart's manifests, Argo CD issues a
helm templatecommand. To use Helm Secrets, it would have to executehelm secrets templateinstead. However, there is no way to edit the Helm command used by Argo CD. One has to configure a custom plugin. This has the drawback of loosing Helm specific features (like specifying value files or parameters). - Helm Secrets works by decrypting secret value files to temporary files on disk before actually calling Helm. While not a big concern, it's not a security best practice. Secrets should never touch the disk when they are unencrypted.
- Helm Secrets seemed unmaintained at the time.
Helm Sops
Since we only needed a small subset of Helm Secrets' functionalities, we decided to write a small Go binary which would transparently wrap Helm so that we could substitute the Helm binary with it in the Argo CD Docker image. Let's introduce Helm Sops!
Helm Sops
Helm Sops is a Helm wrapper which decrypts Sops encrypted value files before invoking Helm. It does so by using named pipes to pass cleartext value files to Helm in order to avoid secrets being written to disk in clear.
Installation
Prerequisites
Helm is needed for Helm Sops to work. Follow the instructions here to install it.
Getting Helm Sops binary
Helm Sops releases
Helm Sops released binaries can be downloaded from GitHub.
Building from sources
Helm Sops can be built using the go build command.
Deploying Helm Sops
Deploy Helm Sops executable (helm-sops) in a directory present in the PATH. When invoking Helm Sops, it will look for a Helm executable named helm in the PATH.
Alternatively, Helm Sops executable can be renamed helm before deploying it. When invoked as helm, Helm Sops will look for a Helm executable named _helm…
Helm Sops is loosely inspired by Helm Secrets but uses named pipes instead of temporary files to pass decrypted secret value files to Helm.
To use it:
- deploy our custom Argo CD Docker image integrating Helm Sops to your Kubernetes cluster,
- install Sops on your workstation and create a
.sops.yamlfile to specify the encryption method to use (GPG key, AWS KMS key, etc), - use Sops to create a file named
secrets.yamland put your Helm secret values inside, -
git commit, -
git push, - finally, visit the Argo CD web interface to deploy your Helm application \o/
- (optional) install Helm Sops on your workstation to be able to use the Argo CD CLI with encrypted value files.
Full step by step instructions can be found here.
Security Considerations
While Sops uses battle tested cryptography, the way we use it with Argo CD makes it vulnerable to the confused deputy problem. Someone which can deploy applications using Argo CD and has access to an encrypted value file but cannot decrypt it, can deploy a new application accompanied by the encrypted value file and make Argo CD decrypt it for them.
This limitation can be problematic in multi-tenant environments. A way to solve it is to make use of AEAD. AWS KMS offers encryption context to this end.
Sops already supports AWS KMS encryption context but it relies on the one specified in the metadata of the encrypted file. We would need Argo CD to be able to overwrite this encryption context when decrypting secret value files so that it cannot be confused anymore. We'd be happy to work with Sops maintainers to add this possibility 😉
Oldest comments (1)
This is rad! Thanks for your efforts creating Helm Sops, exactly what I was looking for after using helm secrets and setting up Argo.