Secrets in ArgoCD with Sops

ArgoCD is a tool that implements the gitops philosophy for deploying applications on Kubernetes. It is a declarative, Git-based deployment system that uses a simple, human-readable manifest file to define the desired state of your application. In this article, we will explore how to use ArgoCD with Sops to manage secrets in a gitops workflow.

We use sops with GCP KMS, the first thing we need is a service account with roles/cloudkms.cryptoKeyDecrypter role and create a secret on Kubernetes:

kubectl create secret generic google-sa --from-file sa.json

The idea is to use Argo Config Management Plugin to decrypt the secret and generate a plane yaml with all Kubernetes objects (deployment, services, ...) and use Cumstom Tooling to install Sops and other tools required.

In the process we have encountered several problems, and one of them is that it is necessary to use yq to correctly format the output yamls and avoid errors such as: error converting YAML to JSON: yaml: line 4: did not find expected ',' or ']'.

These are the values to install argo-cd by Helm. Configuration assumes that all secrets are in secrets.enc file coding with yaml.

server:
  config:
    configManagementPlugins: |
      - name: sops
        init:
          command: ["/bin/sh", "-c"]
          args: ["echo '---' > secrets.yaml && sops -d --input-type yaml --output-type yaml secrets.enc >> secrets.yaml"]
        generate:
          command: ["/bin/sh", "-c"]
          args: ["source /virtualenv-python/bin/activate; pip install yq; cat *.yaml | yq -y"]
repoServer:
  volumes:
    - name: custom-tools
      emptyDir: {}
    - name: virtualenv-python
      emptyDir: {}
    - name: google-sa
      secret:
        secretName: google-sa
        items:
          - key: sa.json
            path: sa.json
  volumeMounts:
    - mountPath: /usr/local/bin/sops
      name: custom-tools
      subPath: sops
    - mountPath: /usr/local/bin/jq
      name: custom-tools
      subPath: jq
    - mountPath: /etc/secrets/sa.json
      name: google-sa
      subPath: sa.json
    - mountPath: /virtualenv-python
      name: virtualenv-python
  env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /etc/secrets/sa.json
  initContainers:
    - name: custom-tools
      image: alpine:3.8
      command: ["/bin/sh", "-c"]
      args:
        - wget https://github.com/mozilla/sops/releases/download/v3.7.3/sops-v3.7.3.linux.amd64;
          chmod a+x sops-v3.7.3.linux.amd64;
          mv sops-v3.7.3.linux.amd64 /custom-tools/sops;
          wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64;
          chmod a+x jq-linux64;
          mv jq-linux64 /custom-tools/jq;
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    - name: virtualenv-python
      image: python:3.7
      command: ["/bin/sh", "-c"]
      args:
        - python3 -m venv /virtualenv-python
      volumeMounts:
        - mountPath: /virtualenv-python
          name: virtualenv-python

helm upgrade sops argo/argo-cd --values values-sops.yaml

And an application example to try it.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/Callepuzzle/manifests
    targetRevision: main
    path: poc-argocd
    plugin:
      name: sops
  destination:
    name: ''
    namespace: ''
    server: 'https://kubernetes.default.svc'