How to manage Kubernetes secrets securely in Git

Kubernetes has a declarative approach in managing resources.

While it's a common practice to keep the Kubernetes manifest files in a Git repository, storing and managing Kubernetes secrets has always been a challenge. Sealed Secrets was developed to address this problem.

It's a Kubernetes operator which allows you to store secrets in Git. It uses asymmetric key encryption to encrypt the secrets, so that only the operator in the Kubernetes cluster can decrypt them. Therefore, the encrypted SealedSecrets are safe to store in a Git repository.

In this article, we will install the Sealed Secrets operator and demonstrate how to use it.

Sealed Secrets is composed of two parts:

• The Sealed Secrets operator which runs in the Kubernetes cluster
• The kubeseal client-side command line tool, which is used to encrypt the secrets and generates the SealedSecrets Kubernetes resources.

The original article is available here.

Prerequisites

kubectl connected to a Kubernetes cluster

Install kubeseal

kubeseal is a command line tool to encrypt secrets and generate SealedSecrets. On macOS you can install kubeseal with Homebrew.

brew install kubeseal

You can find installation instructions for other platforms here: https://github.com/bitnami-labs/sealed-secrets#installation

Create a Kubernetes namespace for Sealed Secrets

Let's start by creating a namespace for the Sealed Secrets operator.

kubectl create namespace sealed-secrets

Deploy the Sealed Secret operator

Download the Kubernetes manifests for the Sealed Secret operator.

curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.12.4/controller.yaml > sealed-secrets.yaml

As we do not want the operator to run in the kube-system namespace, we use sed to change it to sealed-secrets.

sed -i -e 's/kube-system/sealed-secrets/g' sealed-secrets.yaml

Now we use kubectl to deploy the Sealed Secret operator.

kubectl apply -f sealed-secrets.yaml
service/sealed-secrets-controller created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
rolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
clusterrole.rbac.authorization.k8s.io/secrets-unsealer created
serviceaccount/sealed-secrets-controller created
customresourcedefinition.apiextensions.k8s.io/sealedsecrets.bitnami.com created
role.rbac.authorization.k8s.io/sealed-secrets-service-proxier created
role.rbac.authorization.k8s.io/sealed-secrets-key-admin created
clusterrolebinding.rbac.authorization.k8s.io/sealed-secrets-controller created
deployment.apps/sealed-secrets-controller created

When the Sealed Secret operator starts, it will generate a public/private RSA key pair if it does not exist yet. Afterwards it will print the public key which can be used for encryption / SealedSecret creation.

Verify deployment

Verify the deployment by checking the status Kubernetes pods in the sealed-secrets namespace.

kubectl get pods --namespace sealed-secrets

NAME                                        READY   STATUS    RESTARTS   AGE
sealed-secrets-controller-6f5785b7f-m9hmq   1/1     Running   0          20m

Create the demo Kubernetes namespace

kubectl create namespace demo

namespace/demo created

Create a Kubernetes secret

Use kubectl to create a local secret manifest file.

We need to pass the following options:

• namespace=demo -- Specify the namespace where the secret should live
• from-literal -- Specify the secret key=value pairs
• output=yaml -- Specify the output format of kubectl
• dry-run=client -- Only print the secret without actually creating it / sending it to the Kubernetes API server.
• > db-secret.yaml -- This tells kubectl to store it's output in db-secret.yaml

kubectl create secret generic db-secret
--namespace=demo
--from-literal=username=admin
--from-literal=password=Y4nys7f11
--dry-run=client
--output=yaml > db-secret.yaml

cat db-secret.yaml

apiVersion: v1
data:
  password: WTRueXM3ZjEx
  username: YWRtaW4=
kind: Secret
metadata:
  creationTimestamp: null
  name: db-secret
  namespace: demo

As you see above, the normal Kubernetes resource contains the the kev=value pair, where the value is just base64 encoded and not encrypted.

Encrypt Kubernetes secret

Use kubeseal to encrypt the db-secret. The output is a SealedSecret resource which contains the encrypted secret.

We need to pass the following options:

• controller-namespace=sealed-secret -- Specify the namespace where the Sealed Secret operator runs (default is kube-system)
• format=yaml -- Specify the output format of kubeseal (default is json)
• < db-secret.yaml -- This tells kubeseal to take the db-secret.yaml as input
• > sealed-db-secret.yaml -- This tells kubeseal to store it's output in sealed-db-secret.yaml

kubeseal
--controller-namespace=sealed-secrets
--format=yaml < db-secret.yaml > sealed-db-secret.yaml

cat sealed-db-secret.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: db-secret
  namespace: demo
spec:
  encryptedData:
    password: AgDj/I4L89v9km3cuYyckAe1/sNcjH67fjhPAjZep37bQ+CCxPvQzEXJrQvI0fzzjF0cHU003JYCFUok4DFMi8nMw9YnXM//JiyBGvlEhmmQ/VI+HgruWcOEIuWw+NfR6GWVn0dLidnYOwnvJG4sAx3iPXcR2oyivj2v5yICx4CgAOS88tZF1AaSxYqhbAYfnD8IiNdfH6+gwYXwLp2pS1f/HLawtmnvAZLVb9JqP1Nidvj34a79xVbGZ7wz04W55uU/gNNZkP5QIvJ49DZkx4JqmKLISrXziPkgt/ZqB+346oXEM+iKoPOgn0sX8M6NcY8hd7SiT9RJ3fQ4TDGOfufn8fTXVtleSN3tOicMpbMhZ7XP/0x1bvN72yYbMPAbZxf+G6Qs6CwlG0ftU/FrD3W5e38ypg5E9Uk7GW0o2A74IOrpmUswDDeR1j+0TGIlOYcwAuQREy1aDcgvm+9Cn1KFp0Fd+q0DxnGf2V31a22DoE1Bxj+R4Jq5bLzI0ZBt/VYO65b1MnXetz3cy3T8gKTFzeflorPU5GBIwKs7AeDUgvXpefp/XfW70afBKhk1la6LstBUjVP6TPI7yKpVWq0FgR0MjeyZFp4eEezDoQJOX/siy5/weWv7sOfifpDp2lxSqPcTd+c1wa512tb6KSapu0vvb7JL9VTyNWdw6Th2ymD2RWtOV8dRXTLXkzd9Q34/8Y0OR4n/cSY=
    username: AgB+7E5iWzUELYgjxwwwnAk1SlR+ZwQV7OafBsZIPA47KgSlp8Xys7slJ2STIQ4nfmg/UTJimTN8mNvF3QK6zEfAeglN75r0HxMe8aWPnJ4OrKwuy8LUpEIKtdlPHLnyHNF2v5Nos/Gej1cshuA27EliqCwRLE7Z4FCxwiUIX0zBraWOJcXXzVylRK3WA2uGrBsH9P9k7+V1h/B66K4djCL47uOLaBQTXRlW45av1cSsDWLpq1em/6Og+GA5txxsNaCDADR2ybqpPg6RRoC1ayf7z+cYrFV84Wc9KNb3+cxHjuEEW/c3PT0xqyqZq7Iqr6254SVTv1bB0joD7wK7yEPUFJSc8M3GnXsG3Eq81QgpjeKHY1uZnT4B5/ztxSeczQoKksjvICUeysb09tZMRgF80A74uAv4e0gyRXBEgk7uPQg/JTqKFo77qntK5lzAGSVJQlumAq8FBeAKwwzOJrltbC8yMprQzapXBj8Vo43gEY1Avkc2Pov/bceppuah2Q9JFPCetpEtI+lruRGlZUR9Iw3T4QcunLXfgC73xWiPY6B/I+dFh2oDoiabo4gonGDLEhtAmtQU6MARWVBbs6AFGHQYRdvU23Xc5ISocG69mwFwevtwkOHQtx+VOcMN1Tmnj5htCb1CaK+rD2xfHjaWZ3VjorWuUCG5kXz5pjeRw95s+h8ru2emRqokj1jUf6xqc03QIg==
  template:
    metadata:
      creationTimestamp: null
      name: db-secret
      namespace: demo

You can now add this SealedSecret resource in Git. As you see it contains the secret, but the values are encrypted. Only the Sealed Secret operator in the cluster has the private key to decrypt the information.

Apply the SealedSecret resource

Use kubectl to apply the SealedSecret resource.

kubectl apply -f sealed-db-secret.yaml

sealedsecret.bitnami.com/db-secret created

Once you apply the SealedSecret, the Sealed Secret operator gets triggered, which is using its private key to decrypt the information and is generating the corresponding Kubernetes secret resource.

Verify deployment

Verify the deployment by checking the SealedSecret and Kubernetes secrets in the demo namespace.

kubectl get sealedsecrets,secrets -n demo

NAME                                 AGE
sealedsecret.bitnami.com/db-secret   55s

kubectl get secrets db-secret -n demo -o yaml

NAME                         TYPE                                  DATA   AGE
secret/db-secret             Opaque                                2      55s

apiVersion: v1
data:
  password: WTRueXM3ZjEx
  username: cHJvZHVzZXI=
kind: Secret
metadata:
  creationTimestamp: "2020-07-02T09:39:15Z"
  name: db-secret
  namespace: demo
  ownerReferences:
  - apiVersion: bitnami.com/v1alpha1
    controller: true
    kind: SealedSecret
    name: db-secret
    uid: 77d1231f-921c-4335-823f-643ec82fdb78
  resourceVersion: "55133137"
  selfLink: /api/v1/namespaces/demo/secrets/db-secret
  uid: 2df6c273-ca03-4523-b45d-f9849b39b92e
type: Opaque

Summary

Sealed Secret was designed to easily fit into automated workflows such as GitOps. Once a Kubernetes secret is converted into a SealedSecret, only the operator in the cluster can decrypt the original secret.

With SealedSecret, the creation and update of Kubernetes secrets does not require high privileges, which makes it easy and secure.