DEV Community

loading...
Cover image for Store your Kubernetes Secrets in Git thanks to Kubeseal. Hello SealedSecret!
Stack Labs

Store your Kubernetes Secrets in Git thanks to Kubeseal. Hello SealedSecret!

Aurélie Vache
DevRel - Google Developer Expert on Cloud - Docker captain - CNCF Ambassador - CKAD - Speaker - Sketchnoter - Technical writer - Conferences organizer - Women in tech association Leader - Mentor
Updated on ・6 min read

In the Kubernetes world when we want to handle sensitive data the usage is to store it in a Secret in a Kubernetes cluster. Cool, but when we want to save our Secrets in our source code (Git) repositories, a security problem appears.
We will see in this article, a solution that can helps us ;-).

Kubernetes Secrets

Secrets in Kubernetes are used to store sensitive data, like password, keys, certificates and token. Secrets are encoded in base64 and automatically decoded when they are attached and read by a Pod.

Alt Text

Cool, so what's the problem?

/!\ Be careful:

Alt Text

A secret in Kubernetes cluster is encoded in base64 but not encrypted!

Theses data are "only" encoded so if a user have access to your secrets, he can simply base64 decode to see your sensitive data (kubectl get secret my-secret -o jsonpath="{.data.password}" | base64 -D for Mac and --decode instead of -D for Linux platform).

And suddenly, since the secrets aren't encrypted, it can be unsecure to commit them to your Git repository :-(.

In the DevOps world, we can build, test, package and deploy our applications and infrastructure through a CI (Continuous Integration) & CD (Continuous Delivery) pipeline but we can also handle this through GitOps (a way of managing all our configurations and infrastructure through code in Git repositories).

So if we want to store in Git repositories our Kubernetes manifests, a problem appears. And as usual, several solutions exists , and today in this article we will see one of them: kubeseal.

Encrypt/Seal your Secrets with Kubeseal

Secrets are cool but what about encrypted secrets with kubeseal?
The goal of kubeseal, a bitnami tool, is to encrypt your Kubernetes Secret into a SealedSecret.
Thanks to that, it will be safe to store in public Git repository.
The SealedSecret can be decrypted only by the controller running in the target cluster and nobody else.

Overview

Alt Text

As you can see in the schema, a sealed-secret-controller run in the Kubernetes cluster. He listens when a new SealedSecret object appears, unsealed it (thanks to known certificates) and create a Kubernetes secret in the same namespace as the SealedSecret.

Be careful, if you delete the SealedSecret in your cluster, the linked Secret will be deleted too.

Usage - Step by step

After the theory, it's time to see all the steps.
Let's split the steps in two parts: server and client part.

Server part

We will:

  • deploy the sealed-secrets-controller in the cluster
  • retrieve generated certificate keypair

1. Install SealedSecret CRD, server-side controller into kube-system namespace

$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.15.0/controller.yaml
Enter fullscreen mode Exit fullscreen mode

NOTE: If you can't (or don't want) to use the kube-system namespace, please consider this approach.

NOTE: if you want to install it on a GKE cluster for which your user account doesn't have admin rights, please read this guide.

Once you deploy the YAML Kubernetes manifest file, it will create the SealedSecret resource and install the controller into kube-system namespace, create a service account and necessary RBAC roles.

After a few moments, the controller will start, generate a key pair, and be ready for operation.

Resources created:

$ kubectl get all -n kube-system
NAME                                             READY   STATUS    RESTARTS   AGE
...           1/1     Running   63         2d
pod/sealed-secrets-controller-123456-abcdef   1/1     Running   2          4d

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/sealed-secrets-controller   ClusterIP   xxx.xxx.xxx.xxx   <none>        8080/TCP   4d

NAME                                        DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/sealed-secrets-controller   1         1         1            1           4d

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/sealed-secrets-controller-123abc456def   1         1         1       4d
Enter fullscreen mode Exit fullscreen mode

If the controller Pod is not in Running state, check its logs:

$ kubectl logs sealed-secrets-controller-123456-abcdef -n kube-system
Enter fullscreen mode Exit fullscreen mode

2. Retrieves the certificate keypair generated by the controller at start:

$ kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key
NAME                       TYPE                DATA   AGE
sealed-secrets-key<ID>     kubernetes.io/tls   2      4d
Enter fullscreen mode Exit fullscreen mode
$ kubectl get secret sealed-secrets-key<ID> -o yaml -n kube-system

apiVersion: v1
data:
  tls.crt: <encoded_crt>
  tls.key: <encoded_key>
kind: Secret
metadata:
  creationTimestamp: "2021-02-20T20:37:11Z"
  generateName: sealed-secrets-key
  labels:
    sealedsecrets.bitnami.com/sealed-secrets-key: active
  name: sealed-secrets-key<ID>
  namespace: kube-system
  resourceVersion: "123456"
  selfLink: /api/v1/namespaces/kube-system/secrets/sealed-secrets-key<ID>
  uid: abc123-45def-67gh-89ij-687vf6
type: kubernetes.io/tls
Enter fullscreen mode Exit fullscreen mode

Here you can see tls.crt and tls.key data that you can base64 decode and store in your Vault or in your secret management tool.

Client part

We will:

  • install needed CLI tools
  • create a Kubernetes Secret
  • seal it into a SealedSecret
  • convert JSON to YAML file and clean it
  • deploy it in the Kubernetes cluster
  • check if everything have been deployed & created

0. Pre-requisites

Install following tools:

For MacOS you can install it via Brew:

$ brew install kubeseal
Enter fullscreen mode Exit fullscreen mode

For others platform you can follow the installation guide.

$ kubectl krew install neat
Enter fullscreen mode Exit fullscreen mode

1. Create Kubernetes secret

$ kubectl create secret generic my-token --from-literal=my_token='123456789abc123def456ghi789' --dry-run=client -o yaml -n my-namespace > my-token.yaml
Enter fullscreen mode Exit fullscreen mode

The YAML file should be like this:

$ cat my-token.yaml

apiVersion: v1
data:
  my_token: <a_token>
kind: Secret
metadata:
  creationTimestamp: null
  name: my-token
  namespace: my-namespace
Enter fullscreen mode Exit fullscreen mode

Or you can create it also with Kustomize:

$ vi kustomization.yaml

namespace: my-namespace
secretGenerator:
- name: my-token
  literals:
  - my_token=<a_token>
generatorOptions:
  disableNameSuffixHash: true

$ kubectl kustomize . > my-token.yaml
Enter fullscreen mode Exit fullscreen mode

2. Seal the secret (with the retrieved certificate)

$ kubeseal --cert tls.crt --format=yaml < my-token.yaml > mysealedtoken.yaml
Enter fullscreen mode Exit fullscreen mode

The sealed YAML file should be like this:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: my-token
  namespace: my-namespace
spec:
  encryptedData:
    my_token: <encrypted_token>
  template:
    metadata:
      creationTimestamp: null
      name: my-token
      namespace: my-namespace
Enter fullscreen mode Exit fullscreen mode

3. Clean the YAML manifest file with "kubectl neat" command

$ kubectl neat -f mysealedtoken.yaml > mycleanedsealedtoken.yaml
Enter fullscreen mode Exit fullscreen mode

The cleaned sealed secret YAML file should be like this:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: my-token
  namespace: my-namespace
spec:
  encryptedData:
    my_token: <encrypted_token>
  template:
    metadata:
      creationTimestamp: null
      name: my-token
      namespace: my-namespace
    type: Opaque
Enter fullscreen mode Exit fullscreen mode

4. Deploy SealedSecret in the cluster

$ kubectl apply -f mycleanedsealedtoken.yaml -n my-namespace
Enter fullscreen mode Exit fullscreen mode

5. Check in your cluster

You can check/verify in your Kubernetes cluster if the SealedSecret and Secret have been correctly deployed:

$ kubectl get sealedsecret -n my-namespace
NAME           TYPE       DATA    AGE
my-token      Opaque       1      4d

$ kubectl get secret -n my-namespace
NAME          AGE
my-token    4d
Enter fullscreen mode Exit fullscreen mode

Ok, my Secret have been successfully created by the controller! :-)

Details

Be careful, the SealedSecret and Secret resources must have the same namespace and name. This is a feature to prevent other users on the same cluster from re-using your sealed secrets.

Debug

As you know, some issues can appear, so in order to debug/troubleshoot the behavior of the controller, you can watch their logs:

$ kubectl logs sealed-secrets-controller-<podID> -n kube-system
Enter fullscreen mode Exit fullscreen mode

A possible issue is that you sealed the secret with another certificate than the ones the controller know ;-).

Conclusion

Store your sensitive data in a Kubernetes Secret object is a common practice, but don't forget that a Secret is only encoded and not encrypted. So if you want to store them in a Git Repository (in GitHub or Gitlab repositories for example), you'll need to find a secure solution. Kubeseal is one of them and I hope this article helps you to see that it can be a solution that you can try in your side.

Discussion (1)

Collapse
csgeek profile image
csgeek

That's nifty. I've used ansible vaults in the past for a similar purpose but it's nice seeing tools designed for k8s with the same concept in mind.