Sealed secrets - the secret sauce for managing secrets

Kubernetes provides a rudimentary secret object which encodes the input configuration into base64 for storage. This solution is inadequate mainly due to

• anybody can decode the secret
• due to the above it is anti-"everything as code" There are other reasons like enforcement, RBAC ...

Bitnami sealed secrets (Bitnami-SS) is a compelling solution to tackle this problem. Main advantages/features of Bitnami-SS are:

• Based on encryption (encrypt the secret with public key while decryption happens in the cluster with private key)
• Better control: secret mutations can be scoped to none(mode=strict) allowed within a namespace(mode=namespace-wide) or cluster-wide(mode=cluster-wide)
• Inbuilt key rotation mechanism for security purposes.

With this, the encrypted sealed secret manifests are what gets stored in the source control and GitOps becomes viable again :)

Installation

❯ helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-

❯ helm upgrade --install ss-app sealed-secrets/sealed-secrets --namespace=sealed-secrets --create-namespace --set=keyrenewperiod=1h

# Check
❯ kubectl get pods -n sealed-secrets
NAME                                     READY   STATUS    RESTARTS   AGE
ss-app-sealed-secrets-556c68c858-zshwb   1/1     Running   0          5m13s

Encryption

Install kubeseal binary from https://github.com/bitnami-labs/sealed-secrets/releases

# Blueprint for a secret from literal
❯ kubectl create secret generic db-creds --from-literal=user=adam --from-literal=password=paSSwoRD --dry-run=client -o yaml | tee secret.yaml
apiVersion: v1
data:
  password: cGFTU3dvUkQ=
  user: YWRhbQ==
kind: Secret
metadata:
  creationTimestamp: null
  name: db-creds

# Create sealed secret manifest (and commit to source-code)
❯ kubeseal --controller-name=ss-app-sealed-secrets --controller-namespace=sealed-secrets -o yaml <secret.yaml | tee sealed-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: db-creds
  namespace: default
spec:
  encryptedData:
    password: AgAwmSBxsO7GsRjI5Nqhm4DdPfW6x1JPlLtyb4lG+Cmhy1iZiXoRHiTeyr70HT/KLqEDcu2DFbX5jwsUHv4pADLE4ARUECQklIUHlG8dcCXoufkFUrAhp4qzpMwm9IiN394/oHlHhXdGNjO6ORW39B6SgSdkIIZkTlDanyyYzJB0dwjNLvsjGzCNZ+BY58DTRSTEKofe6rsYwdSQKFb1fa/CyiA9m6vuOzbJHmzJv4EMJeXtaYjF6LLhfRtGNpzN5IUck+tiPJu8cirZ1SV6vdTNuBNMYEn8mvhSPQ8PwzaWt3+rQ0TrEWYPdRuG30UUT4LTd19PC08knMC2ayVn+FzpdS/nVoyoJCUW8nIkhjUr51Eq8ulwNa4Ws316KViUk6zRlxRWhY/+cTPpBuhbHkn9x60yyw4hr+Ef6QxID6EoroxFC6DOq3O5NqkkNJWSSosTCw0FPFXuLsvmhrk1m01Yn4jYcTnhJVyCLkoIrzlnvqVxhOniGY05s0EsveaAKpphZmN/xEEW7KeiFD+Khgc24u13O94OeEs3E4oIbEfDvdn3ynygaSyFc5nO70Hqn6TUT6Xxmzp9xS3G2Pxrm16QeTbVe42U8Urpr460YYgnQL9D2bKAXDcCxtFVqxjZuP5qgbj/egfsXwjeu1HY7Q3R+sfZndAxvyYIAVYhG7ikL+cpIQVhb9ZspDjPK3ebmFb74yXdC7itUA==
    user: AgBHkkXsSoRchQhWATyFKeYynn+eBuT1NK5n4Z0pN1VXwWaIy037dBksZYDZ2rMEKs7wYZD3ln5lXaA9i8/BDGbg155jl4rFaCr/7c4PoruMFtIIXCF/ic7sdgS0dskOs2qLZm2XPvcY7TyQj/7d/XOYVd4RuancgHZ9NlBjyALVvdSEjVRBRWIEYwT81p+pKFdgdxp3IkhJ23mrmtqqIQ9OuE6ogwjgWXMdba41WNi+h9Dxi9fvoKF18KcStwjB+xk5Fs2FCMoib190zWojtGMQqO8urZpjwMy60kM17ejVahwdT+z0skGOtna0KcmHfkZWk/NyMY1ivyTzF2NmfCecPlGOYr3X6wYIEFySZwnR1SIGLFvRTb1s9ymGbH0mC60SR9YkxowixSFOaW+k0uAFZcLd+98Bu9obLX6hN8QzzSdiaGIWACi6KNdXBKoi3xn0L0J4Pg+nhYwq4dJ4a1HFo2PWZKwZI3letWL0hV3E1Z9WJYZ+GhasESfxeexNyVvDUAfZ/+W+opW+RI/lEonz9fRL+6JMgviiU6oa+1HteIZ4AE9Xv6bg1OnuXbZxn5oNkfAuLrmqn9bqozXfes6g3W1FFCosstS4xjlX/C/ODv2dPdSPQDQOGn/2Dao3+GWKPmlYD0aKHhjK38uCsstPQbC7yMxYI0RuQqzFmQn4rHJ2PN0yLFspT6qiVhTBYs2ajeR9
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: db-creds
      namespace: default

Decryption and verification

# Apply the manifest
❯ kubectl create -f sealed-secret.yaml
sealedsecret.bitnami.com/db-creds created

# Check if decoding was okay
❯ kubectl logs -f ss-app-sealed-secrets-556c68c858-zshwb -n sealed-secrets
2022/04/25 09:10:30 Starting sealed-secrets controller version: 0.17.4
controller version: 0.17.4
...
2022/04/25 09:10:31 HTTP server serving on :8080
2022/04/25 09:18:36 Updating default/db-creds
2022/04/25 09:18:36 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"db-creds", UID:"7f6012e2-be9d-4512-a38a-710aa9cbd8be", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"35964", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully
2022/04/25 09:18:36 Updating default/db-creds
2022/04/25 09:18:36 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"db-creds", UID:"7f6012e2-be9d-4512-a38a-710aa9cbd8be", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"35966", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

# Check if secret got created
❯ kubectl get sealedsecrets.bitnami.com
NAME       AGE
db-creds   114s

❯ kubectl get secret db-creds -o json | jq ".data | map_values(@base64d)"
{
  "password": "paSSwoRD",
  "user": "adam"
}