Sealed secrets - Bring your own keys and multi-cluster scenario

Bitnami sealed secrets have a key rotation done every 30 days by default and every sealed-secret-controller creates it's own private key. This makes it hard to adopt for multi-cluster scenarios. It can be mitigated by using one's own key pair for encryption/decryption across multiple clusters.

Using own key pair

1. Preparation

❯ kubectl create ns sealed-secrets
❯ export PRIVATEKEY="acmetls.key"
❯ export PUBLICKEY="acmetls.crt"
❯ export NAMESPACE="sealed-secrets"
❯ export SECRETNAME="acme-keys"

# Create key pair
❯ openssl req -x509 -nodes -newkey rsa:4096 -keyout "$PRIVATEKEY" -out "$PUBLICKEY" -subj "/CN=sealed-secret/O=sealed-secret"
Generating a 4096 bit RSA private key
.....++
......++
writing new private key to 'acmetls.key'
-----
❯ ls acme*
acmetls.crt acmetls.key

1. Prepare and update new keys into controller

❯ kubectl create ns sealed-secrets
namespace/sealed-secrets created

# Create secret and label it as "active"
❯ kubectl -n "$NAMESPACE" create secret tls "$SECRETNAME" --cert="$PUBLICKEY" --key="$PRIVATEKEY"
kubectl -n "$NAMESPACE" label secret "$SECRETNAME" sealedsecrets.bitnami.com/sealed-secrets-key=active
secret/acme-keys created
secret/acme-keys labeled

# Install the controller
❯ helm upgrade --install ss-app sealed-secrets/sealed-secrets --namespace=sealed-secrets

# delete existing key
❯ kubectl get pods -n sealed-secrets
NAME                                     READY   STATUS    RESTARTS   AGE
ss-app-sealed-secrets-556c68c858-qq5k6   1/1     Running   0          3m47s
❯ kubectl delete pod ss-app-sealed-secrets-556c68c858-qq5k6 -n sealed-secrets
pod "ss-app-sealed-secrets-556c68c858-qq5k6" deleted

# another instance of the pod will come up with the new "acme-keys"
❯ kubectl logs ss-app-sealed-secrets-556c68c858-gqpwf -n sealed-secrets
2022/04/25 13:02:01 Starting sealed-secrets controller version: 0.17.4
controller version: 0.17.4
2022/04/25 13:02:01 Searching for existing private keys
2022/04/25 13:02:02 ----- sealed-secrets-key5nhjv
2022/04/25 13:02:02 ----- sealed-secrets-key98m56
2022/04/25 13:02:02 ----- sealed-secrets-keyztj4v
2022/04/25 13:02:02 ----- acme-keys
2022/04/25 13:02:02 HTTP server serving on :8080

1. Encrypt with new keys

❯ kubectl create secret generic secret-name --dry-run=client --from-literal=foo=bar -o yaml | kubeseal --controller-name=ss-app-sealed-secrets --controller-namespace=sealed-secrets -o yaml --cert=$PUBLICKEY | kubectl apply -f -
sealedsecret.bitnami.com/secret-name created

1. Verify decryption

❯ kubectl logs ss-app-sealed-secrets-556c68c858-gqpwf -n sealed-secrets | tail -2
2022/04/25 13:08:07 Updating default/secret-name
2022/04/25 13:08:07 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"secret-name", UID:"c9ee00ac-c974-43f7-9b7d-16cda1b516ff", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"57973", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

❯ kubectl get sealedsecrets.bitnami.com
NAME          AGE
secret-name   47s

❯ kubectl get secrets secret-name
NAME          TYPE     DATA   AGE
secret-name   Opaque   1      57s

Toying with the secret rotation interval during creation of the sealed-secrets controller

• Edit --key-renew-period=<value> in the command line args for the controller (in-case of deployment)
• Use --set=keyrenewperiod=<value> while installing through Helm

Note: keyrenewperiod=0 stops keys from being rotated; though seemingly easy solution, not recommended in production use-cases