DEV Community

Cover image for Sealed secrets - Bring your own keys and multi-cluster scenario
Ashok Nagaraj
Ashok Nagaraj

Posted on

10 3

Sealed secrets - Bring your own keys and multi-cluster scenario

Bitnami sealed secrets have a key rotation done every 30 days by default and every sealed-secret-controller creates it's own private key. This makes it hard to adopt for multi-cluster scenarios. It can be mitigated by using one's own key pair for encryption/decryption across multiple clusters.

Using own key pair
  1. Preparation
❯ kubectl create ns sealed-secrets
❯ export PRIVATEKEY="acmetls.key"export PUBLICKEY="acmetls.crt"export NAMESPACE="sealed-secrets"export SECRETNAME="acme-keys"

# Create key pair
❯ openssl req -x509 -nodes -newkey rsa:4096 -keyout "$PRIVATEKEY" -out "$PUBLICKEY" -subj "/CN=sealed-secret/O=sealed-secret"
Generating a 4096 bit RSA private key
.....++
......++
writing new private key to 'acmetls.key'
-----ls acme*
acmetls.crt acmetls.key
Enter fullscreen mode Exit fullscreen mode
  1. Prepare and update new keys into controller
❯ kubectl create ns sealed-secrets
namespace/sealed-secrets created

# Create secret and label it as "active"
❯ kubectl -n "$NAMESPACE" create secret tls "$SECRETNAME" --cert="$PUBLICKEY" --key="$PRIVATEKEY"
kubectl -n "$NAMESPACE" label secret "$SECRETNAME" sealedsecrets.bitnami.com/sealed-secrets-key=active
secret/acme-keys created
secret/acme-keys labeled

# Install the controller
❯ helm upgrade --install ss-app sealed-secrets/sealed-secrets --namespace=sealed-secrets

# delete existing key
❯ kubectl get pods -n sealed-secrets
NAME                                     READY   STATUS    RESTARTS   AGE
ss-app-sealed-secrets-556c68c858-qq5k6   1/1     Running   0          3m47s
❯ kubectl delete pod ss-app-sealed-secrets-556c68c858-qq5k6 -n sealed-secrets
pod "ss-app-sealed-secrets-556c68c858-qq5k6" deleted

# another instance of the pod will come up with the new "acme-keys"
❯ kubectl logs ss-app-sealed-secrets-556c68c858-gqpwf -n sealed-secrets
2022/04/25 13:02:01 Starting sealed-secrets controller version: 0.17.4
controller version: 0.17.4
2022/04/25 13:02:01 Searching for existing private keys
2022/04/25 13:02:02 ----- sealed-secrets-key5nhjv
2022/04/25 13:02:02 ----- sealed-secrets-key98m56
2022/04/25 13:02:02 ----- sealed-secrets-keyztj4v
2022/04/25 13:02:02 ----- acme-keys
2022/04/25 13:02:02 HTTP server serving on :8080
Enter fullscreen mode Exit fullscreen mode
  1. Encrypt with new keys
❯ kubectl create secret generic secret-name --dry-run=client --from-literal=foo=bar -o yaml | kubeseal --controller-name=ss-app-sealed-secrets --controller-namespace=sealed-secrets -o yaml --cert=$PUBLICKEY | kubectl apply -f -
sealedsecret.bitnami.com/secret-name created
Enter fullscreen mode Exit fullscreen mode
  1. Verify decryption
❯ kubectl logs ss-app-sealed-secrets-556c68c858-gqpwf -n sealed-secrets | tail -2
2022/04/25 13:08:07 Updating default/secret-name
2022/04/25 13:08:07 Event(v1.ObjectReference{Kind:"SealedSecret", Namespace:"default", Name:"secret-name", UID:"c9ee00ac-c974-43f7-9b7d-16cda1b516ff", APIVersion:"bitnami.com/v1alpha1", ResourceVersion:"57973", FieldPath:""}): type: 'Normal' reason: 'Unsealed' SealedSecret unsealed successfully

❯ kubectl get sealedsecrets.bitnami.com
NAME          AGE
secret-name   47s

❯ kubectl get secrets secret-name
NAME          TYPE     DATA   AGE
secret-name   Opaque   1      57s

Enter fullscreen mode Exit fullscreen mode
Toying with the secret rotation interval during creation of the sealed-secrets controller
  • Edit --key-renew-period=<value> in the command line args for the controller (in-case of deployment)
  • Use --set=keyrenewperiod=<value> while installing through Helm

Note: keyrenewperiod=0 stops keys from being rotated; though seemingly easy solution, not recommended in production use-cases

Billboard image

Get alerted faster with Checkly

Join Vercel, Render, LinkedIn, and thousands of other teams that rely on Checkly to keep their websites and applications running smoothly with proactive monitoring, instant alerts, and comprehensive full-stack tracing.

Start Monitoring

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Dive into an ocean of knowledge with this thought-provoking post, revered deeply within the supportive DEV Community. Developers of all levels are welcome to join and enhance our collective intelligence.

Saying a simple "thank you" can brighten someone's day. Share your gratitude in the comments below!

On DEV, sharing ideas eases our path and fortifies our community connections. Found this helpful? Sending a quick thanks to the author can be profoundly valued.

Okay