I lost access to a SealedSecrets key once , not because I deleted it, but because I didn't know where it was stored. The cluster kept running, the apps kept deploying, but the moment I tried to rotate the key or redeploy a sealed secret, I hit a wall. The controller couldn't decrypt anything. The only way out was to find the original key, and I had to dig through old manifests and cluster logs to get it back. That’s when I learned the hard way: SealedSecrets keys aren’t magical. They’re just Kubernetes secrets, and they can be lost if you don’t back them up.
The SealedSecrets controller uses a single key to encrypt and decrypt secrets. If that key is lost, all your sealed secrets become unusable. You can’t just regenerate it , the encryption is tied to that specific key. The key is stored as a Kubernetes secret in the sealed-secrets namespace. If you don’t back it up, and it gets deleted or corrupted, you're out of luck.
Here’s the command I use to back it up. It exports the key to a YAML file, which I store off-cluster in version control or a secure backup system:
kubectl get secret sealed-secrets-key -n sealed-secrets -o yaml > sealed-secrets-key-backup.yaml
This is the only way to ensure you can recover from a key loss. If you're using GitOps tools like ArgoCD, make sure this backup is part of your repo and included in your CI/CD pipeline. Otherwise, the moment you redeploy the sealed-secrets controller, the key could be lost if it's not versioned.
If you lose the key, the only way to recover is to restore it from a backup. You can do that by applying the YAML file back into the cluster. Just make sure the namespace and secret name match the original. If you're using ArgoCD, you may need to disable the sealed-secrets app, apply the key, and then re-enable it to avoid reconciliation conflicts.
Don’t assume the key is safe just because it's in the cluster. Back it up, version it, and keep it somewhere you can get to when you need it. That’s the only way to stay ahead of a potential outage.
Top comments (0)