DEV Community

Metronom
Metronom

Posted on

Kubernetes Secrets Are base64, Not Encrypted: How We Leaked a DB Password to Git

Kubernetes Secrets are base64-encoded, not encrypted. We learned that the expensive way: a Secret manifest with a real PostgreSQL password sat in git for months because "it's base64, it's fine." It is not fine. base64 is encoding, not a security control. Here's the leak and the setup that makes it impossible to repeat.

Symptom

secret.yaml with our DB password, base64-encoded, committed to the repo. A new hire on day two ran the reversal out of curiosity:

$ echo 'czNjcjN0' | base64 -d
s3cr3t
Enter fullscreen mode Exit fullscreen mode

Production-adjacent DB password, plaintext, one command, from a value that had been in git history for months. Nobody had done anything malicious. The value was simply readable by every person and CI job that had ever cloned the repo, and by anyone who found the repo later. A leak is not an event, it's a state — and we'd been sitting in it silently the entire time.

Root cause

Straight from the Kubernetes Secrets docs: Secrets are stored unencrypted by default in the API server's data store (etcd) — just base64-encoded. Their contents are readable by anyone with API read access, etcd access, or the ability to create Pods in that namespace. "It's a Secret object" does not mean "it's encrypted."

Worse: deleting the file didn't help. The value stayed in git history. Deleting a committed secret does not un-leak it — we treated the password as fully compromised and rotated it. There's a solid first-principles breakdown of separating config from code at this configuration & secrets writeup.

The fix

1. Right object for the right data

We'd dumped everything — log level, DB host, and the password — into one place. The split:

  • ConfigMap for non-sensitive settings (log level, DB host/port/name). No secrecy, 1 MiB limit, same namespace as the Pod.
  • Secret for passwords, keys, tokens.

Use stringData so you never hand-encode base64:

# secret.yaml (NEVER committed with real values)
apiVersion: v1
kind: Secret
metadata:
  name: myapp-db-secret
  namespace: myapp
type: Opaque
stringData:
  DB_USER: "myapp"
  DB_PASSWORD: "s3cr3t"
Enter fullscreen mode Exit fullscreen mode

2. Templates + .gitignore

The cheapest fix, the one we should've had on day one: commit a template with placeholders, keep the real file out of git.

# .gitignore
secret.yaml
*.secret.yaml
.env
Enter fullscreen mode Exit fullscreen mode 
# secret.example.yaml -- safe to commit
stringData:
  DB_USER: "CHANGE_ME"
  DB_PASSWORD: "CHANGE_ME"
Enter fullscreen mode Exit fullscreen mode

Clone, copy secret.example.yaml to secret.yaml, fill in real values, git never sees them. That's the whole thing we were missing.

3. Sealed Secrets for GitOps

We did want desired cluster state in git for GitOps — which means encrypting the secret before committing, real encryption this time. Bitnami's Sealed Secrets: a controller with a key pair lives in the cluster, kubeseal encrypts your Secret with the public key into a SealedSecret, and only the controller can decrypt it, inside the cluster:

kubectl -n myapp create secret generic myapp-db-secret \
  --from-literal=DB_PASSWORD='s3cr3t' \
  --dry-run=client -o yaml \
  | kubeseal -o yaml > sealed-db-secret.yaml
# safe to commit, even to a public repo
Enter fullscreen mode Exit fullscreen mode

Two things on the wall: the encryption is tied to the namespace + name pair, so a SealedSecret won't decrypt under a different one; and you must back up the controller's private key, or every committed SealedSecret becomes unreadable garbage.

4. dev/prod values without duplicate manifests

Local uses a stub password and LOG_LEVEL=debug; prod a real password and LOG_LEVEL=info. Copying manifests into two folders is how they drift. Kustomize's configMapGenerator has a content-based hash suffix in the generated name. Change the config, the name changes (myapp-config-7c8f...), Kustomize updates the Deployment reference, Kubernetes does a rolling update on its own:

kubectl apply -k overlays/dev
Enter fullscreen mode Exit fullscreen mode

That also fixed a subtler bug: env vars are fixed at Pod startup and don't update when you change a ConfigMap. We'd edited a ConfigMap and stared at a Pod running old values. The hash suffix solves it automatically; without it you need kubectl rollout restart deployment/myapp.

Feeding values into a Pod — and the reload trap

Creating the objects is half the job; you still deliver the values, three ways, each with different reload behavior:

  • env + valueFrom — one key under a specific variable name. Precise, verbose.
  • envFrom — every key at once, optionally with a prefix.
  • volume mount — each key becomes a file (TLS certs, runtime config).

The trap, stated plainly in the ConfigMap docs: env vars are fixed at Pod startup and never update when the underlying ConfigMap/Secret changes — the Pod runs old values until kubectl rollout restart. A volume mount updates automatically (small kubelet sync delay), but only mounted without subPath, and the app still has to re-read the file. We lost an hour editing a ConfigMap and expecting the running Pod to notice. It never did.

The rotation runbook we actually ran

Once we understood the leak, the response was mechanical. Order matters — rotate before you rewrite history, because the old value is already out.

  1. Treat the credential as compromised. Rotate it at the source first: 
# new password in Postgres, then update the (uncommitted) Secret
psql -c "ALTER USER myapp WITH PASSWORD 's0m3-new-value';"
kubectl -n myapp create secret generic myapp-db-secret \
  --from-literal=DB_PASSWORD='s0m3-new-value' \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/myapp -n myapp
Enter fullscreen mode Exit fullscreen mode
  1. Confirm where else the old value leaked: 
git log -p -- secret.yaml | grep -i password   # every historical value, in the open
Enter fullscreen mode Exit fullscreen mode
  1. Purge it from history (git filter-repo), force-push, and rotate anything that shared it. The purge is cosmetic — the rotation in step 1 is the control that actually protects you.

Before / after

Before After
DB password base64 in git history rotated; never committed
Config vs secrets all mixed together ConfigMap + Secret split
Local secrets committed templates + .gitignore
GitOps secrets impossible / unsafe Sealed Secrets (encrypted)
dev/prod values copy-pasted manifests Kustomize overlays + generators

Guardrail

  • base64 is not protection. A Secret with a real value in git equals a leaked value.
  • Deleting the file doesn't help — git history keeps it; rotate.
  • env doesn't hot-reload a ConfigMap/Secret change; a volume mount does (without subPath), but the app still re-reads the file.
  • A missing referenced ConfigMap/Secret stops the Pod from starting unless you mark the source optional: true. A typo in a name became a Pod that never scheduled, with a confusing error, until we learned to read it.
  • Turn on encryption at rest and lock down RBAC in any cluster that holds a real secret. The docs' own remedy is exactly that.

What I'd do differently

Put the .gitignore in the repo template on day zero, and run a pre-commit secret scanner so no secret.yaml with real values can ever be staged. For local k3d, templates plus .gitignore are genuinely enough — Sealed Secrets are overkill there — but the same repo flows to CI and prod, so the direction has to be right from the start.

Bottom line: a Kubernetes Secret in git is a leaked secret, because base64 is encoding not encryption — commit templates, seal anything that must live in git, and rotate the moment a real value touches a commit.

Sources

Top comments (0)