DEV Community

Florient DOGBE
Florient DOGBE

Posted on

Kubernetes Secrets: Getting Out of base64 with External Secrets Operator and Vault

#kubernetes #vault #security #devops

A Secret object in Kubernetes is not encrypted. It is base64-encoded. Every DevOps engineer knows this, yet the single most common finding in cluster audits is still the same: credentials committed to Git in a values.yaml, or a kubectl create secret run by hand six months ago that nobody can rotate because nobody remembers what the value was.

This article walks through a production-grade alternative: HashiCorp Vault as the source of truth, External Secrets Operator (ESO) as the synchronization controller, Kubernetes ServiceAccount tokens as the authentication mechanism, and etcd encryption as the last line of defense. No static Vault token stored in a cluster Secret, no credential in Git, and a full audit trail of who read what.

The problem: base64 is an encoding, not a control

What actually sits in etcd

By default, the API server writes Secret objects to etcd as plain serialized data. Anyone with an etcd backup, a snapshot on an S3 bucket, or read access to the data directory on a control plane node gets every credential in the cluster:

# On a control-plane node, with etcdctl available
etcdctl get /kubernetes.io/secrets/payments/payments-api-db \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key | strings
Enter fullscreen mode Exit fullscreen mode

The password is right there. Same thing for any velero backup of the namespace, any kubectl get secret -o yaml dumped into a ticket, any GitOps repository holding the manifest.

Why the usual workarounds fall short

Three patterns are common and each has a real limitation:

  • SealedSecrets: encrypted at rest in Git, good. But rotation means re-encrypting and re-committing, the controller's private key becomes a single point of failure, and you still have no audit of consumption.
  • SOPS + age/KMS: same encryption-in-Git model, better key management, still a Git commit for every rotation and no central inventory of secrets.
  • Vault Agent Injector sidecars: strong, but it couples every workload to Vault, adds a sidecar per pod, and files on tmpfs do not work for anything expecting a Secret reference (imagePullSecrets, TLS in an Ingress, operator-consumed credentials).

External Secrets Operator takes a different position: the secret lives in Vault, the operator reconciles a native Kubernetes Secret from it, and the application keeps using envFrom or a volume mount as usual. Git contains only a reference — a path, never a value.

Target architecture 

Vault (KV v2 + database engine)
      ▲  auth/kubernetes login (SA JWT, audience=vault)
      │
External Secrets Operator (controller in external-secrets ns)
      │  reconciles every refreshInterval
      ▼
Kubernetes Secret (namespace payments) ──► Deployment envFrom
Enter fullscreen mode Exit fullscreen mode

The only credential that ever transits is a short-lived, audience-bound ServiceAccount token that ESO requests from the API server via the TokenRequest API. No Vault token is stored anywhere in the cluster.

Step 1 — Configure Kubernetes auth in Vault

Enable a dedicated auth mount per cluster so you can revoke one cluster without touching the others.

export VAULT_ADDR=https://vault.internal.example.com:8200

vault auth enable -path=kubernetes-prod kubernetes
Enter fullscreen mode Exit fullscreen mode

If Vault runs outside the cluster, it needs to validate incoming JWTs against the cluster's API server. Create a dedicated reviewer ServiceAccount with the system:auth-delegator role:

oc create namespace vault-auth
oc create serviceaccount vault-token-reviewer -n vault-auth
oc adm policy add-cluster-role-to-user system:auth-delegator \
  -z vault-token-reviewer -n vault-auth
Enter fullscreen mode Exit fullscreen mode 
apiVersion: v1
kind: Secret
metadata:
  name: vault-token-reviewer-token
  namespace: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-token-reviewer
type: kubernetes.io/service-account-token
Enter fullscreen mode Exit fullscreen mode 
oc apply -f vault-token-reviewer-token.yaml

REVIEWER_JWT=$(oc get secret vault-token-reviewer-token -n vault-auth \
  -o jsonpath='{.data.token}' | base64 -d)
K8S_CA=$(oc get secret vault-token-reviewer-token -n vault-auth \
  -o jsonpath='{.data.ca\.crt}' | base64 -d)
K8S_HOST=$(oc whoami --show-server)

vault write auth/kubernetes-prod/config \
  kubernetes_host="${K8S_HOST}" \
  kubernetes_ca_cert="${K8S_CA}" \
  token_reviewer_jwt="${REVIEWER_JWT}" \
  disable_local_ca_jwt=true
Enter fullscreen mode Exit fullscreen mode

Now the policy. Scope it to one application path, read-only, including the KV v2 metadata/ path so ESO can list keys when you use dataFrom:

vault policy write payments-read - <<'EOF'
path "secret/data/prod/payments/*" {
  capabilities = ["read"]
}
path "secret/metadata/prod/payments/*" {
  capabilities = ["read", "list"]
}
EOF
Enter fullscreen mode Exit fullscreen mode

And the role, bound to one ServiceAccount in one namespace, with a custom audience so a token minted for Vault cannot be replayed against the API server:

vault write auth/kubernetes-prod/role/payments-api \
  bound_service_account_names=eso-payments \
  bound_service_account_namespaces=payments \
  audience=vault \
  token_policies=payments-read \
  token_ttl=20m \
  token_max_ttl=1h
Enter fullscreen mode Exit fullscreen mode

Seed a test secret:

vault kv put secret/prod/payments/postgres \
  username=payments_app \
  password="$(openssl rand -base64 24)"
Enter fullscreen mode Exit fullscreen mode

Step 2 — Install External Secrets Operator 

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm upgrade --install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --set installCRDs=true \
  --set replicaCount=2 \
  --set webhook.replicaCount=2 \
  --set certController.replicaCount=2
Enter fullscreen mode Exit fullscreen mode

On OpenShift the default chart runs cleanly under the restricted-v2 SCC — do not set securityContext.runAsUser, let the namespace annotation assign the UID range.

oc get pods -n external-secrets
oc get crd | grep external-secrets.io
Enter fullscreen mode Exit fullscreen mode

You should see secretstores, clustersecretstores, externalsecrets, clusterexternalsecrets, pushsecrets.

ESO ≥ 0.14 serves external-secrets.io/v1. On older installs, replace v1 with v1beta1 in every manifest below; the schema is identical.

Step 3 — Declare the SecretStore

The ServiceAccount ESO impersonates lives in the application namespace:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eso-payments
  namespace: payments
Enter fullscreen mode Exit fullscreen mode

ESO's controller needs permission to mint a token for it. The Helm chart grants this cluster-wide by default; if you locked it down, add:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eso-token-creator
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts/token"]
    resourceNames: ["eso-payments"]
    verbs: ["create"]
Enter fullscreen mode Exit fullscreen mode

Then the store itself — namespaced, so a team cannot point at another team's Vault path:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-payments
  namespace: payments
spec:
  provider:
    vault:
      server: https://vault.internal.example.com:8200
      path: secret
      version: v2
      caProvider:
        type: ConfigMap
        name: vault-ca
        key: ca.crt
        namespace: payments
      auth:
        kubernetes:
          mountPath: kubernetes-prod
          role: payments-api
          serviceAccountRef:
            name: eso-payments
            audiences:
              - vault
Enter fullscreen mode Exit fullscreen mode 
oc apply -f secretstore.yaml
oc get secretstore vault-payments -n payments \
  -o jsonpath='{.status.conditions[0].reason}{"\n"}'
# Expected: Valid
Enter fullscreen mode Exit fullscreen mode

If you get permission denied, check the audience first — a mismatch between the Vault role audience and serviceAccountRef.audiences is the number one cause.

Step 4 — The ExternalSecret 

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-api-db
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-payments
    kind: SecretStore
  target:
    name: payments-api-db
    creationPolicy: Owner
    deletionPolicy: Delete
    template:
      engineVersion: v2
      type: Opaque
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@pg.payments.svc:5432/payments?sslmode=require"
        PGPASSWORD: "{{ .password }}"
  data:
    - secretKey: username
      remoteRef:
        key: prod/payments/postgres
        property: username
    - secretKey: password
      remoteRef:
        key: prod/payments/postgres
        property: password
Enter fullscreen mode Exit fullscreen mode

creationPolicy: Owner sets an ownerReference on the generated Secret: delete the ExternalSecret and the Secret disappears with it. The template block is what makes ESO practical — applications rarely want raw key/value pairs, they want a connection string, a .pgpass, or a kubeconfig.

To pull a whole KV entry without listing every key:

  dataFrom:
    - extract:
        key: prod/payments/postgres
Enter fullscreen mode Exit fullscreen mode

Verify:

oc get externalsecret payments-api-db -n payments
# NAME              STORE            REFRESH   STATUS         READY
# payments-api-db   vault-payments   1h        SecretSynced   True

oc get secret payments-api-db -n payments -o jsonpath='{.data.DATABASE_URL}' | base64 -d
Enter fullscreen mode Exit fullscreen mode

Git now contains the ExternalSecret manifest. It holds a Vault path, a role name and a template. Nothing confidential.

Rotation that actually rotates

Static secrets

refreshInterval controls how often ESO re-reads Vault. One hour is a reasonable default; for high-value credentials, drop to 5m. Rotating a password becomes a single Vault write:

vault kv put secret/prod/payments/postgres \
  username=payments_app \
  password="$(openssl rand -base64 24)"
Enter fullscreen mode Exit fullscreen mode

Force an immediate reconcile instead of waiting:

oc annotate externalsecret payments-api-db -n payments \
  force-sync="$(date +%s)" --overwrite
Enter fullscreen mode Exit fullscreen mode

Dynamic credentials

The real win is Vault's database secrets engine: ESO fetches a freshly generated database user on every refresh.

vault write database/roles/payments-app \
  db_name=postgres-prod \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT payments_rw TO \"{{name}}\";" \
  default_ttl=1h \
  max_ttl=24h
Enter fullscreen mode Exit fullscreen mode 
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-api-db-dynamic
  namespace: payments
spec:
  refreshInterval: 30m
  secretStoreRef:
    name: vault-payments-db
    kind: SecretStore
  target:
    name: payments-api-db-dynamic
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: database/creds/payments-app
Enter fullscreen mode Exit fullscreen mode

Making pods notice

A Secret update does not restart pods. Env vars from envFrom are frozen at container start; volume-mounted Secrets are refreshed by the kubelet within ~60s but the app must re-read the file. Two clean options:

# Option A: Stakater Reloader watches the Secret and rolls the Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
  annotations:
    secret.reloader.stakater.com/reload: "payments-api-db"
spec:
  template:
    spec:
      containers:
        - name: api
          image: registry.example.com/payments/api:1.14.2
          envFrom:
            - secretRef:
                name: payments-api-db
Enter fullscreen mode Exit fullscreen mode

Option B, preferable for dynamic credentials: mount the Secret as a volume and have the application reload the file on a SIGHUP or on connection failure.

Auditing who reads what

Enable a Vault audit device — this is the piece SealedSecrets and SOPS cannot give you:

vault audit enable file file_path=/vault/audit/vault-audit.log log_raw=false
Enter fullscreen mode Exit fullscreen mode

Every ESO login and read is logged with the ServiceAccount identity:

jq -r 'select(.request.path | startswith("secret/data/prod/payments"))
       | [.time, .auth.display_name, .request.path, .request.operation]
       | @tsv' /vault/audit/vault-audit.log
Enter fullscreen mode Exit fullscreen mode 
2025-03-11T09:14:02Z  kubernetes-prod-eso-payments  secret/data/prod/payments/postgres  read
Enter fullscreen mode Exit fullscreen mode

Secret values are HMAC'd in the log, so the audit trail itself is not a secret store. On the cluster side, make sure the API server audit policy records reads of Secret objects at Metadata level at minimum — on OpenShift, oc patch apiserver cluster --type=merge -p '{"spec":{"audit":{"profile":"WriteRequestBodies"}}}'.

Encrypting etcd: defense in depth

ESO removes secrets from Git, not from etcd. The generated Secret is still a regular object. Enable encryption at rest.

On OpenShift it is a one-line change, handled by the operators with automatic key rotation:

oc patch apiserver cluster --type=merge \
  -p '{"spec":{"encryption":{"type":"aescbc"}}}'

oc get openshiftapiserver cluster \
  -o jsonpath='{range .status.conditions[?(@.type=="Encrypted")]}{.reason}: {.message}{"\n"}{end}'
Enter fullscreen mode Exit fullscreen mode

On vanilla Kubernetes, use a KMS v2 provider so the data encryption key is never on the node:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
      - configmaps
    providers:
      - kms:
          apiVersion: v2
          name: vault-kms
          endpoint: unix:///opt/kms/vault-kms.sock
          timeout: 3s
      - identity: {}
Enter fullscreen mode Exit fullscreen mode

Add --encryption-provider-config=/etc/kubernetes/enc/enc.yaml and --encryption-provider-config-automatic-reload=true to the API server, then rewrite existing objects:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -
Enter fullscreen mode Exit fullscreen mode

Security best practices

  • One SecretStore per namespace, one Vault role per application. ClusterSecretStore is convenient and dangerous: a single misconfigured policy exposes every path to every namespace. Use it only for genuinely shared material (internal CA bundle, registry pull secret) with a read-only policy.
  • Bind roles on both name and namespace. bound_service_account_namespaces=* turns the whole cluster into a single trust boundary.
  • Always set a custom audience. Without it, any component able to read a default projected token could replay it against Vault.
  • Keep token TTLs short (token_ttl=20m). ESO re-authenticates transparently on every reconcile.
  • Never grant create/update on Vault paths unless you actually use PushSecret. Read-only is the default posture.
  • Restrict get secrets with RBAC. ESO is pointless if every developer has view on the namespace — view includes Secret contents in many legacy ClusterRole bindings. Audit them: oc adm policy who-can get secrets -n payments.
  • Pin deletionPolicy: Delete so an orphaned Secret cannot outlive its ExternalSecret and become an untracked credential.
  • Enable etcd encryption and encrypt backups. An unencrypted Velero bucket undoes everything above.
  • Alert on SecretSyncedError. A silently failing ExternalSecret means a stale credential; scrape externalsecret_status_condition from the ESO metrics endpoint and alert after two consecutive failures.
  • Scan Git anyway. Add gitleaks or trufflehog to CI — the operator does not stop someone from pasting a token into a ConfigMap.

Key takeaways

Base64 is a transport encoding, and treating it as protection is how credentials end up in public repositories. The fix is architectural, not cosmetic: move the source of truth out of the cluster, keep only references in Git, and let a controller reconcile short-lived material into native Secrets.

The combination described here gives you four properties that are hard to get otherwise. Git holds zero secret values, so a repository leak is a topology disclosure, not a breach. Authentication uses ephemeral, audience-scoped ServiceAccount tokens, so there is no long-lived Vault token to steal or rotate. Rotation is a single vault kv put or, better, automatic via dynamic credentials with a one-hour TTL. And every read is attributed to a named Kubernetes identity in the Vault audit log.

Start small: one namespace, one SecretStore, one ExternalSecret, with the legacy Secret left in place until the new one verifies. Then enable etcd encryption, delete the old manifests from Git history with git filter-repo, and rotate everything that was ever committed — because it is already compromised.

Top comments (0)