External Secrets Operator + HashiCorp Vault: GitOps Secret Lifecycle in Kubernetes

Originally published at woitzik.dev

Kubernetes Secrets are not secret.

base64 is not encryption. Anyone with kubectl get secret access can decode them instantly. Secrets stored in etcd are encrypted at rest only if you've explicitly configured encryption providers — and most clusters haven't. And if you're managing secrets in Git (even with SOPS or Sealed Secrets), the ciphertext is committed to version control forever.

The proper solution is an external secret store: a system specifically designed for secret storage, with access control, audit logging, and rotation built in. HashiCorp Vault is the most common choice. External Secrets Operator bridges Vault to Kubernetes — syncing secrets into the cluster without storing them in Git.

This post covers the full setup running on my k3s cluster: Vault deployment, bootstrap sequence, ClusterSecretStore, and the first real ExternalSecret.

View the complete homelab infrastructure source on GitHub 🐙

The Architecture

Git (no secrets)
      ↓ ArgoCD syncs
  Vault (KV v2)          ←── You store secrets here
      ↑
External Secrets Operator
      ↓ creates/syncs
  k8s Secret (in-cluster, not in Git)
      ↓ consumed by
  Application Pod

The key property: nothing sensitive is ever committed to Git. ArgoCD manages all Kubernetes manifests except Secrets. Vault holds the actual values. ESO syncs them into the cluster on a refresh interval. Applications consume k8s.io/v1/Secret objects as normal — nothing changes from the application's perspective.

Step 1: Deploy Vault via ArgoCD

# kubernetes/system/vault/application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://helm.releases.hashicorp.com
    targetRevision: 0.28.1
    chart: vault
    helm:
      values: |
        server:
          standalone:
            enabled: true
          dataStorage:
            enabled: true
            size: 5Gi
            storageClass: nfs-client
        ui:
          enabled: true
          serviceType: ClusterIP
  destination:
    server: https://kubernetes.default.svc
    namespace: vault
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

NFS-backed storage for the Vault data directory. Vault runs as a single-node instance (standalone mode) — sufficient for a homelab, and it avoids the complexity of Raft consensus with multiple replicas.

Step 2: The Bootstrap Sequence

Vault ships sealed. Before it can serve any secrets, you must unseal it. This is a one-time manual operation — by design. Unsealing requires a quorum of key shares, so no single compromise can unlock the store.

# 1. Wait for the Vault pod to be running
kubectl wait --for=condition=Ready pod/vault-0 -n vault --timeout=120s

# 2. Initialize Vault (5 key shares, 3 required to unseal)
kubectl exec vault-0 -n vault -- vault operator init \
  -key-shares=5 \
  -key-threshold=3

# Output (save these — you cannot recover them):
# Unseal Key 1: abc...
# Unseal Key 2: def...
# Unseal Key 3: ghi...
# Unseal Key 4: jkl...
# Unseal Key 5: mno...
# Initial Root Token: hvs.xxx...

# 3. Unseal with 3 of the 5 keys
kubectl exec vault-0 -n vault -- vault operator unseal <key-1>
kubectl exec vault-0 -n vault -- vault operator unseal <key-2>
kubectl exec vault-0 -n vault -- vault operator unseal <key-3>

# 4. Verify unsealed
kubectl exec vault-0 -n vault -- vault status
# Sealed: false

Store the unseal keys and root token somewhere secure. Losing them means losing access to your Vault permanently. A password manager with hardware 2FA (Vaultwarden, 1Password, Bitwarden) works. Do not commit them to Git.

After a Vault pod restart (node reboot, update), you need to unseal again with 3 keys. Auto-unseal via AWS KMS or Azure Key Vault removes this manual step in production environments — acceptable for a homelab to skip.

Step 3: Enable KV v2 and Write Your First Secret

# Authenticate with the root token
kubectl exec -it vault-0 -n vault -- /bin/sh
vault login <root-token>

# Enable KV v2 at the 'secret/' path
vault secrets enable -path=secret kv-v2

# Write the first secret
vault kv put secret/authelia \
  hmac-secret="$(openssl rand -base64 32)"

# Verify
vault kv get secret/authelia

KV v2 maintains version history. You can roll back to a previous version of a secret, see who wrote what and when (with audit logging enabled), and compare versions. This is what makes Vault appropriate for compliance contexts — it's not just a secret store, it's a secret lifecycle management system.

Step 4: Deploy External Secrets Operator

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.10.4
    chart: external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Step 5: Create the Token Secret and ClusterSecretStore

ESO needs a way to authenticate against Vault. The simplest approach for a single-cluster setup: a Kubernetes secret containing the Vault root token (or a scoped AppRole token for production).

# Create the token secret that ESO will use to authenticate against Vault
kubectl create secret generic vault-token \
  -n external-secrets \
  --from-literal=token=<vault-root-token>

Then the ClusterSecretStore:

# kubernetes/system/external-secrets/cluster-secret-store.yml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: http://vault.vault.svc.cluster.local:8200
      path: secret
      version: v2
      auth:
        tokenSecretRef:
          name: vault-token
          namespace: external-secrets
          key: token

ClusterSecretStore (vs SecretStore) is cluster-scoped — any namespace can reference it. For multi-tenant clusters where namespaces shouldn't cross-read each other's secrets, use namespace-scoped SecretStore instead.

The path: secret and version: v2 match the KV mount we created in step 3.

Step 6: The First ExternalSecret

# kubernetes/apps/authelia/external-secret.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: authelia-secrets
  namespace: apps
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: authelia-secrets
    creationPolicy: Merge
  data:
    - secretKey: hmac-secret
      remoteRef:
        key: secret/authelia
        property: hmac-secret

Three things worth noting:

refreshInterval: 1h — ESO re-reads from Vault every hour. If you rotate the secret in Vault, the k8s Secret is updated within an hour. No pod restart required for most applications that read secrets from mounted files (as opposed to environment variables, which require a restart).

creationPolicy: Merge — Instead of creating a new Secret from scratch, ESO merges the Vault-sourced key into an existing Secret. This is useful when a Secret needs some values from Vault (sensitive) and others from a ConfigMap (non-sensitive). The application sees a single unified Secret.

remoteRef.key — The full Vault path is secret/data/authelia (KV v2 prepends data/), but ESO handles the /data/ prefix automatically when version: v2 is set. You write secret/authelia in the ExternalSecret.

Verifying It Works

# Check the ExternalSecret sync status
kubectl get externalsecret authelia-secrets -n apps

# Output:
# NAME               STORE   REFRESH INTERVAL   STATUS         READY
# authelia-secrets   vault   1h                 SecretSynced   True

# Check the resulting k8s Secret
kubectl get secret authelia-secrets -n apps -o jsonpath='{.data.hmac-secret}' | base64 -d

If STATUS shows SecretSyncedError, check:

1. kubectl describe externalsecret authelia-secrets -n apps for the error message
2. Vault pod is running and unsealed (kubectl exec vault-0 -n vault -- vault status)
3. The token secret exists in the external-secrets namespace
4. The KV path actually exists in Vault (vault kv get secret/authelia)

What You Get

• Audit log: Every secret read from Vault is logged. vault audit enable file file_path=/vault/logs/audit.log gives you a full trail of who (which token) accessed what secret and when.
• Rotation without redeployment: Rotate a secret in Vault, ESO syncs it within the refresh interval. For file-mounted secrets, the pod picks it up without restart.
• No secrets in Git: The ExternalSecret manifest commits to Git. It describes what to sync and where from — but not the value. The value stays in Vault.
• Compliance evidence: KV v2 version history + audit log gives you the access evidence ISO 27001 (A.9.4 — System and Application Access Control) and NIS2 require.

Next: AppRole Authentication

The setup above uses the Vault root token for ESO authentication. That works, but the root token has unrestricted access to everything in Vault.

For a more hardened setup, create a Vault AppRole with a policy scoped to only the secrets ESO needs:

# Policy: ESO can only read under secret/data/
vault policy write eso-readonly - <<EOF
path "secret/data/*" {
  capabilities = ["read"]
}
EOF

# AppRole
vault auth enable approle
vault write auth/approle/role/eso \
  policies="eso-readonly" \
  token_ttl=1h \
  token_max_ttl=4h

# Get role_id and secret_id for ESO
vault read auth/approle/role/eso/role-id
vault write -f auth/approle/role/eso/secret-id

Update the ClusterSecretStore to use AppRole auth instead of tokenSecretRef. This follows the principle of least privilege — a compromise of the ESO token only exposes read access to secrets, not root-level Vault control.