DEV Community

Guy Weitzman
Guy Weitzman

Posted on

Bootstrapping HashiCorp Vault on Kubernetes the production way

#kubernetes #vault #devops #security

Every "Vault on Kubernetes" tutorial gets you to a running pod in ten minutes. Almost none of them tell you that the pod you just started will lose every secret the moment it restarts, that its Ready probe is lying to you, or that the auth setup you copied grants every workload in the cluster access to every secret in Vault.

I run Vault on Kubernetes in production. The gap between a demo install and something you can put on-call for is almost entirely in the parts the tutorials skip: the unseal strategy, the identity model for workloads, the policy layout for multiple teams, and how you rotate a secret without a redeploy. The rest of this is the actual commands and output for each of those, and a checklist at the end to measure your own cluster against.

Everything below was run on a local kind cluster (Kubernetes 1.29.14) with the official HashiCorp Vault Helm chart 0.34.0 (Vault 2.0.3). No dev mode, no shortcuts that don't survive a restart.

Why dev mode is a trap

The single most common way people get Vault onto Kubernetes is:

helm install vault hashicorp/vault --set "server.dev.enabled=true"
Enter fullscreen mode Exit fullscreen mode

Dev mode boots Vault already initialized, already unsealed, with an in-memory storage backend and a fixed root token. It is genuinely useful for a five-minute demo. It is also a trap, because it teaches you a mental model that is wrong in three ways that matter in production:

  1. It is unsealed automatically. Real Vault comes up sealed. Its storage is encrypted with a key that Vault itself does not have on disk. Until you provide the unseal material, Vault can start, serve health checks, and do nothing else. Dev mode hides the single most important operational fact about Vault.
  2. Storage is in memory. Restart the pod and every secret, policy, and auth mount is gone. You will never notice in a demo and you will always notice in production.
  3. The root token is printed to the log. Fine for a laptop, a disaster anywhere else.

So we skip dev mode. Everything dev mode does for you automatically is the part you actually have to get right in production.

Deploying the chart (without dev mode)

Here is the install, and the one design decision baked into it. I run a single Vault node backed by integrated storage (Raft). Raft is what HashiCorp recommends for HA today, it keeps Vault's data on a PersistentVolume so it survives restarts, and it scales to a 3-node quorum by bumping one number. On a laptop I run one replica so the unseal flow is demonstrable without three times the memory. The unseal mechanics are identical at one node or three.

values.yaml:

global:
  tlsDisable: true          # TLS is off for this walkthrough; see the checklist. false in prod

server:
  standalone:
    enabled: false
  ha:
    enabled: true
    replicas: 1
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable     = 1
          address         = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        storage "raft" {
          path = "/vault/data"
        }
        service_registration "kubernetes" {}
  dataStorage:
    enabled: true
    size: 1Gi

injector:
  enabled: false            # keep the focus on the k8s auth method itself
Enter fullscreen mode Exit fullscreen mode 
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --version 0.34.0 --values values.yaml
Enter fullscreen mode Exit fullscreen mode

A few seconds later the pod is running but not ready:

$ kubectl get pods -n vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   0/1     Running   0          6s
Enter fullscreen mode Exit fullscreen mode

That 0/1 is intentional. Vault's readiness probe reports not ready while it is sealed, and it is sealed because we have not initialized it yet. If you look at what the probe is actually checking, Vault tells you exactly what state it is in:

Readiness probe failed:
Key                     Value
---                     -----
Seal Type               shamir
Initialized             false
Sealed                  true
Version                 2.0.3
Storage Type            raft
HA Enabled              true
Enter fullscreen mode Exit fullscreen mode

Initialized: false, Sealed: true. Kubernetes will keep this pod out of the Service endpoints until Vault is unsealed, which means a sealed Vault takes itself out of rotation instead of serving errors. That's the behavior you want. Now initialize it.

Initializing and the unseal strategy

Initialization generates Vault's root key (the key that ultimately protects storage) and returns the unseal key split into shares using Shamir's Secret Sharing, plus a root token. The shares reconstruct the unseal key, which decrypts the root key, which decrypts everything else. The two numbers you choose here are a real operational decision:

kubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=5 -key-threshold=3 -format=json
Enter fullscreen mode Exit fullscreen mode

-key-shares=5 -key-threshold=3 means the key is split into 5 shares and any 3 of them reconstruct it. This is the classic setup for a reason: you can distribute one share each to five trusted operators, tolerate losing two of them (a lost laptop, someone on leave), and still never let a single person unseal Vault alone. One share is useless; three is a quorum. Choose these numbers to match your on-call reality, not the tutorial's.

I ask for JSON output so the result is machine-parseable. The structure looks like this (values redacted - losing these means losing the data):

{
  "unseal_keys_b64":       [ "<share 1>", "<share 2>", "...", "<share 5>" ],
  "unseal_keys_hex":       [ "..." ],
  "unseal_shares":         5,
  "unseal_threshold":      3,
  "recovery_keys_b64":     [],
  "root_token":            "hvs.<redacted>"
}
Enter fullscreen mode Exit fullscreen mode

This output appears exactly once and is never recoverable. If you lose all copies of enough shares to reach the threshold, the data in Vault is cryptographically gone - there is no reset. In production these shares should be encrypted to individual operators' PGP keys (vault operator init supports -pgp-keys) so that no plaintext share ever touches the operator's terminal or your CI logs. For this walkthrough they live in a scratch file that is deleted with the cluster.

Now the pod is initialized but still sealed. We unseal by feeding it shares one at a time. Watch the progress counter:

$ vault operator unseal <share 1>
Sealed              true
Unseal Progress     1/3

$ vault operator unseal <share 2>
Sealed              true
Unseal Progress     2/3

$ vault operator unseal <share 3>
Sealed              false
HA Enabled          true
HA Mode             active
Enter fullscreen mode Exit fullscreen mode

The third share crosses the threshold, Vault reconstructs the unseal key in memory, uses it to decrypt the root key, decrypts its storage, and comes alive. Confirm:

$ vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Storage Type            raft
Cluster Name            vault-cluster-4aa18d6b
HA Mode                 active
Active Since            2026-07-22T09:44:29Z
Raft Applied Index      38
Enter fullscreen mode Exit fullscreen mode

And now the pod flips to ready on its own:

$ kubectl get pods -n vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   1/1     Running   0          8m
Enter fullscreen mode Exit fullscreen mode

What "restart-safe" actually costs you

Here is the fact that dev mode hides and that you must design around: Shamir unseal is manual. The reconstructed key lives only in the pod's memory. Delete the pod - a node reboot, an eviction, a chart upgrade, a spot instance reclaim - and Vault comes back sealed. Watch it happen:

$ kubectl delete pod vault-0 -n vault
pod "vault-0" deleted

$ kubectl get pod vault-0 -n vault
NAME      READY   STATUS    RESTARTS   AGE
vault-0   0/1     Running   0          6s

$ vault status
Initialized             true      <- data survived on the Raft PVC
Sealed                  true      <- but it will NOT unseal itself
Unseal Progress         0/3
Enter fullscreen mode Exit fullscreen mode

Initialized: true (the data persisted, thanks to Raft on the PV) but Sealed: true. Until a human - or automation - supplies three shares again, every secret in Vault is unreadable and every workload depending on it is broken. Re-supplying the three shares brings it straight back:

$ vault operator unseal <share 1>   # 1/3
$ vault operator unseal <share 2>   # 2/3
$ vault operator unseal <share 3>
Sealed        false
HA Mode       active
Enter fullscreen mode Exit fullscreen mode

This is the crux of your unseal-strategy decision, and it is a real fork:

  • Manual Shamir + operators on call - most secure (no external system holds your key), most painful (a 3 AM reboot pages a human to type shares).
  • KMS auto-unseal - a seal stanza points at a cloud KMS (AWS KMS, GCP Cloud KMS, Azure Key Vault) or a Transit engine on another Vault. The KMS wraps Vault's root key directly; Vault unwraps it on boot and a restarted pod unseals itself in seconds with no human.

Under auto-unseal there is no Shamir unseal key at all - that is the whole point. Instead Vault issues recovery keys (also Shamir-split) that authorize break-glass operations like root-token generation and rekey, rather than unsealing. That is also why the recovery_keys_b64 array was empty in our Shamir init above: it only populates under auto-unseal. Migrating an already-initialized Shamir cluster to a KMS seal is a deliberate, one-node-at-a-time operation using vault operator unseal -migrate, after which the old unseal shares become the recovery keys.

Auto-unseal needs cloud credentials, so it is out of scope to configure here - but decide between these two before you go to production, not after your first unplanned reboot. The seal stanza goes in the same raft config block shown above.

Workload identity: the Kubernetes auth method

Vault is now running, unsealed, and restart-aware. It holds nothing useful yet, and no workload can talk to it. The next three sections build that out: how a pod proves who it is, what each identity is allowed to read, and how you rotate what is inside.

A workload should never carry a long-lived Vault token in an environment variable or a mounted file. The whole reason to run Vault on Kubernetes is that Kubernetes already issues every pod a verifiable identity - its ServiceAccount token - and Vault can trust that identity directly. That is the Kubernetes auth method: a pod presents its ServiceAccount JWT, Vault asks the Kubernetes API "is this token real?" via the TokenReview API, and if so it mints a short-lived Vault token scoped to whatever policy you mapped that ServiceAccount to.

Enable it and point Vault at the in-cluster API server:

$ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

$ vault write auth/kubernetes/config \
    kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443
Success! Data written to: auth/kubernetes/config
Enter fullscreen mode Exit fullscreen mode

I passed no CA certificate and no reviewer token, on purpose. Running in-cluster, Vault reads the CA and its own ServiceAccount token from the standard in-pod mount (/var/run/secrets/kubernetes.io/serviceaccount/) and uses its own token as the reviewer JWT to call TokenReview. Reading the config back confirms it:

$ vault read auth/kubernetes/config
Key                        Value
---                        -----
disable_local_ca_jwt       false
kubernetes_host            https://10.96.0.1:443
token_reviewer_jwt_set     false            <- no reviewer token passed; Vault uses its own
Enter fullscreen mode Exit fullscreen mode

For Vault to be allowed to call TokenReview, its own ServiceAccount needs the system:auth-delegator ClusterRole. The Helm chart wires this up for you - it is worth knowing it exists, because if you deploy Vault without the chart this is the RBAC people forget:

$ kubectl get clusterrolebindings | grep vault
vault-server-binding   ClusterRole/system:auth-delegator   19m
Enter fullscreen mode Exit fullscreen mode

Policy layout for a multi-team cluster

This is where most setups quietly become insecure. It is tempting to write one policy that grants broad access and map every workload to it. Do that and a compromised pod in one team can read every other team's secrets. The fix is path-scoped, least-privilege policies, one per workload identity, laid out along a path convention you enforce cluster-wide.

Pick a path layout up front. A common one is secret/data/<team>/<app>/.... Then each team's policy only ever references its own subtree. Here is a read-only policy for a payments team:

# payments-ro.hcl
path "secret/data/payments/*" {
  capabilities = ["read"]
}
path "secret/metadata/payments/*" {
  capabilities = ["read", "list"]
}
Enter fullscreen mode Exit fullscreen mode

Two details that trip people up on the KV v2 engine:

  • The API path has data/ (and metadata/) injected into it - secret/payments/db on the CLI is really secret/data/payments/db in a policy. Get this wrong and your reads 403 with a policy that looks correct.
  • Read and list live on separate paths. A workload that reads a known key needs only data/...; a workload that enumerates keys needs list on metadata/.... Grant the narrower set. 
$ vault policy write payments-ro payments-ro.hcl
Success! Uploaded policy: payments-ro

$ vault kv put secret/payments/db username=payments_app password=s3cr3t-from-vault
======= Secret Path =======
secret/data/payments/db
Key                Value
---                -----
version            1
Enter fullscreen mode Exit fullscreen mode

Now the binding - a Vault role in the Kubernetes auth mount that says "the ServiceAccount payments-api in namespace apps gets the payments-ro policy, and its tokens live for 20 minutes":

$ vault write auth/kubernetes/role/payments-api \
    bound_service_account_names=payments-api \
    bound_service_account_namespaces=apps \
    policies=payments-ro \
    audience=vault \
    ttl=20m
Success! Data written to: auth/kubernetes/role/payments-api
Enter fullscreen mode Exit fullscreen mode

bound_service_account_names and bound_service_account_namespaces are the security boundary. Only that exact ServiceAccount in that exact namespace can assume this role. A pod in a different namespace, or the same namespace with a different ServiceAccount, is rejected before any secret is in play. This is how you keep team A out of team B's secrets: their identities map to different roles, and each role maps to a policy scoped to a different path.

Proving it from a workload

Now run a pod as the payments-api ServiceAccount and have it authenticate with only its projected token, then read the secret.

apiVersion: v1
kind: Pod
metadata:
  name: payments-api
  namespace: apps
spec:
  serviceAccountName: payments-api
  containers:
    - name: app
      image: hashicorp/vault:2.0.3
      command: ["sh", "-c", "sleep 3600"]
      env:
        - name: VAULT_ADDR
          value: "http://vault.vault.svc:8200"
      volumeMounts:
        - name: vault-token
          mountPath: /var/run/secrets/tokens
  volumes:
    - name: vault-token
      projected:
        sources:
          - serviceAccountToken:
              path: vault-token
              audience: vault           # must match the role's audience
              expirationSeconds: 600
Enter fullscreen mode Exit fullscreen mode

I use a projected ServiceAccount token with an explicit audience: vault rather than the legacy auto-mounted token. Audience-bound, short-lived tokens are the current best practice: the token is only valid for Vault, and it expires in ten minutes whether or not anyone uses it.

Inside that pod, the login is one call. The workload reads its own token off disk and exchanges it:

$ JWT=$(cat /var/run/secrets/tokens/vault-token)
$ vault write auth/kubernetes/login role=payments-api jwt="$JWT"
Enter fullscreen mode Exit fullscreen mode

Vault verifies the JWT against the Kubernetes API and returns a token. The metadata on that token is the proof that identity flowed correctly end to end (client token redacted):

{
  "auth": {
    "client_token": "<redacted>",
    "policies": ["default", "payments-ro"],
    "metadata": {
      "role": "payments-api",
      "service_account_name": "payments-api",
      "service_account_namespace": "apps"
    },
    "lease_duration": 1200,
    "renewable": true
  }
}
Enter fullscreen mode Exit fullscreen mode

policies: [default, payments-ro], stamped with the ServiceAccount's own name and namespace, leased for 1200 seconds (the 20m we set). Now use it to read:

$ export VAULT_TOKEN=$(vault write -field=token auth/kubernetes/login role=payments-api jwt="$JWT")

$ vault kv get secret/payments/db
====== Data ======
Key         Value
---         -----
password    s3cr3t-from-vault
username    payments_app
Enter fullscreen mode Exit fullscreen mode

The workload never held a Vault credential of its own. It presented a Kubernetes identity and got exactly the access we granted that identity, nothing more. The denied case confirms it:

$ vault kv get secret/other-team/db
Error reading secret/data/other-team/db: Error making API request.
URL: GET http://vault.vault.svc:8200/v1/secret/data/other-team/db
Code: 403. Errors:

* permission denied
Enter fullscreen mode Exit fullscreen mode

A 403 permission denied on a path outside its policy. The boundary is enforced by Vault, not by convention.

Safe secret rotation

In KV v2 a write creates a new version instead of overwriting the old one, which is what makes rotation safe to do while workloads are live. Writing a new value:

vault kv put secret/payments/db username=payments_app password=<new-password>
Enter fullscreen mode Exit fullscreen mode

...creates version 2 and leaves version 1 intact and readable at -version=1. The safe rotation sequence for a database credential is therefore:

  1. Provision the new credential on the backing system (create the new DB password) before touching Vault.
  2. vault kv put the new value - now version 2 exists, version 1 still works.
  3. Let workloads pick up version 2. If they read Vault on a schedule or on restart, this is automatic; if they cache at startup, roll them.
  4. Only once nothing is using version 1, revoke the old credential on the backing system.

Because both versions coexist, there is no window where the secret is half-rotated and something is broken. And if the new value is bad, vault kv rollback -version=1 restores the previous one instantly.

Better still: stop rotating static secrets by hand. Vault's dynamic secrets engines (database, AWS, PKI, and more) generate a fresh, short-lived credential per lease and revoke it automatically when the lease ends. A workload asks for a database credential, gets one that is valid for an hour, and Vault deletes it afterward. So instead of rotating a shared password every 90 days and hoping nothing cached it, every workload gets its own credential that expires in an hour. Static KV is the right tool for genuinely static config; dynamic secrets are the right tool for anything that grants access to another system.

The production checklist

If you are holding a Vault install up against production, this is the line between a demo and something you can put on-call for:

  • [ ] Not dev mode. No server.dev.enabled. Vault comes up sealed and persists to a real volume.
  • [ ] Integrated storage (Raft) or a real backend, on a PersistentVolume, so restarts keep their data. For real HA, 3 or 5 Raft nodes, not 1.
  • [ ] An unseal strategy you have decided on purpose: manual Shamir with shares distributed to operators (PGP-encrypted at init), or KMS auto-unseal for hands-off restarts. Test a pod delete and confirm it recovers the way you expect.
  • [ ] Unseal shares and the root token stored offline and split, never in CI logs or a wiki. The root token is revoked after setup (vault token revoke) and re-minted only when needed.
  • [ ] TLS enabled end to end (global.tlsDisable=false with real certs). This walkthrough left it off to stay focused; production does not.
  • [ ] Kubernetes auth method for workload identity - no static Vault tokens in pods. Projected ServiceAccount tokens with an explicit audience.
  • [ ] One least-privilege policy per workload identity, scoped to a path convention (secret/data/<team>/<app>/...). No shared broad-grant policy. Verify a cross-team read returns 403.
  • [ ] Short token TTLs on auth roles so a leaked token expires on its own.
  • [ ] Audit device enabled (vault audit enable) so every access is logged - the first thing an incident responder will ask for.
  • [ ] Rotation done additively (KV v2 versions), and dynamic secrets engines for anything that grants access to another system.

Standing Vault up is the easy part. The unseal strategy, the identity model, the policy boundaries, and the rotation discipline are what decide whether it holds up at 3 AM. Get those right and you stop hoping Vault survives the next node reboot.

Top comments (0)