Common Problems Running HashiCorp Vault on Kubernetes (and How to Actually Fix Them)

Vault and Kubernetes are both excellent at hiding the actual cause of a
failure behind a generic-looking error. Put them together — Vault
injecting secrets into pods via a sidecar, authenticating pods via
Kubernetes service account tokens — and you get failure modes that look
identical from the outside ("pod stuck in Init," "502 from the app,"
"permission denied") but come from very different root causes. Below are
the problems that come up most often, roughly in the order I'd check
them.

1. The Agent Injector silently doesn't inject

Symptom: you've annotated the pod spec with
vault.hashicorp.com/agent-inject: "true", but the pod comes up with no
vault-agent-init or vault-agent container at all — no error, it just
doesn't happen.

This is almost always one of:

• The mutating webhook isn't reaching the pod's namespace. Check whether the namespace has a label that a namespaceSelector on the webhook is excluding (common when someone added kube-system-style exclusions broadly and it caught more than intended):

  kubectl get mutatingwebhookconfiguration vault-agent-injector-cfg -o yaml
  kubectl get namespace <ns> --show-labels

• The injector pod itself isn't healthy. kubectl -n vault logs deploy/vault-agent-injector-injector will usually show TLS cert errors here if the injector's self-signed cert (used for the webhook callback) has expired or the CA bundle in the webhook config is stale.
• The annotation is on the wrong object. vault.hashicorp.com/* annotations must be on the pod template (spec.template.metadata in a Deployment), not on the Deployment's own metadata. This is the single most common cause of "I annotated it and nothing happened."

2. Vault Agent sidecar is running, but the secret file never appears

If the sidecar container is present and running but the app container
never sees /vault/secrets/<name>, check the sidecar's own logs first
— not the app's:

kubectl logs <pod> -c vault-agent

Typical causes:

• Template rendering failure — a typo in the agent-inject-template annotation (wrong path, wrong Go template syntax) fails silently from the app's perspective but errors clearly in the agent's own log.
• Policy doesn't grant read on the secret path — Vault Agent authenticates fine but gets a 403 on the actual secret read. This looks like "nothing happens" unless you specifically check the agent log for the permission denied response.
• Wrong secret engine version assumed in the template — KV v2 paths need a data/ segment (secret/data/myapp/config) that KV v1 doesn't. Copying a template from a v1 setup into a v2-backed Vault is a classic silent-failure source.

3. Kubernetes auth method: intermittent "permission denied" after it worked fine for weeks

This one is almost always the Kubernetes 1.21+ bound service account
token change. Since Kubernetes moved to time-bound, audience-scoped
service account tokens (as opposed to the old long-lived tokens
auto-mounted from a Secret), Vault's Kubernetes auth backend needs to be
configured to either:

• Use iss validation compatible with your cluster's actual token issuer (kubectl get --raw /.well-known/openid-configuration to confirm what the cluster is actually issuing), or
• Use a bound, non-expiring token explicitly created for Vault's own reviewer service account, configured via kubernetes_ca_cert / token_reviewer_jwt in the auth config.

If this was configured correctly at setup time but breaks later — often
weeks in — it's frequently because the reviewer JWT itself was a
short-lived token that expired, not a config drift issue. Check:

vault read auth/kubernetes/config

and confirm the token_reviewer_jwt isn't sitting on a token that has
since expired.

4. Vault pods CrashLoopBackOff after a node restart or cluster upgrade

If you're running Vault in-cluster (not as a managed/external
service) with Raft integrated storage, this is usually Vault coming back
up sealed, and whatever auto-unseal mechanism you configured not
being reachable yet at boot:

• Cloud KMS auto-unseal (AWS KMS / Azure Key Vault / GCP KMS): check that the pod's identity (IRSA, workload identity, or managed identity) has permission to the key and that there's no chicken-and-egg problem where the identity provider itself isn't ready yet during a full cluster bootstrap.
• Manual unseal: if you're still using unseal keys held by operators, a restart means someone has to run vault operator unseal three times per pod. This doesn't scale past a handful of pods and is usually the point at which teams move to cloud auto-unseal.

Distinguish "sealed" from "actually crashing" before debugging further:

kubectl exec -it vault-0 -- vault status

A sealed-but-otherwise-fine Vault reports its seal status cleanly; a
genuinely crashing pod won't even get that far, and the real error is in
kubectl logs — often a Raft storage corruption or disk permission
issue after a node replacement changed the underlying PV's ownership.

5. Leases expiring under load, and apps not handling renewal

Vault Agent handles token and lease renewal for you — if the
application actually re-reads the rendered secret file instead of
caching the value in memory at startup. This is an application-level bug
that shows up as a Vault problem: the secret rotates correctly, the file
on disk updates correctly, and the app keeps using the old database
password anyway because it read it into memory once at boot and never
looked at the file again.

The fix isn't in Vault — it's making sure the app either:

• Watches the secret file for changes (inotify or a polling read), or
• Is restarted/rolled by a process that reacts to vault.hashicorp.com/agent-inject-command firing on template change (a common pattern is having the sidecar send SIGHUP to the app process, or having the injector template out a checksum the app polls).

Worth checking explicitly during any "why did prod start throwing auth
errors overnight" investigation, since it doesn't show up in Vault's own
audit log as an error at all — from Vault's side, everything renewed
successfully.

6. NetworkPolicy quietly blocking Vault API calls

If Vault Agent can't reach the Vault API at all, the error is usually
obvious (connection refused/timeout in the agent log). But partially
restrictive NetworkPolicy resources cause a subtler failure: init
succeeds (maybe egress to Vault's ClusterIP is allowed) but subsequent
renewal calls fail if a policy is scoped too narrowly to only allow
traffic during pod startup timing, or scoped to the wrong port when
Vault is listening on a non-default port behind a Kubernetes Service.
Confirm the actual policy match, not just its presence:

kubectl get networkpolicy -n <ns> -o yaml
kubectl exec <pod> -c vault-agent -- wget -qO- https://vault.vault.svc:8200/v1/sys/health

7. Injector webhook timeout under cluster load

The mutating webhook has a timeoutSeconds (default 30s in most Helm
chart versions, but drops to lower defaults in some setups) — under
cluster-wide load or if the injector pod itself is under-resourced, pod
creation can start failing cluster-wide with admission webhook denied
the request errors that have nothing to do with the actual pod spec.
This tends to surface during mass rollouts or node scale-up events,
which makes it easy to misdiagnose as a scheduling problem rather than a
webhook capacity problem. Give the injector pod real resource
requests/limits and consider running more than one replica behind its
Service — it's a single point of failure for all pod creation in
namespaces it covers, not just Vault-using ones, since the webhook fires
on every pod create by default unless scoped with a
namespaceSelector/objectSelector.

The pattern underneath all of this

Almost every one of these failures looks, from the app team's side,
like "the app can't reach the database" or "the pod won't start." The
actual cause is almost never in the application code — it's in the
handshake between three systems (Kubernetes' identity model, Vault's
auth/policy model, and the injector's webhook mechanics) that only
becomes visible if you check each layer's own logs independently
instead of inferring backward from the app's symptom. The fastest path
through any of these, in practice, is:

1. kubectl describe pod — did the webhook even fire, did the sidecar get injected.
2. Vault Agent's own log — did auth succeed, did the template render.
3. vault status / vault read auth/kubernetes/config — is Vault itself healthy and is the auth backend actually configured the way you think it is.

Only after ruling those out is it worth looking at the application
itself — and even then, per #5 above, the bug is often "the app isn't
watching the file," not anything Vault did wrong.

---

Curious which of these has bitten you — the token-issuer change from
Kubernetes 1.21+ is the one I've seen catch out the most teams who
configured Vault correctly once and never revisited it.