DEV Community: whiteSama001

Common Problems Running HashiCorp Vault on Kubernetes (and How to Actually Fix Them)

whiteSama001 — Sun, 06 Sep 2026 11:11:06 +0000

Vault and Kubernetes are both excellent at hiding the actual cause of a
failure behind a generic-looking error. Put them together — Vault
injecting secrets into pods via a sidecar, authenticating pods via
Kubernetes service account tokens — and you get failure modes that look
identical from the outside ("pod stuck in Init," "502 from the app,"
"permission denied") but come from very different root causes. Below are
the problems that come up most often, roughly in the order I'd check
them.

1. The Agent Injector silently doesn't inject

Symptom: you've annotated the pod spec with
vault.hashicorp.com/agent-inject: "true", but the pod comes up with no
vault-agent-init or vault-agent container at all — no error, it just
doesn't happen.

This is almost always one of:

The mutating webhook isn't reaching the pod's namespace. Check whether the namespace has a label that a namespaceSelector on the webhook is excluding (common when someone added kube-system-style exclusions broadly and it caught more than intended):

  kubectl get mutatingwebhookconfiguration vault-agent-injector-cfg -o yaml
  kubectl get namespace <ns> --show-labels

The injector pod itself isn't healthy. kubectl -n vault logs deploy/vault-agent-injector-injector will usually show TLS cert errors here if the injector's self-signed cert (used for the webhook callback) has expired or the CA bundle in the webhook config is stale.
The annotation is on the wrong object. vault.hashicorp.com/* annotations must be on the pod template (spec.template.metadata in a Deployment), not on the Deployment's own metadata. This is the single most common cause of "I annotated it and nothing happened."

2. Vault Agent sidecar is running, but the secret file never appears

If the sidecar container is present and running but the app container
never sees /vault/secrets/<name>, check the sidecar's own logs first
— not the app's:

kubectl logs <pod> -c vault-agent

Typical causes:

Template rendering failure — a typo in the agent-inject-template annotation (wrong path, wrong Go template syntax) fails silently from the app's perspective but errors clearly in the agent's own log.
Policy doesn't grant read on the secret path — Vault Agent authenticates fine but gets a 403 on the actual secret read. This looks like "nothing happens" unless you specifically check the agent log for the permission denied response.
Wrong secret engine version assumed in the template — KV v2 paths need a data/ segment (secret/data/myapp/config) that KV v1 doesn't. Copying a template from a v1 setup into a v2-backed Vault is a classic silent-failure source.

3. Kubernetes auth method: intermittent "permission denied" after it worked fine for weeks

This one is almost always the Kubernetes 1.21+ bound service account
token change. Since Kubernetes moved to time-bound, audience-scoped
service account tokens (as opposed to the old long-lived tokens
auto-mounted from a Secret), Vault's Kubernetes auth backend needs to be
configured to either:

Use iss validation compatible with your cluster's actual token issuer (kubectl get --raw /.well-known/openid-configuration to confirm what the cluster is actually issuing), or
Use a bound, non-expiring token explicitly created for Vault's own reviewer service account, configured via kubernetes_ca_cert / token_reviewer_jwt in the auth config.

If this was configured correctly at setup time but breaks later — often
weeks in — it's frequently because the reviewer JWT itself was a
short-lived token that expired, not a config drift issue. Check:

vault read auth/kubernetes/config

and confirm the token_reviewer_jwt isn't sitting on a token that has
since expired.

4. Vault pods CrashLoopBackOff after a node restart or cluster upgrade

If you're running Vault in-cluster (not as a managed/external
service) with Raft integrated storage, this is usually Vault coming back
up sealed, and whatever auto-unseal mechanism you configured not
being reachable yet at boot:

Cloud KMS auto-unseal (AWS KMS / Azure Key Vault / GCP KMS): check that the pod's identity (IRSA, workload identity, or managed identity) has permission to the key and that there's no chicken-and-egg problem where the identity provider itself isn't ready yet during a full cluster bootstrap.
Manual unseal: if you're still using unseal keys held by operators, a restart means someone has to run vault operator unseal three times per pod. This doesn't scale past a handful of pods and is usually the point at which teams move to cloud auto-unseal.

Distinguish "sealed" from "actually crashing" before debugging further:

kubectl exec -it vault-0 -- vault status

A sealed-but-otherwise-fine Vault reports its seal status cleanly; a
genuinely crashing pod won't even get that far, and the real error is in
kubectl logs — often a Raft storage corruption or disk permission
issue after a node replacement changed the underlying PV's ownership.

5. Leases expiring under load, and apps not handling renewal

Vault Agent handles token and lease renewal for you — if the
application actually re-reads the rendered secret file instead of
caching the value in memory at startup. This is an application-level bug
that shows up as a Vault problem: the secret rotates correctly, the file
on disk updates correctly, and the app keeps using the old database
password anyway because it read it into memory once at boot and never
looked at the file again.

The fix isn't in Vault — it's making sure the app either:

Watches the secret file for changes (inotify or a polling read), or
Is restarted/rolled by a process that reacts to vault.hashicorp.com/agent-inject-command firing on template change (a common pattern is having the sidecar send SIGHUP to the app process, or having the injector template out a checksum the app polls).

Worth checking explicitly during any "why did prod start throwing auth
errors overnight" investigation, since it doesn't show up in Vault's own
audit log as an error at all — from Vault's side, everything renewed
successfully.

6. NetworkPolicy quietly blocking Vault API calls

If Vault Agent can't reach the Vault API at all, the error is usually
obvious (connection refused/timeout in the agent log). But partially
restrictive NetworkPolicy resources cause a subtler failure: init
succeeds (maybe egress to Vault's ClusterIP is allowed) but subsequent
renewal calls fail if a policy is scoped too narrowly to only allow
traffic during pod startup timing, or scoped to the wrong port when
Vault is listening on a non-default port behind a Kubernetes Service.
Confirm the actual policy match, not just its presence:

kubectl get networkpolicy -n <ns> -o yaml
kubectl exec <pod> -c vault-agent -- wget -qO- https://vault.vault.svc:8200/v1/sys/health

7. Injector webhook timeout under cluster load

The mutating webhook has a timeoutSeconds (default 30s in most Helm
chart versions, but drops to lower defaults in some setups) — under
cluster-wide load or if the injector pod itself is under-resourced, pod
creation can start failing cluster-wide with admission webhook denied the request errors that have nothing to do with the actual pod spec.
This tends to surface during mass rollouts or node scale-up events,
which makes it easy to misdiagnose as a scheduling problem rather than a
webhook capacity problem. Give the injector pod real resource
requests/limits and consider running more than one replica behind its
Service — it's a single point of failure for all pod creation in
namespaces it covers, not just Vault-using ones, since the webhook fires
on every pod create by default unless scoped with a
namespaceSelector/objectSelector.

The pattern underneath all of this

Almost every one of these failures looks, from the app team's side,
like "the app can't reach the database" or "the pod won't start." The
actual cause is almost never in the application code — it's in the
handshake between three systems (Kubernetes' identity model, Vault's
auth/policy model, and the injector's webhook mechanics) that only
becomes visible if you check each layer's own logs independently
instead of inferring backward from the app's symptom. The fastest path
through any of these, in practice, is:

kubectl describe pod — did the webhook even fire, did the sidecar get injected.
Vault Agent's own log — did auth succeed, did the template render.
vault status / vault read auth/kubernetes/config — is Vault itself healthy and is the auth backend actually configured the way you think it is.

Only after ruling those out is it worth looking at the application
itself — and even then, per #5 above, the bug is often "the app isn't
watching the file," not anything Vault did wrong.

Curious which of these has bitten you — the token-issuer change from
Kubernetes 1.21+ is the one I've seen catch out the most teams who
configured Vault correctly once and never revisited it.

Debugging Redis Remote Access on RHEL: A Four-Layer Problem That Looked Like One

whiteSama001 — Sun, 06 Sep 2026 10:34:00 +0000

When a request comes in as "the app can't connect to Redis," it's tempting to assume it's a one-line fix — open a port, flip a config flag, done. In practice, on a hardened RHEL/CentOS host, a single "connection refused" error can be hiding four separate problems stacked on top of each other: network-level firewall rules, SELinux policy, file-level permissions, and Redis's own safety defaults. Each layer will silently swallow the connection and give you almost no useful signal about which one is actually blocking you.

This is a walkthrough of how I diagnosed and fixed exactly that scenario — a Redis instance that needed to accept connections from an application server on the same internal subnet, but wouldn't.

The symptom

A service running on a separate host in the same internal network needed to read/write to a Redis instance. The redis-cli -h <redis-host> ping command from the app server just hung, then timed out. No obvious error in the Redis logs. systemctl status redis showed it running happily on the Redis host itself, and redis-cli ping worked fine locally on that box.

That local-works, remote-fails split is the first useful clue: it rules out a broken Redis process and points squarely at something in the path between the two hosts, or in how Redis is bound.

Layer 1: Is Redis even listening on the right interface?

The default Redis config binds to 127.0.0.1 only — deliberately, since Redis has no authentication by default and was never meant to be exposed carelessly. First check:

sudo ss -tlnp | grep redis

If you only see 127.0.0.1:6379, that's your answer for this layer. The fix is in redis.conf:

bind 0.0.0.0 -::1
# or, more safely, bind to the specific internal interface:
bind 127.0.0.1 10.0.x.x

Restart the service and re-check with ss. If the socket now shows the right interface, move to the network layer — this alone rarely fixes remote access on a hardened host.

Layer 2: The firewall

RHEL-family systems default to firewalld. Even if the app team swears "the network team already opened the port," verify it yourself:

sudo firewall-cmd --list-all

If 6379/tcp isn't in the services or ports list for the active zone, add it:

sudo firewall-cmd --zone=internal --add-port=6379/tcp --permanent
sudo firewall-cmd --reload

Don't forget any cloud-level security group or NSG sitting in front of the host — firewalld being correct doesn't mean the packet ever arrives if a cloud ACL drops it first. Test from the app server with:

nc -zv <redis-host> 6379

If that now succeeds but redis-cli ping still hangs or gets refused, the packet is arriving — the block is happening inside the host, which is where most people stop looking too early.

Layer 3: SELinux — the layer everyone forgets

This is the one that trips people up because it fails silently and looks identical to a network problem from the outside. SELinux enforces policy independently of standard file/network permissions, and a denial doesn't show up in the application's own logs — it shows up in the audit log.

Check for denials:

sudo ausearch -m avc -ts recent

or, more readably:

sudo sealert -a /var/log/audit/audit.log

Two common findings in this scenario:

Port context: if you changed Redis to listen on a non-default port, SELinux won't know that port is allowed for the redis_port_t type until you tell it:

   sudo semanage port -a -t redis_port_t -p tcp <new_port>

File context on data/config paths: if the Redis data directory or config file was moved, copied, or restored from a backup in a way that didn't preserve SELinux labels, the daemon can be denied access to its own files:

   sudo restorecon -Rv /var/lib/redis /etc/redis

Never disable SELinux to make this go away — setenforce 0 "fixes" the symptom by removing a security boundary you presumably want on a production host. Diagnose the specific denial and allow exactly that.

Layer 4: File ownership

Related but distinct from SELinux context: standard Unix ownership. If the Redis data directory, RDB/AOF files, or config file aren't owned by the redis user (often because of a manual cp or a restore run as root), the daemon can fail to start cleanly or fail to persist — which then surfaces upstream as flaky connectivity when the process restarts unexpectedly.

sudo chown -R redis:redis /var/lib/redis
sudo chmod 750 /var/lib/redis

Layer 5: Redis's own protected mode

Even once the network, SELinux, and ownership are all correct, Redis has one more guard rail: protected mode. If Redis is bound to a non-loopback address without a password configured, it refuses external connections by default and logs a warning about it. This is Redis protecting you from accidentally exposing an unauthenticated instance to the world.

The correct fix is not to disable protected mode — it's to set a password:

requirepass <strong-password>

Then connect as:

redis-cli -h <redis-host> -a <strong-password> ping

If for some internal-only, tightly firewalled reason you genuinely don't want auth, you can explicitly set protected-mode no — but treat that as a deliberate, documented exception, not a quick fix.

Why this matters beyond Redis

The specific commands here are Redis-and-RHEL-flavored, but the debugging shape generalizes to almost any "service X can't talk to service Y" ticket on a hardened Linux estate:

Confirm the process is actually listening where you think it is (ss/netstat).
Confirm the packet can physically arrive (firewall, security groups, routing).
Confirm the kernel-level mandatory access control isn't silently dropping it (SELinux/AppArmor — check the audit log, don't guess).
Confirm file/process ownership is correct.
Only then look at the application's own safety defaults.

Skipping straight to "just open the port" or "just disable SELinux" will often look like it worked in a rushed fix, while actually just removing a control someone put there on purpose. Working through the layers in order — and confirming each one with a command, not an assumption — gets you to a fix you can actually explain and defend later, which matters a lot more in a regulated environment (this happened on banking infrastructure) than in a hobby project.

A couple of open questions I'd flag to anyone in a similar spot

Two things I left as follow-ups rather than closing out immediately, because they're policy decisions, not bugs:

dir setting: where RDB/AOF snapshots actually land matters for backup and disk-capacity planning — worth confirming it points somewhere with monitored, appropriately-sized storage rather than leaving the default.
Eviction policy: maxmemory-policy needs to match how the application actually uses Redis (cache vs. durable store). Getting this wrong either silently drops data you needed or lets Redis OOM under load.

Neither is a "fix" — they're the kind of thing worth a short design conversation with whoever owns the data before you consider the job fully done.

If you've hit a similar multi-layer wall with Redis, Postgres, or any other service on a hardened RHEL box, I'd be curious to hear which layer got you — SELinux is my usual bet.