The config change that didn't restart anything

Noam Tzuberi — Wed, 04 Nov 2026 10:12:32 +0000

You change a value in your Helm chart. helm upgrade succeeds. The ConfigMap in the cluster shows the new value. And your app keeps behaving exactly as before.

Nothing failed — that's the problem. Kubernetes did precisely what it was told.

Why it happens

A Deployment only rolls its pods when the pod template changes. A ConfigMap is a separate object. When you update it, the pods already running don't notice:

Values injected as environment variables are read once, at container start. They stay old until the pod restarts.
Values mounted as files do update eventually, but only if your app re-reads the file. Most apps read config once at boot.

So the upgrade "worked", the config is "deployed", and production runs the old settings until something else happens to restart the pods — a node drain, a crash, the next image release. Then the change appears hours or days later, and nobody connects it to the upgrade.

Why AI-written charts hit this often

Ask an assistant for a Helm chart and you get a clean Deployment plus a ConfigMap, wired through envFrom. It is correct YAML and it passes every lint. The missing piece isn't a syntax error, it's a lifecycle detail — the kind of thing that only shows up in production.

The fix: make the config part of the pod template

Add a checksum of the ConfigMap as a pod annotation:

spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Now any config change changes the annotation, the pod template changes, and Kubernetes does a normal rolling update. Same for Secrets — add a second checksum.

Three checks I now do on every chart

Every ConfigMap and Secret the app reads has a checksum annotation on the Deployment.
After helm upgrade, check that new pods actually started (kubectl rollout status), not just that the release succeeded.
Log the effective config (minus secrets) at startup, so "which value is running?" takes one log search.

The broader lesson

AI tools are good at producing configuration that looks complete. The gaps are in what happens over time: restarts, rollouts, caches, retries. That's where I spend my review time.