Our session store is Redis, deployed from an internal Helm chart. The chart generates its own password the first time it is installed. The Secret template asks the cluster whether the Secret already exists, using Helm's lookup function, and if it does it reuses the password inside. If not, it creates one with randAlphaNum. Our pipeline ran helm upgrade for two years and the password never changed.
In October we moved the platform charts to Argo CD. The first sync went through clean. Over the next day, every time we merged anything to the deployment repository, a few customers were logged out, and the session service's logs filled with WRONGPASS from Redis whenever one of its pods restarted.
Argo CD does not run helm install. It renders charts with helm template and applies the result itself. Helm only talks to the cluster during a real install or upgrade, and when it renders without one, lookup returns an empty result. To our chart that meant the Secret did not exist, so every render generated a new password. Argo CD caches rendered manifests by commit, so a fresh password arrived with each new commit to the repository, and Argo CD applied it. Redis had read its password from the Secret when it started and kept the old one. Session service pods that restarted read the new one and were refused. Pods that had not restarted kept working, which is why it looked random.
We pinned the password that afternoon by writing the live value into the Secret by hand and turning off automatic sync for that one application. The chart no longer generates secrets at all. The password is created once in our secrets manager, and an External Secrets resource writes it into the cluster, so rendering the chart produces the same output every time. A CI job now renders every chart twice with helm template and fails if the two outputs differ, which catches any template that depends on randomness or on what the cluster contains.
A chart that asks the cluster a question is only a chart when there is a cluster to answer. Argo CD never asked it, and our template answered on the cluster's behalf.
– Sergey Shinder
Top comments (0)