DEV Community

Sergey Shinder
Sergey Shinder

Posted on

Our pod sent the words DB_PASSWORD as its database password because we declared the password one line too late

Our returns service reads one environment variable to find its database, DATABASE_URL. It used to come whole from a Secret. In October we split it, because the password rotates and the host does not: DB_PASSWORD comes from the Secret with secretKeyRef, and DATABASE_URL is composed in the Deployment as postgresql://returns:$(DB_PASSWORD)@pg-returns:5432/returns.

It worked for a week. Then a tidy-up pull request sorted every env list in our manifests alphabetically, for easier review. DATABASE_URL sorts before DB_PASSWORD. The new pods crashed on start with password authentication failed for user returns. The rolling update stopped there and the old pods kept serving, so no customer noticed, but it took us forty minutes to see why.

Kubernetes expands $(NAME) in an env value using only the variables defined before it in that container's list, plus service variables. If it cannot resolve a reference, it does not fail and does not warn. It leaves the text exactly as written. With DB_PASSWORD now further down, the application received a URL whose password was the fourteen characters $(DB_PASSWORD), and Postgres rightly refused it.

Nothing we normally look at showed it. kubectl get pod -o yaml prints the manifest as written, which looked correct. The Secret was correct. The pods crashed too fast to exec into. We found it by starting a copy of the pod with its command replaced by env, and there was the literal reference in the middle of the URL.

The order of an env list had never been something anyone reviewed, and the sort was a reasonable change. So we stopped depending on it. The service now reads the host, user and password as separate variables and builds the URL itself, so there is nothing left for Kubernetes to expand. For the few other manifests that still compose values, a CI step renders them and fails on any $(NAME) that refers to a variable not defined earlier in the same container. The escape is worth knowing too: $$ becomes a single $, if you ever need the text itself.

Kubernetes does not tell you when a reference stays unresolved. Ours went to the database as a password, and the database was the only thing that noticed.

– Sergey Shinder

Top comments (0)