A support ticket said our dispatch emails were going out twice. Not always, not to any customer we could pick out, and never while we watched. Twenty four pods, one image, one deployment, one ConfigMap, and six of those pods were running a value we had changed in June.
The value is a lease: how long a worker holds a job before the queue decides it has died and gives the job to someone else. In June we cut it from ten minutes to two, because a batch of stuck jobs had taken an evening to come back. The ConfigMap was edited, the pull request was approved, and the change was finished as far as any of us were concerned.
A ConfigMap consumed through envFrom becomes environment variables in the container at start. The process reads them once. Editing the ConfigMap edits the ConfigMap. It does not restart anything and it does not make the deployment roll, because the deployment's own specification has not changed: it names a ConfigMap, and the name is identical before and after. Nothing here is broken. The object you edited is simply not the object the process read.
That service deploys rarely, so nothing rolled. Then in August we upgraded the node pool, nodes drained a few at a time, and six pods came back with the two minute lease while eighteen carried on with ten. A send that takes three minutes, which happens whenever the mail provider is slow, sits inside the old lease and outside the new one. A job already in flight on one pod would be handed to another, and the customer got the same message twice, at a rate of roughly one in nine thousand, decided by which pod picked it up.
Every deployment now carries an annotation holding a hash of its configuration, so editing a ConfigMap changes the pod template and the rollout happens on its own. Each service exposes the hash it actually loaded as a metric, and an alert fires when one fleet reports more than one value for longer than a rollout should take.
A change you have applied and a change that is running are two different states, and we had nothing anywhere that showed the distance between them.
– Sergey Shinder
Top comments (0)