On a Wednesday in March new pods of our basket service began crashing at startup with invalid port: tcp://10.96.41.7:6379. Pods that were already running were fine. A deploy that morning stopped at two pods of twelve, so we rolled it back, and the pods from the rollback crashed in exactly the same way. That was the first useful clue, because it meant the code was not the problem.
The basket service reads REDIS_HOST and REDIS_PORT. Our manifest set only the host, because the code defaults the port to 6379 and that had always been right. That morning the platform team had given the basket team its own Redis, in our namespace, behind a Service called redis.
By default Kubernetes gives every container a set of environment variables for every Service in its namespace, as it stood when the pod started. For a Service called redis these include REDIS_SERVICE_HOST and REDIS_SERVICE_PORT, and also, for compatibility with the links of early Docker, REDIS_PORT, set to tcp:// followed by the address and port. A variable declared in the pod spec wins over these, so anything we set ourselves was safe. The one value we had left to a default in code was now set for us, in a format our parser rejected. Running pods kept the environment they had started with, which is why only new pods failed and why it looked so much like a bad release.
It cost us more than a stuck deploy. The autoscaler's new pods crashed too, so through the lunchtime peak ten pods did the work of fourteen.
We set REDIS_PORT explicitly within twenty minutes, then looked at what else we were being given. Our namespace had sixty three Services, which meant every pod in it carried over four hundred variables nobody had asked for. Our shared chart now sets enableServiceLinks to false, and an admission policy does the same for any pod that slips past the chart. Every setting our services read from the environment now carries the service's own prefix, BASKET_REDIS_PORT for example, so no Service name can ever collide with it. And at startup each service logs where every setting came from: the manifest, a file or a default.
An environment variable has no owner. The process that reads it cannot tell who wrote it, and in a shared namespace the writer can be anyone who creates a Service.
– Sergey Shinder
Top comments (0)