In August a node in our production cluster came under memory pressure and the kubelet evicted every checkout pod on it, while pods of other services on the same node, using more memory, stayed put. Checkout lost a third of its capacity for six minutes. The events explained it in one word: the evicted pods were BestEffort. They had no resource requests at all.
They were supposed to. Since April version five of our shared service chart had set default requests and limits for every service, sized from a month of real usage. Every service repository bumped the chart version. In staging every pod had requests. In production twenty three services out of thirty one had none.
Our deploy script runs helm upgrade with --reuse-values and sets only the image tag. It was written that way three years ago, so that values set by hand at a service's first install would survive every later deploy without anyone repeating them. What --reuse-values actually does is take the previous release's values, including the defaults of the chart that release was built from, and use them in place of the new chart's defaults. A key that exists only in the new chart's defaults never reaches the templates. Our resources block was wrapped in a with statement, so an absent value rendered nothing: no error, a valid manifest, no requests.
Staging is rebuilt every week with fresh installs, so it read the new defaults. Production releases went back three years, each inheriting the values of the one before. Services installed after April were fine, which is why the eight with requests were precisely the eight newest.
We now deploy with an explicit values file per environment, kept in git and passed on every upgrade, and the script refuses --reuse-values. Getting there meant first collecting what each production release had been carrying invisibly, with helm get values, which turned up forty one settings nobody could explain and two that contradicted what git said. Every pull request now shows a helm diff against production, so a change in defaults appears as a change. And an admission policy rejects pods without memory requests in production namespaces.
A deploy that only changes the image tag is still deciding what to do with everything else. Ours had decided, every time, to keep things exactly as they were in 2023.
– Sergey Shinder
Top comments (0)