We run OpenSearch for parcel search as a StatefulSet on EKS, on a node group of memory optimised instances. In October, ahead of the peak season, we scaled it from two replicas to three. The third pod sat in Pending for six hours, through the busiest evening of the month, while two nodes carried the load we had sized for three.
The event on the pod said that a node had a volume node affinity conflict. The cluster autoscaler had not added anything. It had considered the pod and concluded that no node it was able to create would help.
The memory optimised node group spans only two of our three availability zones, eu-west-1a and eu-west-1b, because the instance type was not offered in the third zone when we built it. Each replica keeps its data on an EBS volume, and an EBS volume lives in exactly one zone and can only be attached to an instance in that zone. The new replica's volume had been created in eu-west-1c.
That came from our StorageClass. A storage class has a volumeBindingMode, and when it is not set the default is Immediate: the volume is provisioned as soon as the claim exists, before the scheduler has chosen a node for the pod. The provisioner picked a zone with no idea where the pod could run, and from then on the pod was tied to eu-west-1c, where we had no node of the kind it needed. The first two replicas had landed in a and b by luck.
The default class that EKS creates uses WaitForFirstConsumer. Ours was a gp3 class someone wrote two years ago without the field. With WaitForFirstConsumer the claim waits until a pod using it has been scheduled, and the volume is created in that node's zone.
The fields of a storage class cannot be changed once it exists, so we created a new class with the mode set and made it the default. Our claim template names no class, so we deleted the stuck pod and its empty claim and let the StatefulSet create both again. A policy check now rejects any StorageClass without WaitForFirstConsumer, and any pod Pending for more than ten minutes raises an alert.
Immediate binding decides where the data will live before anyone has decided where the work is able to run.
– Sergey Shinder
Top comments (0)