On a Thursday in May one of our Kubernetes nodes failed its health checks and was replaced, and the pod for our customs document store moved to another node. It took forty one minutes to start. For all of that time export shipments could not have their commercial invoices attached, and two hundred pallets waited at the dock for their paperwork.
The store is a StatefulSet with a 600 gigabyte volume holding about nine million scanned documents. The new pod sat in ContainerCreating. Its events showed the volume attached and mounted within a minute, then nothing. On the node, the kubelet was busy and the volume was running at the limit of its provisioned IOPS.
A year earlier we had changed the image to run as a non root user, and so that the user could write to the volume we set fsGroup in the pod's security context. With fsGroup set, Kubernetes by default changes the group ownership and permissions of everything on the volume to match, recursively, every time the volume is mounted. The documentation warns that for large volumes this can take a long time. When we added the setting the volume held a few thousand files and the walk was instant. Nine million files later it visited every one of them on every mount, although they all had the right owner already, because the default policy, Always, does not look before it changes.
We had never seen it because the pod almost never moved. When its container restarted the volume stayed mounted, so the walk never ran, and its node had been up for months. The one mount that mattered was the one after a node died.
The pod spec now sets fsGroupChangePolicy to OnRootMismatch, which lets the kubelet skip the walk when the volume's top directory already has the expected owner and permissions. The next move took fifty seconds. Once a month we drain the node under a copy of the store in staging, restored from a production snapshot, and record how long the pod takes to be ready. And the documents are moving to object storage, where a node failure has nothing to walk.
We chose a setting for a few thousand files and left it running against nine million. Its cost grew with the disk, and it was only charged on the day we needed the pod back fastest.
– Sergey Shinder
Top comments (0)