Fix conflicting SELinux labels of volume in K8s 1.37

Kubernetes 1.37 shipped on 26 August 2026, and with it SELinuxMount went stable and on by default. A qualifying PersistentVolume is now handed to the container runtime with -o context=<label> instead of being walked inode by inode. On a volume with four million files, that converts minutes of pod startup into milliseconds, which is the reason KEP-1710 exists in the first place. It also means a mounted volume carries exactly one SELinux label for as long as it stays mounted, and two pods with different labels can no longer share it on the same node.

The second pod sits in ContainerCreating. Kubelet emits an event naming the conflict, phrased in KEP-1710 as "volume XYZ is already mounted with a different SELinux label", and described in the docs as conflicting SELinux labels of a volume, with the other pod's name attached when both live in the same namespace.

Which pods actually take the new mount path?

Four conditions, all of them, per the security-context task docs: the pod uses a PersistentVolumeClaim; spec.securityContext.seLinuxChangePolicy is nil or MountOption; the containers using that PVC set seLinuxOptions; and the PV is an in-tree iscsi, rbd, or fc volume, or a CSI volume whose CSIDriver sets spec.seLinuxMount: true (a field that has been available since 1.25). Fail any gate and you get 1.36 behaviour.

The uncomfortable consequence is who qualifies. A pod that never sets seLinuxOptions is untouched. The pods that pass all four gates are the ones that deliberately pin a label: sidecar-injected workloads, OpenShift-style deployments, anything setting seLinuxOptions.level so two workloads can share a volume on purpose. The teams that did the SELinux work are the ones this lands on, which puts it alongside the mirror pod secret reference break and the DRA GPU double-allocation hazard in the 1.37 category of "correct configuration is the entry condition for the failure".

flowchart TD
  A["Pod with a PVC starts on a 1.37 node"] --> B{"seLinuxChangePolicy = Recursive?"}
  B -->|Yes| R["Runtime relabels every inode (1.36 behaviour)"]
  B -->|No| C{"Containers set seLinuxOptions?"}
  C -->|No| R
  C -->|Yes| D{"CSIDriver seLinuxMount: true?"}
  D -->|No| R
  D -->|Yes| E["kubelet mounts with -o context=label"]
  E --> F{"Volume already mounted on this node under another label?"}
  F -->|Yes| G["Pod stuck in ContainerCreating\nconflicting SELinux labels of volume"]
  F -->|No| H["Pod runs; chcon on that volume returns EOPNOTSUPP"]

One command scopes the cluster:

kubectl get csidriver -o custom-columns=NAME:.metadata.name,SELINUXMOUNT:.spec.seLinuxMount

AWS EBS CSI and Ceph RBD both advertise seLinuxMount: true, so most SELinux-enforcing fleets on RHEL, CoreOS, Bottlerocket, or Flatcar are in scope the moment the nodes reach 1.37. The CSI driver object reference documents the field if you need to confirm what a third-party driver claims.

Four post-upgrade symptoms that get blamed on the same thing

Upgrade threads currently blend these together. They have different causes and different discriminating checks.

Symptom after upgrading to 1.37	Cause	Check that proves it
Pod stuck in ContainerCreating, event names two SELinux labels	Two labels, one -o context= mount, same node	kubectl describe pod; node metric volume_manager_selinux_volume_context_mismatch_errors_total
Nothing changed, startup still slow on large volumes	A gate failed, usually seLinuxMount: false on the driver or containers with no seLinuxOptions	findmnt -o TARGET,OPTIONS on the node; no context= in the options
App runs, then chcon / restorecon / setfattr fails with "Operation not supported"	Kernel forbids per-file relabelling on mountpoint-labelled filesystems	Reproduce with chcon -t svirt_sandbox_file_t <file> inside the container
Second pod on a shared NFS export silently gets the first pod's label	Superblock reuse; KEP-1710 notes NFS needs nosharecache, CIFS nosharesock	Compare context= in findmnt against the pod's own seLinuxOptions.level

Row three deserves a look at the source rather than the release notes. SELinux disables setxattr on security.selinux for filesystems labelled at the mountpoint, in selinux_inode_setxattr() in security/selinux/hooks.c, added by the commit "selinux: disable setxattr on mountpoint labeled filesystems". Red Hat's SELinux documentation states the user-visible result directly: chcon on a filesystem mounted with a context option returns "Operation not supported". Any application that manages its own file labels on a PVC stops working on 1.37, and Kubernetes reports nothing at all, because from kubelet's point of view the pod started fine.

Why does the recommended pre-flight check come up clean on EKS?

The docs and the SIG-Storage announcement of 22 April 2026 both point at the SELinuxWarningController. It watches running pods, emits an event on both sides of a conflict, and raises selinux_warning_controller_selinux_volume_conflict. The docs also record that it ships disabled and is turned on with --controllers=*,selinux-warning-controller on kube-controller-manager, with the SELinuxChangePolicy gate (GA since 1.36).

That is a control-plane flag. EKS, GKE, and AKS do not expose it. So the advice showing up in upgrade checklists, "check for conflict events before you upgrade", returns zero on managed clusters because no controller was ever counting. The reachable signal is per-node: scrape kubelet /metrics for volume_manager_selinux_volume_context_mismatch_warnings_total, which KEP-1710 describes as counting pods that would fail once SELinuxMount is enabled. On a 1.36 fleet that counter is the honest forecast, and even then a zero only tells you the conflicting pair never happened to co-locate during the window you were scraping.

The failure is intermittent, because it depends on scheduling

Both pods have to land on the same node for the conflict to fire. A ReadWriteMany PVC shared between a privileged log-shipper and an unprivileged app can pass a staging upgrade for weeks, then break the first time a descheduler run, a node drain, or a bin-packing change puts them together. Absence of failures on upgrade day is weak evidence. This is the same reasoning that makes cluster-wide guarantees like default-deny egress worth more than per-workload assumptions: anything that only holds under a particular placement will eventually meet a different placement.

The blanket Recursive opt-out gives back the thing you upgraded for

The circulating fix is seLinuxChangePolicy: Recursive, pushed cluster-wide through MutatingAdmissionPolicy, Kyverno, or Gatekeeper. The docs suggest roughly that, and I would not apply it fleet-wide. Setting it everywhere returns every qualifying volume to full-tree relabelling, which is the startup cost KEP-1710 was written to remove, and on a large volume that is the difference between a pod starting in milliseconds and a pod starting in minutes. Scope it to the namespaces that genuinely share a volume across two labels, keep the mount path for everything else, and accept that you now have a per-namespace policy to maintain.

There is a second-order problem with the opt-out that I have seen bite in a different guise: the field can be silently dropped. An API server older than 1.36 prunes seLinuxChangePolicy from the pod spec, so the manifest in Git says Recursive and the running pod says nothing. Schema mismatches between what you submit and what the API server retains are a recurring class of Kubernetes bug, the same shape as "unrecognized format int32". Confirm against the live object, not the file.

What to check before you promote 1.37 to production

1. List the exposed drivers. Run the kubectl get csidriver command above. Any driver with SELINUXMOUNT: true puts every PVC it serves in scope.
2. Find the conflicting pairs. For each of those PVCs, list the pods that mount it and compare seLinuxOptions.level across them. Differing levels on one PVC is the blast radius, and it is usually a short list.
3. Scrape the forecast metric on every node while still on 1.36: volume_manager_selinux_volume_context_mismatch_warnings_total from kubelet /metrics. Treat zero as unproven until you have confirmed from step 2 that a conflicting pair exists and has been co-located.
4. Enable the SELinuxWarningController if you run your own control plane: --controllers=*,selinux-warning-controller, with the SELinuxChangePolicy gate. On EKS, GKE, and AKS, write in the upgrade plan that this check is unavailable and that the node metric is the substitute, so nobody later reads a clean event log as a pass.
5. Set seLinuxChangePolicy: Recursive on the pairs from step 2 only, then verify it persisted: kubectl get pod -o jsonpath='{.spec.securityContext.seLinuxChangePolicy}'.
6. Grep your images and charts for chcon, restorecon, and setfattr run against a PVC path. Those calls will start returning EOPNOTSUPP on qualifying volumes, with no Kubernetes event to explain it. Reproduce with chcon -t svirt_sandbox_file_t <file> inside the container before the upgrade, not after the pager goes off.

Steps 1 and 2 take an afternoon on a mature cluster. Step 6 is the one teams skip, and it is the one that produces a silent application failure weeks later with no trace in kubectl describe.

FAQ

Does SELinuxMount affect every pod with a PVC in Kubernetes 1.37?
No. All four gates must pass: PVC-backed volume, seLinuxChangePolicy nil or MountOption, seLinuxOptions set on the containers using that PVC, and either an in-tree iscsi/rbd/fc volume or a CSI volume whose CSIDriver sets spec.seLinuxMount: true. Otherwise the runtime relabels recursively as before.

Why does chcon fail with "Operation not supported" on a PVC after upgrading?
SELinux refuses setxattr on security.selinux for filesystems labelled at the mountpoint, in selinux_inode_setxattr() in security/selinux/hooks.c. Once kubelet mounts with -o context=, per-file relabelling inside that volume is denied by the kernel.

How do I opt a single workload out?
Set spec.securityContext.seLinuxChangePolicy: Recursive on the pod, then confirm with kubectl get pod -o jsonpath='{.spec.securityContext.seLinuxChangePolicy}'. An API server older than 1.36 can prune the value without error.

Can I use the SELinuxWarningController on a managed cluster?
No. It needs a kube-controller-manager flag that EKS, GKE, and AKS do not expose. Use volume_manager_selinux_volume_context_mismatch_warnings_total from kubelet on each node.

Do NFS and CIFS volumes get per-pod contexts?
Not by default. KEP-1710 notes NFS needs nosharecache and CIFS needs nosharesock, otherwise superblock reuse hands the second pod the label from the first mount.

---

Originally published at indragustiprasetya.com