DEV Community

yutianle
yutianle

Posted on

Kubernetes priority classes can preempt security controls

Kubernetes priority classes can preempt security controls

Kubernetes priority controls scheduling order, not trust. A PriorityClass maps a name to an integer, and a pod with a higher value may cause lower-priority pods to be evicted when the cluster needs capacity. The feature is useful for latency-sensitive services and critical jobs. It can also remove the pod that performs logging, policy enforcement, or incident response.

The Kubernetes documentation warns that users who can create high-priority pods can disrupt other workloads in a cluster where not all users are trusted. That warning applies beyond ordinary applications. An admission controller, security agent, or monitoring collector with a modest priority can be displaced by a workload with a higher one. A PodDisruptionBudget does not make the control absolute. Kubernetes supports PDBs during preemption, but the scheduler treats them as best effort and may still preempt pods when no acceptable victim set is found.

Priority is a cluster-scoped decision

PriorityClass is a cluster-scoped API object. Higher numbers mean higher priority. A default class can apply when a workload omits priorityClassName, while a pod that explicitly names a class receives that value. The scheduler uses priority when choosing victims, and kubelet node-pressure eviction also considers pod priority alongside resource usage and QoS.

Those interactions make a simple policy review insufficient. A namespace may look constrained while a workload can select a dangerous cluster-wide class. A service account may create jobs that inherit a high-priority template. A namespace quota can limit the number of pods, but it does not automatically prevent one pod from using a preempting class.

The safer pattern is to limit which identities may create or use privileged classes, make production classes explicit, and keep security infrastructure on nodes or workloads whose failure is separately budgeted. Check whether the control plane can continue collecting evidence when a high-priority workload triggers repeated evictions.

Test the actual failure path

Create a non-production cluster or namespace with the same admission policies and node pool configuration. Record the priority values of security agents, DNS, ingress, logging, and application tiers. Submit a pod with preemptionPolicy: Never first and confirm that it waits without evicting anything. Then, only in the test environment, exercise a preempting class and observe events, PDB behavior, restart policy, and loss of telemetry.

Review ResourceQuota, RBAC, admission policy, and the default PriorityClass together. A quota that blocks pods is not enough if it permits a single pod with a cluster-wide high priority. Use kubectl get priorityclass and inspect workload templates for explicit values. Add an alert for preemption events and for a drop in the expected replica count of security services.

Priority classes should express availability requirements with a clear owner. A high number is not a substitute for High Availability, scheduling constraints, or protected network paths. The team that can raise a workload's priority can also decide which lower-priority controls disappear first.

References

Top comments (0)