Priorities, Preemption and QoS: Making Sure Your Critical Pods Survive Saturation

When a node runs out of memory at 3 a.m., three different subsystems can decide which container dies: the scheduler (preemption at admission time), the kubelet eviction manager (node-pressure eviction at runtime), and the Linux OOM killer (cgroup-level, no Kubernetes semantics at all). They do not share the same ranking rules, they do not read the same fields, and only one of them cares about your PodDisruptionBudget.

This article unpacks that interaction: how PriorityClass, QoS class and evictionHard thresholds combine, why your logging DaemonSet gets killed before a batch job, and how to design a priority taxonomy that does not produce cascading preemption.

Context: three killers, one node

A saturated node is not a single failure mode. You need to distinguish:

Mechanism	Trigger	Ranking input	Honors PDB?
Scheduler preemption	A pending pod cannot be scheduled	PriorityClass.value, preemptionPolicy	Best-effort only
Kubelet node-pressure eviction	memory.available, nodefs.available, pid.available… crossing a threshold	usage vs. requests, then priority, then overage	No
Kernel OOM killer	cgroup limit or global memory exhaustion	oom_score_adj derived from QoS	No

The classic incident: an operator sets priorityClassName: high-priority on a batch workload "to make sure it runs", the scheduler preempts a handful of pods on node A, those pods reschedule onto node B, push node B above memory.available<100Mi, the kubelet evicts there too — and because the evicted pods have no memory requests at all (BestEffort), the cascade continues across the cluster. Priority was set, QoS was not, and nobody looked at evictionHard.

QoS classes: the only thing the kubelet really ranks on

QoS is derived, never declared. The rules are strict:

• Guaranteed: every container in the pod sets requests == limits for both CPU and memory.
• Burstable: at least one container sets a CPU or memory request (or limit), but the Guaranteed condition is not met.
• BestEffort: no requests and no limits anywhere in the pod.

Check it, do not assume it:

oc get pods -n payments -o custom-columns=\
'NAME:.metadata.name,QOS:.status.qosClass,PRIO:.spec.priority,PC:.spec.priorityClassName'

QoS drives two things. First, oom_score_adj on every container process:

• Guaranteed → -997
• BestEffort → 1000
• Burstable → min(max(2, 1000 - (1000 × memoryRequest) / nodeAllocatableMemory), 999)

So under a global kernel OOM event, a Guaranteed pod is almost the last victim, and a Burstable pod with a large request is far safer than one with a token 64Mi. Priority plays no role here — the kernel has never heard of PriorityClass.

# Verify on a node (OpenShift)
oc debug node/worker-02 -- chroot /host /bin/bash -c \
  'for p in $(pgrep -f "nginx"); do echo -n "$p "; cat /proc/$p/oom_score_adj; done'

Second, QoS shapes the kubelet's eviction ordering — indirectly.

Kubelet node-pressure eviction: the real ranking

The eviction manager does not sort by "BestEffort first, then Burstable, then Guaranteed", despite what many blog posts claim. For memory pressure it applies a three-level comparator:

1. Is the pod's memory usage above its memory request? Pods above request sort first.
2. Pod priority (lower priority sorts first).
3. Memory usage minus request (largest overage first).

The practical consequence is elegant: a Guaranteed pod can never exceed its request (request == limit), and a Burstable pod staying inside its request is in the same bucket. They are only considered after every pod that is borrowing memory it never asked for. BestEffort pods have a request of zero, so any usage puts them in bucket 1 — they always die first.

Priority only breaks ties inside a bucket. A system-cluster-critical pod that consumes 4 GiB with a 512 MiB request will be evicted before a priority-0 pod that respects its request. Setting a high PriorityClass on a pod with no memory request buys you almost nothing.

Disk pressure (nodefs.available, imagefs.available, inodes) follows the same pattern using ephemeral-storage requests and local volume usage. The kubelet first garbage-collects dead containers and unused images; only if that is insufficient does it start evicting.

Configuring thresholds on OpenShift

Thresholds are node-level kubelet configuration, delivered through a KubeletConfig bound to a MachineConfigPool:

apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: worker-eviction-tuning
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  kubeletConfig:
    evictionHard:
      memory.available: "500Mi"
      nodefs.available: "10%"
      nodefs.inodesFree: "5%"
      imagefs.available: "15%"
    evictionSoft:
      memory.available: "1Gi"
      nodefs.available: "15%"
    evictionSoftGracePeriod:
      memory.available: "1m30s"
      nodefs.available: "2m"
    evictionMaxPodGracePeriod: 60
    evictionPressureTransitionPeriod: 5m
    evictionMinimumReclaim:
      memory.available: "300Mi"
      nodefs.available: "2Gi"
    systemReserved:
      cpu: "500m"
      memory: "1Gi"
      ephemeral-storage: "2Gi"

oc apply -f kubeletconfig-eviction.yaml
oc get mcp worker -w   # rollout is node-by-node, expect reboots-free kubelet restarts

Three details that bite in production:

• memory.available is computed from cgroup working set (memory.usage_in_bytes minus inactive file cache), not from free -m. A node showing "plenty of free memory" in top can still be under pressure.
• The hard eviction threshold is subtracted from Allocatable. Raising memory.available to 500Mi removes 400 MiB of schedulable capacity per node compared to the default 100Mi. That is the point: you buy headroom so the kubelet reacts before the kernel OOM killer does.
• evictionSoft gives containers a real SIGTERM grace period (capped by evictionMaxPodGracePeriod); evictionHard kills immediately. Always define both, with the soft threshold more conservative.

# What did the kubelet actually load?
oc debug node/worker-02 -- chroot /host \
  cat /etc/kubernetes/kubelet.conf | grep -A12 eviction

# Who got evicted and why
oc get events -A --field-selector reason=Evicted \
  -o custom-columns='TIME:.lastTimestamp,NS:.involvedObject.namespace,POD:.involvedObject.name,MSG:.message'

PriorityClass and preemption: the scheduler's side

PriorityClass is a cluster-scoped object. Its value lands in pod.spec.priority at admission time (mutation is immutable afterwards).

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: platform-critical
value: 1000000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Node agents, ingress, logging, service mesh data plane."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-tier1
value: 100000
globalDefault: false
preemptionPolicy: PreemptLowerPriority
description: "Customer-facing APIs with an SLO."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: batch-non-preempting
value: 10000
globalDefault: false
preemptionPolicy: Never
description: "Scheduled ahead of default workloads in the queue, never evicts anyone."

preemptionPolicy: Never is the single most underrated field here. The pod keeps its place in the scheduling queue (it is considered before lower-priority pending pods) but it will never trigger victim selection. This is exactly what you want for batch, CI runners, Spark executors and anything that can simply wait.

Reserved bands: system-cluster-critical = 2000000000 and system-node-critical = 2000001000 ship with every cluster. Do not create user classes above 1000000000 — leave the top band to the control plane. On OpenShift you will also find openshift-user-critical and openshift-monitoring as managed classes.

oc get priorityclass -o custom-columns='NAME:.metadata.name,VALUE:.value,DEFAULT:.globalDefault,POLICY:.preemptionPolicy'

Avoiding cascading preemption

Preemption cascades happen when high-priority pods are produced faster than capacity, or when victim selection repeatedly fails to actually unblock the pending pod (anti-affinity, topology spread, volume node affinity). Guardrails that work:

1. Few, wide priority bands. Four or five classes with gaps of 10× between values. Dozens of classes with values 100, 101, 102 produce long preemption chains with no operational meaning.

2. Quota the expensive bands. A ResourceQuota with a scopeSelector on PriorityClass caps how much a namespace can consume at a given priority:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tier1-budget
  namespace: payments
spec:
  hard:
    cpu: "64"
    memory: 128Gi
    pods: "120"
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["business-tier1", "platform-critical"]

3. Make DaemonSets explicitly critical. The kubelet grants no special treatment to DaemonSet pods during eviction; only priority and requests count. Node agents must carry system-node-critical (or your own high band) and real requests.

4. Watch the right metrics.

# Scheduler-side
scheduler_preemption_victims
scheduler_preemption_attempts_total

# Kubelet-side
kubelet_evictions{eviction_signal="memory.available"}
kubelet_eviction_stats_age_seconds

A PromQL alert worth having:

sum(rate(kubelet_evictions[15m])) by (node, eviction_signal) > 0

PodDisruptionBudget: what it does and does not protect

This is the most common misconception in the whole topic.

• API-initiated eviction (oc adm drain, descheduler, cluster-autoscaler scale-down) goes through the pods/eviction subresource and respects PDBs. A violated PDB returns 429 TooManyRequests.
• Scheduler preemption tries to pick victims that do not violate PDBs, but if no such set exists it preempts anyway. It is a preference, not a guarantee.
• Kubelet node-pressure eviction bypasses PDBs entirely. The kubelet deletes the pod directly. No API eviction call, no budget check, no minAvailable.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
  namespace: payments
spec:
  minAvailable: 80%
  unhealthyPodEvictionPolicy: AlwaysAllow
  selector:
    matchLabels:
      app: checkout-api

oc get pdb -n payments
oc adm drain worker-02 --ignore-daemonsets --delete-emptydir-data --timeout=300s

unhealthyPodEvictionPolicy: AlwaysAllow (policy/v1, GA since 1.27) prevents the classic deadlock where a PDB blocks a drain because the already-broken pods are counted as "must not be disrupted".

The takeaway: PDBs protect against planned disruption only. Protection against node saturation comes from requests, limits and eviction thresholds — never from a budget.

A reference manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: payments
spec:
  replicas: 6
  selector:
    matchLabels: { app: checkout-api }
  template:
    metadata:
      labels: { app: checkout-api }
    spec:
      priorityClassName: business-tier1
      terminationGracePeriodSeconds: 45
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: checkout-api }
      containers:
        - name: api
          image: registry.example.com/payments/checkout:1.14.2
          resources:
            requests:
              cpu: "500m"
              memory: "1Gi"
              ephemeral-storage: "256Mi"
            limits:
              cpu: "500m"
              memory: "1Gi"
              ephemeral-storage: "1Gi"
          securityContext:
            allowPrivilegeEscalation: false
            runAsNonRoot: true
            capabilities: { drop: ["ALL"] }
            seccompProfile: { type: RuntimeDefault }

requests == limits on CPU and memory → Guaranteed → oom_score_adj: -997, never in the "above request" eviction bucket, and an explicit ephemeral-storage request so disk-pressure ranking treats it fairly.

Security best practices

Priority is a capacity-theft primitive. A tenant able to set priorityClassName: system-cluster-critical can evict your control-plane add-ons from every node.

• Lock down PriorityClass creation. scheduling.k8s.io/priorityclasses must be restricted to cluster admins. Audit with oc adm policy who-can create priorityclasses.
• Restrict consumption, not just creation. RBAC cannot filter which priorityClassName a pod may reference. Use a ValidatingAdmissionPolicy:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: restrict-priority-classes
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >
        !has(object.spec.priorityClassName) ||
        object.spec.priorityClassName in ['batch-non-preempting','business-tier1']
      message: "This namespace may only use batch-non-preempting or business-tier1."

Bind it with a ValidatingAdmissionPolicyBinding scoped by namespace selector so platform namespaces keep access to the high bands.

• Never set globalDefault: true on a high-value class. It silently promotes every pod in the cluster, including untrusted ones, and makes preemption meaningless.
• Enforce requests with LimitRange. BestEffort pods are a denial-of-service vector: they consume memory they never declared and drag the whole node to the eviction threshold. A namespace LimitRange with defaultRequest removes the BestEffort class entirely.
• Reserve capacity for the system. systemReserved plus a non-default evictionHard keeps the kubelet, CRI-O and sshd alive so you can still reach the node during an incident.
• Audit priority changes. Add scheduling.k8s.io resources to the audit policy at RequestResponse level; a new PriorityClass with value 2000000001 is a credible escalation signal.

Conclusion

• QoS is derived from requests/limits — it is the primary defense. requests == limits yields Guaranteed, oom_score_adj: -997, and immunity from the "above request" eviction bucket. No PriorityClass compensates for missing requests.
• The kubelet ranks by overage first, priority second. A high-priority pod exceeding its request dies before a low-priority pod respecting its own.
• The scheduler ranks by priority only. preemptionPolicy: Never lets batch workloads queue ahead without triggering evictions — use it aggressively.
• PDBs do not apply to node-pressure eviction. They cover drains and API-initiated eviction. Stop relying on them for saturation scenarios.
• Tune evictionHard / evictionSoft deliberately. Defaults (memory.available<100Mi) leave almost no reaction time before the kernel OOM killer takes over with no Kubernetes semantics.
• Treat priority as a privilege. Restrict it through admission policy and ResourceQuota scope selectors, and never make a high class the global default.

Design the taxonomy once — four bands, documented, quota-backed — and enforce requests everywhere. Saturation then becomes a predictable degradation instead of a cluster-wide cascade.