The first time in-place Pod resize saved me a restart on a real service, I sat and stared at the terminal for a beat too long. Our stateful container was quietly walking its CPU request up by a full core, no eviction, no rollout, no 3am page. A week later a different Pod on a heavily bin-packed node tried the same trick and the resize just parked itself. Deferred. Waiting. That second experience is the whole reason I care about what landed in Kubernetes v1.37.
The gap in-place resize left behind
In-place Pod resize graduated to GA back in v1.35. It lets you edit the resource requests on a live container and have the kubelet apply the change without a restart, which is a genuinely different feeling on-call than every previous vertical-scaling story Kubernetes has shipped. The catch: if the node does not have enough allocatable headroom, the kubelet writes a Deferred value into the container's resizeStatus and the request sits there until capacity naturally frees up.
Deferred is not the same as Infeasible. Infeasible means the request outright exceeds the physical machine, a LimitRange, or a quota, and the kubelet rejects it immediately. Deferred means the request is valid, the kubelet just does not have the room right now. Before v1.37, "right now" could stretch to "indefinitely", because the kube-scheduler was not watching running Pods for pending resizes and would not preempt anything to make space for one.
Kubernetes v1.37 introduces, in alpha, scheduler preemption for in-place Pod resize, guarded by a new feature gate named InPlacePodVerticalScalingSchedulerPreemption. Turn it on and the kube-scheduler will actively evict lower-priority Pods on the same node so a higher-priority Pod's Deferred resize can finally succeed.
How the scheduler picks up a running Pod
Normally, once a Pod has spec.nodeName set, the scheduler considers it placed and stops looking at it. Under the new gate, the scheduler keeps watching Pods carrying the Deferred condition and pulls them back into active scheduling evaluations, specifically so they can trigger preemption. It holds onto them until the kubelet reports the resize actuated.
A few design choices are worth knowing before you flip the gate on:
- Preemption is single-node. Placement preemption evaluates every node in the cluster to find the best fit; resize preemption is scoped strictly to the Pod's current node. If evicting every eligible lower-priority workload on that node still does not free enough capacity, the resize stays Deferred.
- The scheduler owns all resize preemption. The kubelet has an existing critical-Pod admission handler that can evict locally at admission time, but under this gate that handler stays out of resize decisions. Every eviction runs through the scheduler, so global priority ordering, PodDisruptionBudgets, and graceful termination are respected the same way you already trust them for placement preemption.
- Reserved capacity, not a race. The scheduler treats the requested delta as consumed the moment preemption starts, so a second scheduling loop cannot double-book the same CPU. If a higher-priority resize on the same node lands mid-preemption, the kubelet gives it precedence and the scheduler kicks off a new round.
Turning it off where you do not want it
If you already run a controller that would rather size other Pods down or grow the node itself before anyone gets evicted, this behaviour is going to feel intrusive. v1.37 adds a spec.podPreemptionPolicy field on the Node, with a disableResizePreemption list of controller names that opts that node out:
apiVersion: v1
kind: Node
metadata:
name: <node-name>
spec:
podPreemptionPolicy:
disableResizePreemption:
- "cluster-autoscaler.kubernetes.io/disable-preemption"
- "operator.example.com/policy-override"
Batch-only nodes, or nodes a custom autoscaler wants to own, can flag themselves and keep the scheduler off the eviction path there. Preemption becomes the last-resort tool the doc calls it, not the default.
Trying it locally
The feature gate must be enabled on the kube-apiserver, the kube-scheduler, and the kubelet, and every node has to be on v1.37 or later. The upstream announcement walks through a single-node kind setup that is the fastest way to internalise what the new controller loop actually does: constrain the node's CPU, deploy a low-priority Pod and a high-priority Pod that together nearly fill it, then patch the high-priority Pod's CPU request past the remaining headroom and watch preemption clear the room. Run that once before you argue for it on a real cluster.
The rough edges
It is alpha, so the API shape is not promised across releases. The single-node scope carries a real limit: a resize that needs more room than the entire node can offer, even after evicting every eligible lower-priority Pod, will still park in Deferred. Cases like a memory-hungry VPA target on a small node are still a job for the cluster autoscaler, not for this. And every platform team I have talked to about this has the same fair worry: preemption acting on running Pods, even lower-priority ones, is a behaviour change your batch-job owners deserve a heads-up about before you enable the gate under them.
What I am watching next
Two things. First, whether this graduates at the same cadence core in-place resize did, alpha to GA over a few releases. Second, whether Vertical Pod Autoscaler starts to treat a Deferred status as a first-class recoverable state now that there is finally a story for unsticking one. If both happen, the "no restart, no re-schedule" promise of in-place resize gets a lot closer to the default experience for a lot more workloads, and the next time one of your services quietly walks its CPU up on a full node, you will not have to notice.
Top comments (0)