Mustafa ERBAY

Posted on Sep 3 Originally published at mustafaerbay.com.tr

Growing a Pod Without Restarting It: In-Place Resize

#kubernetes #operations #autoscaling #cgroup

For years, raising a container's CPU request from 500m to 800m in Kubernetes came down to a single sentence: kill the Pod, create a new one. For a web server behind a Deployment, that was tolerable. On minute twenty-five of a half-hour batch job, or on a game server holding session state in memory, it was not. The price of a resource estimate was paid not by whoever made the estimate, but by the workload that got restarted.

That wall came down in December 2025. In-place Pod resize was declared stable in Kubernetes 1.35 — a feature that was alpha in 1.27, beta in 1.33, and first conceived six years earlier. The 1.36 release in April 2026 extended the same capability to the Pod-level resource envelope, and 1.37, released on 26 August 2026, added an alpha mechanism on the scheduler side for deferred resize requests.

So far, good news. My argument is this: the value of this feature is not "you no longer have to estimate requests and limits correctly." The cost of a wrong estimate dropped sharply on the CPU side and barely moved on the memory side. The limitations section of the official documentation spells this out line by line. It is boring to read and expensive to skip.

What became mutable, and what did not

At the heart of the mechanism is the split between desired and actual. spec.containers[*].resources now represents the desired resources and is mutable for CPU and memory. What the container actually has lives in status.containerStatuses[*].resources. A third field, status.containerStatuses[*].allocatedResources, carries the value the Kubelet has confirmed and, in the documentation's own words, exists primarily for internal scheduling logic; it is not the field to watch when monitoring — status...resources is.

This triple has a pleasant side effect. If a node has Pods with a pending or incomplete resize, the scheduler uses the maximum of the container's desired, allocated, and actual requests when making scheduling decisions. In other words, you cannot quietly overcommit a node by riding an in-flight resize.

The only way to trigger a change is the /resize subresource. Resource fields remain immutable through an ordinary Pod update — KEP-1287 records this as a deliberate departure from the alpha behavior. In practice the command looks like this:

kubectl patch pod resize-demo -n qos-example --subresource resize --patch \
  '{"spec":{"containers":[{"name":"pause","resources":{"requests":{"cpu":"800m"},"limits":{"cpu":"800m"}}}]}}'

The --subresource resize argument requires a client of at least v1.32; older kubectl versions report invalid subresource. The set of fields the subresource will let you modify is narrow too: .spec.containers[*].resources, .spec.initContainers[*].resources for sidecars only, and .spec.resizePolicy.

There is a conclusion here that the documentation does not spell out. Since subresources are written in RBAC delimited by a slash, pods/resize is a surface separate from update on pods. I would think twice before handing that permission around: a button that can shift the resource balance of a live node does not belong in the same box as "may restart a Pod."

There is one more gate on the same path: a resize request also passes through the admission chain. Per the KEP, the ResourceQuota pod evaluator was modified to accept Pod updates and verifies that the sum of resources across all Pods in the namespace does not exceed the quota, while LimitRanger checks that the requested values do not violate the namespace's LimitRange minimums and maximums. So in a full namespace a resize can be rejected on quota even when the node has room. Anyone running tight quotas needs that line in their plan.

You make the restart decision

The per-container policy is written with resizePolicy:

resizePolicy:
- resourceName: cpu
  restartPolicy: NotRequired
- resourceName: memory
  restartPolicy: RestartContainer

NotRequired is the default: the change is applied to the running container. RestartContainer behaves exactly as its name suggests. The example scenario in the docs is the critical part: with the configuration above, changing only CPU resizes the container in place, changing only memory restarts it, and changing both at once restarts it as well, because of the memory policy. If the Pod's overall restartPolicy is Never, you cannot write a policy that requires a restart for any resource; all of them must be NotRequired.

Writing RestartContainer for memory looks at first like admitting defeat. In my view it is exactly the right answer for most JVM and CPython processes, and the reason follows shortly.

The real text is in the limitations list

The "Limitations" section of the documentation teaches more than the sections describing what the feature is. Only CPU and memory can be resized; no other resource type.

Decreasing memory limits was unblocked at GA, but without guarantees. If the policy is NotRequired, the Kubelet makes a best-effort attempt to prevent OOM kills: if current usage exceeds the new limit, the resize is skipped and the status stays stuck in an "In Progress" state. The documentation itself says this is best-effort and remains subject to a race where usage spikes right after the check. Shrinking a memory limit on a live workload is like slowly pulling the rug from under a running process — usually fine, and on the day it is not, the container dies.

The QoS class, by contrast, is set in concrete. A Pod's class is determined at creation and cannot be changed by a resize. In a Guaranteed Pod, requests must keep equalling limits; in a Burstable Pod, requests and limits cannot become equal for both CPU and memory simultaneously (that would make it Guaranteed); and a BestEffort Pod cannot have requests or limits added later. Resource requests and limits cannot be removed entirely either, only changed to different values.

The rest is short but sharp: non-restartable init containers and ephemeral containers cannot be resized, while sidecar containers can. Windows Pods are not supported. Pods managed by static CPU or Memory manager policies cannot be resized in place — so latency-sensitive workloads whose nodes were put on static policies for NUMA alignment are not invited to this party. Pods using swap cannot change memory requests unless the memory policy is RestartContainer. Resizing memory-backed emptyDir volumes in place requires both the InPlacePodVerticalScalingMemoryBackedVolumes gate (alpha and disabled in 1.37) and cgroup v2; on cgroup v1 nodes the request is rejected as infeasible. That work goes through the same /resize surface — you patch spec.volumes[].emptyDir.sizeLimit through the subresource. Disk-backed emptyDir volumes and persistent volumes cannot be resized this way.

Reading that list and asking "which of these applies to me?" is the only meaningful work to do before enabling the feature. I do not run Kubernetes on my own server — I have written separately about what resource pressure on a single machine feels like — but half of that list applies to single-node setups anyway.

The runtime wall: the cgroup grows, the heap does not

The most honest sentence in the GA announcement is tucked into its future-work section: Java and Python runtimes do not support resizing memory without a restart, and the topic is still being discussed with Java developers through an open bug.

What this means: when the Kubelet raises a container's cgroup memory limit, the JVM's -Xmx value does not grow with it. The process cannot use the space it has just been granted; it only gains some breathing room at the operating system level. This is not a Kubernetes shortcoming — it is how those runtimes are designed. But the practical consequence is clear: be skeptical of any story that builds memory automation on top of in-place resize. On the CPU side the gain is real and immediate; on the memory side, what you mostly get is a slightly more controlled restart.

Deferred, Infeasible, and the part that is actual operations

The state of a resize request is published in the Pod's conditions. PodResizePending means the Kubelet cannot grant the request immediately, and it comes with two reasons: Infeasible, meaning the request is impossible on this node (asking for more than the node has, for instance), and Deferred, meaning it is not possible right now but might become possible later — if another Pod is removed, say. In the second case the Kubelet retries periodically. PodResizeInProgress means the request was accepted and the changes are being applied; errors show up in the message field with reason: Error.

The retry order for deferred resizes is defined as well. The documented order is: Pods with a higher PriorityClass first, then Guaranteed Pods before Burstable ones, then whichever has been waiting longest. The KEP puts one more criterion ahead of those — resizes that do not increase requests are always attempted first, because they are not expected to fail. The InPlacePodVerticalScalingSchedulerPreemption gate introduced in 1.37 takes this one step further — the scheduler can evict lower-priority Pods to make room on the node for a higher-priority Pod's resize request. It is alpha and disabled by default; I would not turn it on in production.

For observability there are two handles. Which spec generation has been processed is reported by status.observedGeneration and by the observedGeneration field inside the conditions. On the API side, the apiserver_request_total{resource=pods,subresource=resize} counter defined by the KEP is the natural place to watch the success rate of resize requests.

The Pod-level envelope and the trap inside it

Pod-level resize, beta since 1.36, allows the aggregate budget in spec.resources to be changed on a running Pod. The requirement list is long: the PodLevelResources, InPlacePodVerticalScaling, InPlacePodLevelResourcesVerticalScaling, and NodeDeclaredFeatures gates, cgroup v2, a runtime that supports the UpdateContainerResources CRI call (containerd 2.0 and later, or CRI-O), and Linux nodes only.

The Kubelet's ordering is well thought out: when the budget grows, the Pod cgroup is expanded first and the containers after it; when it shrinks, the containers are throttled first and the Pod cgroup is narrowed afterwards. That is the natural way to prevent overshoot.

The trap is in containers with no limit of their own. Such a container inherits its effective boundary from the Pod budget. When you raise the Pod limit from 200m to 300m, that container's implicit limit changes too, and the Kubelet counts this as a resize event for it. If the container's policy is RestartContainer, the result is a restart. The example in the official documentation shows precisely this: a CPU change made at the Pod level restarts the second container. The sentence "Pod-level resize is restart-free" is wrong right there.

The validation rules are sensible too: Pod-level requests cannot be smaller than the sum of the container requests. On the limit side, each container's limit must be less than or equal to the Pod limit, but the sum of container limits may exceed it — that is how a shared pool is supposed to work.

Who gets to automate this

In-place resize is not automation by itself; it is the substrate automation runs on. The consumer is the Vertical Pod Autoscaler. The current mode list reads like this: Off only produces recommendations, Initial only applies them at creation, Recreate evicts the Pod. InPlaceOrRecreate updates in place when possible and falls back to eviction when not, and according to the GA announcement it has reached beta. InPlace is alpha in VPA 1.7.0: it needs Kubernetes 1.33 or later, InPlacePodVerticalScaling in the cluster, and the InPlace gate on both the VPA updater and the admission controller — and the difference is that this mode never falls back to eviction; if a resize cannot be applied, it defers and retries. The Auto mode has been deprecated since VPA 1.4.0 and behaves the same as Recreate.

In my view the healthy first step is this: enable VPA with InPlaceOrRecreate, and set container policies to NotRequired for CPU and RestartContainer for memory. CPU drift then gets corrected without anyone noticing, while memory changes turn into deliberate restarts. InPlace never evicting sounds gentle, but if the node has no room the recommendation can be deferred indefinitely, and automation that quietly sustains resource starvation can be more dangerous than a noisy eviction.

One more thing: if an HPA is scaling the same workload on CPU usage, utilization is computed against the Pod's requests. When VPA changes a request on a live Pod, the HPA's denominator changes with it. Walk that interaction through on paper once before combining both automations on the same resource.

Questions to answer before you enable it

Is the cluster on 1.35 or later, and are the kubectl clients past v1.32?
Do the nodes use a static CPU/memory manager policy or swap? If so, those workloads are out of scope.
Which containers should have RestartContainer as their memory policy? For JVM and CPython processes my default answer is yes.
Who holds pods/resize? That permission is the permission to change a resource budget on a live cluster.
Does monitoring compare spec resources with status resources? Do you have an alert for Pods stuck in PodResizePending?
Are you planning something that requires changing the QoS class later? That road is closed; you have to recreate the Pod.

A controlled door in the immutability wall

Part of the story Kubernetes has told from day one was Pod immutability: if you want to change something, create a new one and throw the old one away. There is now a narrow, heavily regulated door in that wall. The door solves a real problem — after six years of work on it, no less.

The cost is real as well: state now lives between spec and status, and those two can diverge. It used to be enough to read a manifest to know how much CPU a Pod had; today the correct answer is in the state on the node. As someone who thinks Kubernetes is not needed everywhere, let me add this: the feature makes life easier where the complexity is earned, and adds one more field to watch where it is not. Which side you are on is something you will learn by asking your own cluster, after reading the limitations list.

DEV Community