In-place pod resize went GA in Kubernetes 1.35, and every post I found about it covers the same ground: patch the resize subresource, watch the pod not restart, feel good. I believed all of that. What I wanted to know was what the process inside the container sees when the cgroup changes underneath it, because a JVM sizes its heap once, at startup, and I couldn't find anyone who had checked.
So I built a kind cluster on 1.37, put a Node 22 app and a Java 21 app in it, and resized them while a load generator was hitting them. The CPU half of the story is as good as advertised. The memory half is where it gets interesting.
The CPU resize is real
The Node app has one endpoint that burns 30 ms of CPU and returns. Eight clients hit it in a loop, and the pod has a 200m CPU limit, which means it is throttled almost all the time:
08:43:39 rps= 11.6 p50= 700ms p99= 1400ms fail=0
08:43:44 rps= 10.0 p50= 700ms p99= 1500ms fail=0
Then, with the load still running:
kubectl patch pod node-app --subresource resize --patch \
'{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"2"},"limits":{"cpu":"2"}}}]}}'
I had a loop polling cpu.max inside the container every half second. It changed 0.27 seconds after the patch returned. Restart count stayed at zero. The load generator, five seconds later:
08:44:29 rps= 33.2 p50= 210ms p99= 450ms fail=0
08:44:34 rps= 33.4 p50= 210ms p99= 426ms fail=0
Throughput tripled and p99 dropped from 1.5 s to 420 ms, with no restart and no new pod. That is the feature working exactly as described, and it is genuinely nice.
The p50 of 210 ms is not a coincidence, though. Eight requests queued on one event loop at 30 ms each is 240 ms. Node got two cores and used one, because that is what Node does. The resize removed the throttling, and then the app hit its own ceiling, which is a fair reminder that "give it more CPU" only helps a single-threaded process up to exactly one core.
What the runtime thinks it has
Before touching memory I wanted to know what each runtime believed about its limits, so both apps report what they saw at startup and what they see now.
CPU, as reported by the process, while I stepped the limit up:
limit Node os.availableParallelism() JVM availableProcessors()
200m 16 -
500m 16 1
1 1 -
1500m 1 -
2 2 -
3 - 3
The JVM reads the cgroup quota live and rounds up, so it sees the change without a restart. Node reads it live too, but with a rule I had not seen written down anywhere: if the quota is less than one full core, libuv gives you the host's core count. Under a 200m limit my app thought it had 16 CPUs. Under 1500m it thought it had one. If you size a worker pool from that number, a small pod is the one that gets it most wrong.
Memory is a different story, and it is the one that matters.
Memory goes up, the heap does not
The Java pod started with a 512Mi limit. The JVM did what it always does and took a quarter of that for the heap:
max_mb_at_start: 123 cgroup_memory_max: 536870912
Resize the pod to 2Gi. The cgroup follows within a couple of seconds:
max_mb_now: 123 cgroup_memory_max: 2147483648
The container has two gigabytes. The heap is still 123 MB, because MaxHeapSize was calculated once, when the process started, and nothing about it is going to be recalculated. Then I asked the app to allocate 200 MB:
{"error":"OutOfMemoryError","retained_mb":119,"max_mb":123}
An OutOfMemoryError with 1.9 GB of unused limit. No restart, no OOM kill from the kernel, because the kernel is perfectly happy. The process killed itself against a number it wrote down a few minutes earlier.
Node does the same thing, slightly differently. Under a 256Mi limit its heap limit came out at 259 MB. I resized to 1Gi, confirmed the cgroup said 1073741824, then allocated 400 MB of JavaScript objects:
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Exit code 139, one restart, and the load generator saw 1853 failed requests in the five-second window while it came back. After the restart the heap limit was 524 MB, so the new size did take effect, eventually, by way of the crash I had resized to avoid.
The part that would actually cost you time in an incident is what this looks like from outside. The pod says 1Gi. The container's cgroup says 1Gi. Memory usage is sitting at a quarter of that. And the app is dying of memory. Every signal you would normally reach for says "not a memory problem".
The fix is one field, and it restarts the container
Kubernetes knows about this. That is what resizePolicy is for:
resizePolicy:
- resourceName: cpu
restartPolicy: NotRequired
- resourceName: memory
restartPolicy: RestartContainer
CPU changes apply live. Memory changes restart the container, and only the container. I ran the same 512Mi to 2Gi resize against a pod with that policy:
before: ip=10.244.0.8 max_mb: 123 restarts=0
after: ip=10.244.0.8 max_mb: 494 restarts=1
One second from patch to the container reporting ready. Same pod, same IP, no scheduling, no new pod name, and the JVM came up with a heap sized for what it actually has. The event log is honest about it:
Killing Container app resize requires restart
Which is fine, as long as you knew it was going to happen. For a JVM or a Node process I now think RestartContainer on memory is the correct default, and NotRequired is the thing you opt into once you have checked your runtime can grow.
Memory goes down
Shrinking is the case I expected to be ugly, and it was better behaved than growing.
With the Java app holding about 160 MB, I resized the limit from 2Gi down to 200Mi. It went through in a second, no restart, no kill. Then I tried 100Mi, which is below what the process is using:
PodResizeInProgress=True reason=Error
cannot decrease memory limits: attempting to set container "app" memory limit
(104857600) below current usage (168046592)
The kubelet refused, left the cgroup at 200Mi, and kept the pod running. It also kept retrying, so the condition sits there with reason=Error until you change your mind. Patching the memory back to 200Mi cleared it immediately. Nothing died. I would take that over an OOM kill every time, but it does mean a resize can silently not happen, and the only place that shows up is a pod condition most dashboards don't surface.
Two smaller things
Ask for more than the node has and the API server says no on the spot. I requested 64 CPUs on a 16-core node:
Error from server (Forbidden): pods "node-app" is forbidden:
node didn't have enough allocatable resources: cpu, requested: 64000, allocatable: 16000
No pending state, no waiting for a scheduler. The patch is rejected and the pod carries on with what it had.
Deployments don't fight you, but they don't remember either. I resized a Deployment's pod directly. Twenty seconds later it still had the new limit, the Deployment reported a successful rollout, and nothing tried to put it back. Then I changed the CPU limit in the Deployment template to the same value, and the Deployment did what it has always done: new ReplicaSet, new pod, old one gone. Resizing a pod that belongs to a Deployment is a hotfix. The next rollout replaces the pod and the resize with it.
What I got wrong on the way
My first RestartContainer test reported the container as ready one second after the patch, so I immediately queried it and got connection refused. The pod had no readiness probe, and without one the kubelet marks the container ready the moment it starts. The JVM needed another few seconds. "Restarted and ready in one second" was true and useless at the same time, which is the kind of number I would have put in a table if I hadn't tried to use it.
Run it yourself
Everything above is one kind cluster on a laptop. A pod with a limit, the resize patch, and the process reporting its own view:
kind create cluster
kubectl run app --image=node:22-alpine --restart=Never \
--overrides='{"spec":{"containers":[{"name":"app","image":"node:22-alpine","command":["node","-e","setInterval(()=>console.log(require(\"os\").availableParallelism(),require(\"v8\").getHeapStatistics().heap_size_limit>>20),5000)"],"resources":{"requests":{"cpu":"200m","memory":"256Mi"},"limits":{"cpu":"200m","memory":"256Mi"}}}]}}'
kubectl logs -f app &
kubectl patch pod app --subresource resize --patch \
'{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"2","memory":"1Gi"},"limits":{"cpu":"2","memory":"1Gi"}}}]}}'
Mine printed 16 259 before the patch and 2 259 after it, every five seconds, for as long as I cared to watch. Kubernetes 1.37.0 on kind 0.33, Node 22.23 with libuv 1.51, Temurin 21.0.12.
What I would take from this
The feature is smaller than the posts about it make it sound, and also better. CPU: patch it, it works, the process sees it within a second, and I'd do it to a live pod without thinking very hard. Memory: the cgroup moves and nothing inside does. Anything that read its limit at startup, which is the JVM, V8, and most things with a garbage collector, carries on with the old number until it dies of it. When it does, the pod will be sitting there with three quarters of its memory free, looking innocent.
I went in expecting to write "set RestartContainer on memory for JVMs". I came out thinking it should be the default for anything, and NotRequired is what you switch on after you have watched your own runtime grow. The restart cost me one second and a pod IP I got to keep. The alternative cost me 1853 requests.
Top comments (0)