DEV Community: BaoKhang

Why Kubernetes In-Place Memory Resize Gets Stuck "In Progress" on JVM Pods

BaoKhang — Sun, 08 Nov 2026 19:02:37 +0000

You lowered a JVM pod's memory with an in-place resize, and the pod now shows PodResizeInProgress, an Error reason, and an old memory limit that never changes. The kubelet is not broken. It refuses to lower a memory limit below the container's current usage, and a JVM keeps its committed heap even after the objects in it are garbage. The resize completes as soon as the JVM gives the memory back, which I confirmed on a Kubernetes 1.35.8 cluster. Below are the real outputs, the fix, and how long it took.

The short answer

Cause: the kubelet will not apply a memory-limit decrease while the container's current usage is above the new limit. A JVM's usage stays high after garbage collection because it keeps the heap committed.
Symptom: condition PodResizeInProgress=True with reason Error. containerStatuses[].allocatedResources shows the new value, but containerStatuses[].resources still shows the old one.
It retries. The kubelet retried about every 11 to 15 seconds. The pod was not OOM-killed and did not restart. The check protected the JVM.
Fix: make the JVM release memory, then let the resize finish. Releasing after the patch finished the resize on the next retry (about 20 s in my run). Releasing before the patch finished it 2 seconds after the patch, and the error never appeared.
PodResizePending is a different problem. I saw it only when I asked for more memory than the node has (Infeasible).

What it looks like

I ran a JVM pod on a single-node kind cluster, Kubernetes v1.35.8. The pod has Guaranteed QoS with a 900Mi memory request and limit. The JVM (Temurin 21, ZGC, -Xmx600m) allocates 300 MB, holds it for 10 seconds, then drops all of it and idles. Inside the container, memory.current read 417 MB.

I shrank it to 300Mi through the resize subresource:

kubectl patch pod jvm --subresource resize -p \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"300Mi"},"limits":{"memory":"300Mi"}}}]}}'

A few minutes later, kubectl get pod jvm -o yaml (trimmed) showed:

status:
  conditions:
  - message: 'cannot decrease memory limits: [attempting to set pod memory limit (314572800)
      below current usage (435347456), attempting to set container "app" memory limit
      (314572800) below current usage (435347456)]'
    observedGeneration: 2
    reason: Error
    status: "True"
    type: PodResizeInProgress
  containerStatuses:
  - name: app
    allocatedResources:
      cpu: 500m
      memory: 300Mi        # what the kubelet accepted
    resources:
      limits:
        cpu: 500m
        memory: 900Mi      # what is actually applied
    restartCount: 0

At +10 s the condition was PodResizeInProgress=True with no reason. By +31 s the reason was Error. The cgroup never changed: memory.max stayed at 900 MB, and memory.current stayed at 417 MB for the whole minute I watched.

The events show a repeating error, and the kubelet log says why:

Normal   ResizeStarted   Pod resize started: {... "memory":"300Mi" ... "generation":2}
Warning  ResizeError     Pod resize error: {... "error":"cannot decrease memory limits: [attempting to set pod memory limit (314572800) below current usage (435347456) ...]"}
Warning  ResizeError     (repeats)

kubelet: "Allocated pod resize is not currently feasible" err="cannot decrease memory limits: ..."
kubelet: ... ResizePodInPlaceError ... (retries at 18:50:18, :33, :47, :58, 18:51:13, :25)

That matches what the Kubernetes docs say: before decreasing a container's memory limit, if memory usage exceeds the requested limit, the resize is skipped and the status remains in an "In Progress" state. They call this best-effort because usage can spike right after the check. What the docs don't show is the Error reason and the repeating ResizeError events, which look alarming but are the kubelet waiting.

Why a JVM does this

The kubelet compares the container's current memory usage with the new limit. A JVM's usage is dominated by committed heap, and the JVM does not return committed heap just because the objects in it are gone.

In my test the JVM sat idle for more than a minute with ZUncommitDelay=1 and gave nothing back. The likely reason is that no garbage-collection cycle ran after the data was dropped, so there was nothing to trigger an uncommit. I did not confirm that, and the behavior depends on the collector. Don't count on a JVM to release by itself.

`PodResizeInProgress` vs `PodResizePending`

Condition	What it means	What I saw
`PodResizeInProgress`	The kubelet accepted the new resources and is applying them.	With `Error` and "below current usage", the kubelet is retrying a memory decrease.
`PodResizePending`	The resize can't be started yet.	`reason=Infeasible`, message `Node didn't have enough capacity: memory, requested: 68719476736, capacity: 8321712128`, when I asked for 64Gi on an 8 GB node.

If you see Infeasible or Deferred, the problem is node capacity. If you see InProgress plus "below current usage", the problem is the container's own memory, and the fix is below.

The fix: release memory from inside the JVM

On ZGC, lower the soft ceiling and force a collection:

kubectl exec jvm -- jcmd 1 VM.set_flag SoftMaxHeapSize 104857600
kubectl exec jvm -- jcmd 1 GC.run

Releasing while the resize was pending. I ran the two commands against the stuck pod. memory.current fell from 417 MB to 96 MB. The kubelet applied the resize on its next retry, 20 seconds after I started the release. memory.max became 300 MB, restartCount stayed 0, the condition cleared, and a ResizeCompleted event appeared.

Releasing before the patch. I recreated the pod, released first (424 MB → 101 MB), then patched. The resize applied 2 seconds after the patch. For this pod, the events were ResizeStarted and ResizeCompleted only. The Error condition never appeared.

Releasing first is cleaner: no error condition, no wait for a retry. But if you are already stuck, you don't need to re-patch. Free the memory and the kubelet finishes the job.

When it stays stuck

The resize stays "In Progress" if the JVM can't get below the new limit:

Live data is bigger than the new limit. If the heap holds more live objects than the limit, I'd expect no collection to free enough, and the resize to stay pending. I didn't test this case. Pick a limit that fits.
The ceiling is fixed. -Xmx can't change on a running JVM. The new limit must be above -Xmx plus metaspace, code cache, thread stacks and direct buffers. If it isn't, you need a restart, which is what resizePolicy: RestartContainer is for.
Memory outside the heap. SoftMaxHeapSize only moves the heap. Native memory and direct buffers stay.

A debugging checklist

# 1. Which condition? InProgress or Pending, and the reason
kubectl get pod <pod> -o jsonpath='{range .status.conditions[?(@.type=="PodResizeInProgress")]}{.reason}: {.message}{"\n"}{end}'
kubectl get pod <pod> -o jsonpath='{range .status.conditions[?(@.type=="PodResizePending")]}{.reason}: {.message}{"\n"}{end}'

# 2. What is accepted vs actually applied
kubectl get pod <pod> -o jsonpath='allocated={.status.containerStatuses[0].allocatedResources.memory} applied={.status.containerStatuses[0].resources.limits.memory}{"\n"}'

# 3. What the container is really using
kubectl exec <pod> -- sh -c 'cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max'

# 4. The events
kubectl get events --field-selector involvedObject.name=<pod>

If memory.current is above the new limit, the JVM is the thing to fix.

One gotcha from my own test: use the default strategic merge patch, as in the command above. With --type merge, the whole containers list is replaced, and the API server rejects it with "resource requests cannot be removed".

What I didn't test

Collectors other than ZGC in a cluster. I tested ZGC, Shenandoah and G1 in a container separately, in another article, but not through a kubelet.
A multi-container pod, or any version other than v1.35.8.
Whether the retry interval changes under load, or what happens if usage spikes right after the kubelet's check.
How long a release takes on a large heap under real traffic.

Reproduce it

You need Docker, kind and kubectl. Hold.java:

import java.util.*;
public class Hold {
  public static void main(String[] a) throws Exception {
    List<byte[]> l = new ArrayList<>();
    for (int i = 0; i < 300; i++) { byte[] b = new byte[1024*1024]; Arrays.fill(b,(byte)1); l.add(b); }
    System.out.println("allocated 300MB"); System.out.flush();
    Thread.sleep(10000);
    l.clear(); l = null;
    System.out.println("released refs"); System.out.flush();
    Thread.sleep(600000);
  }
}

pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: jvm
spec:
  containers:
  - name: app
    image: eclipse-temurin:21-jdk
    imagePullPolicy: IfNotPresent
    command: ["java","-XX:+UseZGC","-XX:ZUncommitDelay=1","-Xmx600m","/w/Hold.java"]
    volumeMounts: [{name: w, mountPath: /w}]
    resources:
      requests: {cpu: 500m, memory: 900Mi}
      limits:   {cpu: 500m, memory: 900Mi}
    resizePolicy:
    - {resourceName: memory, restartPolicy: NotRequired}
    - {resourceName: cpu, restartPolicy: NotRequired}
  volumes:
  - name: w
    configMap: {name: hold}

kind create cluster --name resize-test --image kindest/node:v1.35.8
kubectl create configmap hold --from-file=Hold.java
kubectl apply -f pod.yaml
kubectl wait --for=condition=Ready pod/jvm --timeout=240s
# wait ~15 s until the JVM logs "released refs", then:
kubectl patch pod jvm --subresource resize -p \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"300Mi"},"limits":{"memory":"300Mi"}}}]}}'
kubectl get pod jvm -o yaml        # PodResizeInProgress, reason Error
kubectl exec jvm -- jcmd 1 VM.set_flag SoftMaxHeapSize 104857600
kubectl exec jvm -- jcmd 1 GC.run  # the kubelet finishes on its next retry
kind delete cluster --name resize-test

The first image pull took about 90 seconds on my machine. The megabyte numbers will differ a little on yours.

FAQ

Why is my Kubernetes pod resize stuck in progress?

The kubelet will not lower a memory limit below the container's current usage, and it keeps retrying. Check kubectl get pod <pod> -o yaml for PodResizeInProgress with a message saying "below current usage". Free memory inside the container and the resize finishes.

What is the difference between PodResizePending and PodResizeInProgress?

PodResizePending means the resize can't start yet. I saw it with reason=Infeasible when the request exceeded the node's capacity. PodResizeInProgress means the kubelet accepted the resize and is applying it. For a memory decrease it stays that way while usage is above the new limit.

Why won't the kubelet complete an in-place memory decrease for my Java pod?

A JVM keeps its committed heap after garbage collection, so the container's usage stays above the new limit. On ZGC or Shenandoah, lower SoftMaxHeapSize and run GC.run. On G1, use GC.run or G1PeriodicGCInterval.

Will the pod be OOM-killed while the resize is stuck?

Not in my test. The pod kept running at 900Mi, with 0 restarts, until I freed memory. The Kubernetes docs say the check is best-effort, with no guarantees.

Do I need to patch again after freeing memory?

No. The kubelet retries on its own. In my run it completed 20 seconds after I released memory, with no new patch.

How long does the kubelet take to apply the resize after the JVM releases memory?

20 seconds in the run where I released after the patch (the kubelet was retrying about every 11 to 15 seconds). If you release before the patch, it applied 2 seconds after.

Where Warden fits

I maintain mnemo-jvm-warden, an open-source Kubernetes controller and sidecar that does the release-then-resize order. The sidecar lowers the soft heap ceiling where the collector supports it, forces GC.run, and checks that the process's resident memory dropped. Only then does the resize reach the kubelet. That order is the second case in this article, where the error never appears. I haven't tested that exact path in the cluster in this article.

You don't need a tool for the fix: the two jcmd commands above work on their own.

Your G1 Pod Ignores SoftMaxHeapSize. Here's What Works Instead

BaoKhang — Sun, 08 Nov 2026 18:39:14 +0000

G1GC vs. ZGC vs. Shenandoah for Kubernetes 1.35 In-Place Pod Resize

When a pod shrinks in place, the JVM has to hand memory back to the operating system before the new limit applies. I tested how G1, ZGC and Shenandoah do that in a Linux container. All three released memory quickly in my test, but they differ in what triggers the release and in what SoftMaxHeapSize does. G1 accepts that flag and silently ignores it.

The short answer

ZGC and Shenandoah: SoftMaxHeapSize works at runtime. Lower it, run GC.run, and memory drops.
G1: SoftMaxHeapSize is accepted by jcmd and has no effect. By default nothing returns memory. Two things did, in my test: an explicit GC.run, and G1PeriodicGCInterval set to a few seconds.
Release speed: I measured no real difference. Each collector released a 300 MB working set within about 1 to 7 seconds. I did not measure large heaps or busy applications, which is where I'd expect differences, so treat "G1 is slower" as unproven.
The real G1 difference is the missing ceiling. Nothing holds G1's heap below a target after you release, so you size -Xmx for the peak and accept that it can grow back.
Lowering the limit first killed all three (OOMKilled, exit 137). Release first, then resize.

What each collector gives you

	ZGC	Shenandoah	G1
`SoftMaxHeapSize` changes behavior	Yes	Yes	No (flag accepted, no effect)
Returns memory by default	After `ZUncommitDelay` (300 s by default)	After `ShenandoahUncommitDelay`	No, not without a trigger
What triggers a release	`SoftMaxHeapSize` + `GC.run`, or the uncommit delay	Same	`GC.run`, or `G1PeriodicGCInterval`
Pause cost of the trigger	Not measured	Not measured	`GC.run` is a full, stop-the-world GC. Not measured at scale.

The test

The setup is the same as in my earlier test of RestartContainer: Eclipse Temurin 21.0.13 in a Linux container on Docker Desktop, cgroup v2, and a small program that allocates and touches 300 MB, holds it for 10 seconds, then drops all of it and idles. docker update --memory=300m lowers the cgroup limit on the running container in place. That is the kernel mechanism a pod resize uses, but there is no kubelet.

For each collector, I ran two cases. Naive: lower the limit straight away. Ordered: set SoftMaxHeapSize=100MB, run GC.run, wait until memory.current falls, then lower the limit.

Collector	`memory.current` before	Naive: lower limit to 300 MB	Ordered: release first, then lower
ZGC (`-Xmx600m`, limit 900 MB)	403 MB	OOMKilled, exit 137	404 → 89 MB in 3 s, then still running
Shenandoah (`-Xmx500m`, limit 700 MB)	372 MB	OOMKilled, exit 137	361 → 64 MB in 1 s, then still running
G1 (`-Xmx600m`, 4 MB regions, limit 900 MB)	456 MB	OOMKilled, exit 137	463 → 143 MB in 2 s, then still running

G1 passes the ordered case, but I want to be exact about why. In that run I set SoftMaxHeapSize too, and it did nothing. The explicit GC.run did the work.

What actually released memory on G1

I tested each G1 lever on its own, starting from about 460 to 530 MB in use:

Lever on G1	Result
Nothing (default), wait 25 s	459 → 459 MB. No release.
`SoftMaxHeapSize=100MB` only, wait 20 s	535 → 535 MB. No effect.
`GC.run` only	472 → 150 MB
`G1PeriodicGCInterval=3000` set live, no `GC.run`	518 → 156 MB in 7 s

SoftMaxHeapSize is listed as a manageable flag on G1 in JDK 21, so jcmd VM.set_flag returns "Command executed successfully". That success message is misleading. Nothing in G1 reads the flag. If your automation sets the flag and then checks that the command succeeded, it will believe it lowered the ceiling on G1 when it did not.

How to make G1 release memory back to the OS

Pick one of these. I tested the first two.

Run GC.run from outside. jcmd <pid> GC.run triggers a full GC, after which G1 shrinks its heap. It is immediate, and it is a stop-the-world collection, so its pause grows with the live set.
Enable periodic GC. G1PeriodicGCInterval is a manageable flag, default 0 (off). Set it at startup, or live: jcmd <pid> VM.set_flag G1PeriodicGCInterval 3000 (milliseconds). In my test, memory fell within 7 seconds on an idle JVM. A related flag, G1PeriodicGCSystemLoadThreshold, is also manageable and, per JEP 346, can hold the periodic GC back while the system is busy. I did not test it.
Tune MinHeapFreeRatio and MaxHeapFreeRatio. These are manageable and control how much free heap G1 keeps after a collection. I did not test them here.

What this means for in-place resize

Release before you resize, on every collector. The failure was identical across all three: lowering the limit while the JVM held committed memory got it killed.
Don't trust "command succeeded" on G1. Verify that memory.current actually dropped, then resize. In Kubernetes, the kubelet's check also compares current usage to the new limit and holds the resize while usage is higher, although the docs say that check is best-effort.
Plan the ceiling on G1. With ZGC and Shenandoah, SoftMaxHeapSize keeps the heap near your target after release. On G1 nothing does, so a G1 heap that was just released can grow back toward -Xmx before the limit drops. I did not test how fast. The practical rule is the one from my earlier article: the smallest limit must exceed -Xmx plus non-heap memory, and on G1 you cannot rely on a soft ceiling to enforce it.
Mind the delay on ZGC. ZUncommitDelay defaults to 300 seconds. I shortened it to 1 second for the test. Without that, a ZGC release can take minutes unless you also lower the soft ceiling and wait for the uncommit.

What I didn't test

Heap sizes above about 600 MB, live data that doesn't die, or JVMs under load. A full GC on G1 can cost far more at scale, and I haven't measured it.
Whether G1 re-expands after release.
G1PeriodicGCSystemLoadThreshold, MinHeapFreeRatio and MaxHeapFreeRatio.
A real kubelet. I used docker update.
G1 with the default region size. I set 4 MB regions because my test allocates 1 MB arrays, which are "humongous" objects in G1's default configuration, and the first G1 run ran out of heap.

Reproduce it

You need Docker. Save the files below in an empty folder.

Hold.java:

import java.util.*;
public class Hold {
  public static void main(String[] a) throws Exception {
    List<byte[]> l = new ArrayList<>();
    for (int i = 0; i < 300; i++) { byte[] b = new byte[1024*1024]; Arrays.fill(b,(byte)1); l.add(b); }
    System.out.println("allocated 300MB"); System.out.flush();
    Thread.sleep(10000);
    l.clear(); l = null;
    System.out.println("released refs"); System.out.flush();
    Thread.sleep(600000);
  }
}

run.sh runs the naive and ordered cases for any collector:

#!/bin/bash
# usage: ./run.sh <naive|ordered> <jvm flags...>      (run from the folder containing Hold.java)
# Example: LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1
set -u
mode=$1; shift; n=warden-test
docker rm -f $n >/dev/null 2>&1
docker run --rm -v "$PWD":/w -w /w eclipse-temurin:21-jdk javac Hold.java
docker run -d --name $n --memory=${LIM:-700}m --memory-swap=${LIM:-700}m -v "$PWD":/w -w /w \
  eclipse-temurin:21-jdk java "$@" -Xmx${XMX:-500}m Hold >/dev/null
for i in $(seq 1 60); do docker logs $n 2>&1 | grep -q 'released refs' && break; sleep 1; done; sleep 2
mem() { echo $(( $(docker exec $n cat /sys/fs/cgroup/memory.current) / 1048576 )); }
echo "memory.current after the app dropped its data: $(mem) MB"
if [ "$mode" = ordered ]; then
  docker exec $n jcmd 1 VM.set_flag SoftMaxHeapSize 104857600   # 100 MB soft ceiling
  docker exec $n jcmd 1 GC.run
  for i in $(seq 1 30); do [ "$(mem)" -lt 250 ] && break; sleep 1; done
  echo "memory.current after SoftMaxHeapSize + GC.run: $(mem) MB"
fi
docker update --memory=300m --memory-swap=300m $n        # in-place cgroup change, no restart
sleep 8
docker inspect -f 'running={{.State.Running}} OOMKilled={{.State.OOMKilled}} exit={{.State.ExitCode}}' $n
docker rm -f $n >/dev/null 2>&1

chmod +x run.sh
LIM=900 XMX=600 ./run.sh naive   -XX:+UseG1GC -XX:G1HeapRegionSize=4m
LIM=900 XMX=600 ./run.sh ordered -XX:+UseG1GC -XX:G1HeapRegionSize=4m
LIM=900 XMX=600 ./run.sh naive   -XX:+UseZGC -XX:ZUncommitDelay=1
LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1
LIM=700 XMX=500 ./run.sh naive   -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000
LIM=700 XMX=500 ./run.sh ordered -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000

g1-probe.sh tests each G1 lever on its own:

#!/bin/bash
# Which G1 lever actually returns memory? Run from the folder containing Hold.java.
set -u
docker run --rm -v "$PWD":/w -w /w eclipse-temurin:21-jdk javac Hold.java
start() { docker rm -f $1 >/dev/null 2>&1
  docker run -d --name $1 --memory=900m --memory-swap=900m -v "$PWD":/w -w /w eclipse-temurin:21-jdk \
    java -XX:+UseG1GC -XX:G1HeapRegionSize=4m -Xmx600m Hold >/dev/null
  for i in $(seq 1 60); do docker logs $1 2>&1 | grep -q 'released refs' && break; sleep 1; done; sleep 2; }
mem() { echo $(( $(docker exec $1 cat /sys/fs/cgroup/memory.current) / 1048576 )); }

echo "1) nothing (default), wait 25s";           start g1; b=$(mem g1); sleep 25; echo "   $b MB -> $(mem g1) MB"
echo "2) SoftMaxHeapSize=100MB only, wait 20s";  start g1; b=$(mem g1)
docker exec g1 jcmd 1 VM.set_flag SoftMaxHeapSize 104857600 >/dev/null; sleep 20; echo "   $b MB -> $(mem g1) MB"
echo "3) GC.run only";                           start g1; b=$(mem g1)
docker exec g1 jcmd 1 GC.run >/dev/null; sleep 4; echo "   $b MB -> $(mem g1) MB"
echo "4) G1PeriodicGCInterval=3000 set live";    start g1; b=$(mem g1)
docker exec g1 jcmd 1 VM.set_flag G1PeriodicGCInterval 3000 >/dev/null
for i in $(seq 1 40); do [ "$(mem g1)" -lt 250 ] && break; sleep 1; done; echo "   $b MB -> $(mem g1) MB (after ${i}s)"
docker rm -f g1 >/dev/null 2>&1

Expect OOMKilled=true exit=137 for naive runs and running=true for ordered ones. The megabyte numbers will differ a little on your machine.

FAQ

Which garbage collector is best for Kubernetes in-place pod resize?

In my test, ZGC and Shenandoah are the easiest, because SoftMaxHeapSize changes their behavior and keeps the heap near a target. G1 also works if something triggers a release, but it has no soft ceiling. Pick the collector your latency requirements need first, and then use the matching release method.

How do I make G1 release memory back to the OS?

Run jcmd <pid> GC.run, or set G1PeriodicGCInterval to a value in milliseconds (default 0, off). In my test both returned memory within a few seconds on a small idle heap.

Does `SoftMaxHeapSize` work with G1?

No. jcmd VM.set_flag SoftMaxHeapSize succeeds on G1 in JDK 21, but I saw no change in memory over 20 seconds, so I conclude G1 ignores it. It works on ZGC and Shenandoah.

Why was my pod OOMKilled after an in-place memory decrease?

The limit dropped while the JVM still held more committed memory than the new limit allows. In my test this killed the JVM on G1, ZGC and Shenandoah every time. Release memory first, check the resident memory, then lower the limit.

Is `GC.run` safe to use in production?

It triggers a full collection. On G1 that is stop-the-world, and its pause grows with the live set. I only tested a 600 MB heap on an idle JVM, so measure the pause on your own heap before relying on it.

Do I still need `RestartContainer`?

Only if the heap ceiling has to move. See my earlier article on RestartContainer for the case where it is and isn't required.

Where Warden fits

I maintain mnemo-jvm-warden, an open-source Kubernetes controller and sidecar that does the release-then-resize order for you. From its code and docs, it adjusts to the collector like this:

Detection: the agent reads the target's GC beans and records, per collector, whether a soft max and uncommit are supported. ZGC and Shenandoah support both. G1 supports uncommit only. Serial, Parallel and Epsilon are treated as unsupported, and the agent stays read-only for them.
Release: it sets SoftMaxHeapSize only where the collector has it, so on G1 that step is skipped instead of being sent and ignored. It then forces GC.run and polls the committed heap until it drops and holds steady. It checks the process's resident memory before it shrinks the pod.
Boost on G1: a G1 workload that wants a Boost lease must already launch with -Xmx covering the largest heapTarget it will request, because there is no soft ceiling to raise later. The agent rejects a heap target above the launch-time -Xmx for every collector, and the Boost docs note that G1 has no live soft-max knob, so a G1 Boost only ever confirms that the allocation change landed.

The conclusions above do not depend on any tool. The jcmd commands work without it.

Why Kubernetes 1.35 resizePolicy: RestartContainer Isn't Required for ZGC/Shenandoah

BaoKhang — Sun, 08 Nov 2026 17:51:08 +0000

Why Kubernetes 1.35 resizePolicy: RestartContainer Isn't Required for ZGC/Shenandoah

A lot of current advice says to set resizePolicy: RestartContainer for memory on every JVM container. For growing the heap ceiling, that advice is right. For shrinking and regrowing within a ceiling you chose at launch, I tested ZGC and Shenandoah in a Linux container and found a restart isn't needed, as long as the JVM releases memory before the limit drops.

The short version

Kubernetes 1.35 can change a running container's memory in place (the pods/resize subresource is GA).
A JVM fixes its maximum heap at startup. Raising the limit in place does not raise the heap. Two popular articles are right about this.
That does not mean every memory change needs a restart. Three different operations hide inside "resize memory", and only one of them requires it.
On ZGC and Shenandoah you can lower and raise the heap's soft ceiling at runtime. In my test, releasing memory first let a container's limit drop from 900 MB (or 700 MB) to 300 MB with no restart. Lowering the limit first got the JVM OOM-killed every time.

What the advice says, and where it's right

Rack2Cloud's VPA in production article says: "-Xmx is a startup parameter. The JVM reads it once at initialization and sets the maximum heap ceiling for the lifetime of the process." It concludes that resizePolicy: RestartContainer for memory "is the correct setting for JVM services, not NotRequired". Its Kubernetes 1.35 article lists RestartContainer as appropriate "for memory decreases on workloads where the allocator won't release memory without a restart".

Shubham Chepe's post on the JVM and in-place resize gives the same prescription: "CPU NotRequired, memory RestartContainer". He explains why: "If you raise the memory limit in place, the heap ceiling doesn't grow." On shrinking he says it is "riskier still, since the JVM may already be using more than the new limit allows."

Both are correct that:

the heap ceiling comes from -Xmx or -XX:MaxRAMPercentage at launch, and
a bigger cgroup limit does not make a running JVM use it.

If you need the heap ceiling itself to move, you need a restart. This article doesn't dispute that.

Three operations, not one

Operation	Needs a restart?
Grow beyond the `-Xmx` the JVM launched with	Yes. The ceiling is fixed.
Shrink the container's memory	Not on ZGC or Shenandoah, if the JVM releases memory first.
Grow back up to the launch-time ceiling	No. Raise the container limit, then raise the soft ceiling.

The advice treats all three as the first one. That is the right call if you size -Xmx to today's limit. It is the wrong one if you set -Xmx for the peak and treat memory below that as adjustable.

ZGC and Shenandoah make this possible because of SoftMaxHeapSize, a manageable flag. You can change it on a running JVM:

jcmd <pid> VM.set_flag SoftMaxHeapSize <bytes>
jcmd <pid> GC.run

The collector tries to stay under the soft limit and uncommits what it frees. -Xmx remains the hard ceiling.

The test

I wanted to see what actually happens, so I ran it. The setup:

Eclipse Temurin 21.0.13 in a Linux container on Docker Desktop, on cgroup v2 (/sys/fs/cgroup/memory.current was present).
A small program that allocates and touches 300 MB, holds it for 10 seconds, then drops all references and idles.
docker update --memory=300m --memory-swap=300m to lower the cgroup limit on the running container. This changes memory.max in place, which is the same kernel mechanism a pod resize uses, but it is not the kubelet.
Two runs per collector. Naive: lower the limit straight away. Ordered: set SoftMaxHeapSize=100MB, run GC.run, wait for memory.current to fall, and only then lower the limit.
Uncommit delays were shortened (ZUncommitDelay=1, ShenandoahUncommitDelay=1000) so the test finishes quickly. Defaults are longer.

Results:

Collector	`memory.current` before	Naive: lower limit to 300 MB	Ordered: release first, then lower
ZGC (`-Xmx600m`, limit 900 MB)	403 MB	OOMKilled, exit 137	404 → 89 MB in 3 s, then limit lowered: still running
Shenandoah (`-Xmx500m`, limit 700 MB)	372 MB	OOMKilled, exit 137	361 → 64 MB in 1 s, then limit lowered: still running

With the ordered sequence, memory.max read back as 314572800 (300 MB) and the container never restarted.

What the kubelet adds

My test used Docker, so it had no kubelet. In a real cluster, the kubelet tries to avoid an OOM kill when a memory limit goes down: according to the Kubernetes docs, if current usage is above the requested limit, the resize is skipped and stays "In Progress". The docs call this best-effort, with no guarantee, and note a race where usage can spike right after the check.

For a JVM, that has a practical consequence. Committed heap counts as memory in use even when most of it is free, so the resize usually sits pending until the JVM gives memory back. Releasing first is what gets usage under the new limit, so the kubelet applies the resize. The ordered sequence works with the kubelet's check, not against it.

When RestartContainer is still the right answer

You need the heap ceiling to change. -Xmx is fixed per process.
You run G1 and haven't enabled periodic GC. G1 has no soft ceiling and returns heap mainly at a full GC or concurrent cycle, so release is slower and less predictable. I didn't test it here.
You can accept the restart. A restart discards compiled code, loaded classes, pools and caches, and a service with few replicas loses capacity while it warms up. If that is fine for you, RestartContainer is simpler and needs no extra tooling.
Your limit is already close to -Xmx plus non-heap memory. Shrinking safely needs room: the smallest limit must exceed -Xmx plus metaspace, code cache, thread stacks and direct buffers.

What this test does not show

It is a synthetic program, not a production service, and I tested ZGC and Shenandoah on one JDK build.
ZGC and Shenandoah used slightly different heap and limit sizes (600/900 MB and 500/700 MB). My first ZGC run at 500 MB ran out of heap while filling 300 MB, so I gave it more room.
It was not run through a kubelet. The kubelet's check will change what you see on a real cluster.
I haven't measured how long a release takes on a large heap under real load.

Reproduce it

You need Docker. Save the two files below in an empty folder.

Hold.java allocates and touches 300 MB, holds it for 10 seconds, then drops it:

import java.util.*;
public class Hold {
  public static void main(String[] a) throws Exception {
    List<byte[]> l = new ArrayList<>();
    for (int i = 0; i < 300; i++) { byte[] b = new byte[1024*1024]; Arrays.fill(b,(byte)1); l.add(b); }
    System.out.println("allocated 300MB"); System.out.flush();
    Thread.sleep(10000);
    l.clear(); l = null;
    System.out.println("released refs"); System.out.flush();
    Thread.sleep(600000);
  }
}

run.sh starts a container, optionally releases memory first, then lowers the cgroup limit in place:

#!/bin/bash
# usage: ./run.sh <naive|ordered> <jvm flags...>      (run from the folder containing Hold.java)
# Example: LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1
set -u
mode=$1; shift; n=warden-test
docker rm -f $n >/dev/null 2>&1
docker run --rm -v "$PWD":/w -w /w eclipse-temurin:21-jdk javac Hold.java
docker run -d --name $n --memory=${LIM:-700}m --memory-swap=${LIM:-700}m -v "$PWD":/w -w /w \
  eclipse-temurin:21-jdk java "$@" -Xmx${XMX:-500}m Hold >/dev/null
for i in $(seq 1 60); do docker logs $n 2>&1 | grep -q 'released refs' && break; sleep 1; done; sleep 2
mem() { echo $(( $(docker exec $n cat /sys/fs/cgroup/memory.current) / 1048576 )); }
echo "memory.current after the app dropped its data: $(mem) MB"
if [ "$mode" = ordered ]; then
  docker exec $n jcmd 1 VM.set_flag SoftMaxHeapSize 104857600   # 100 MB soft ceiling
  docker exec $n jcmd 1 GC.run
  for i in $(seq 1 30); do [ "$(mem)" -lt 250 ] && break; sleep 1; done
  echo "memory.current after SoftMaxHeapSize + GC.run: $(mem) MB"
fi
docker update --memory=300m --memory-swap=300m $n        # in-place cgroup change, no restart
sleep 8
docker inspect -f 'running={{.State.Running}} OOMKilled={{.State.OOMKilled}} exit={{.State.ExitCode}}' $n
docker rm -f $n >/dev/null 2>&1

Then run the four cases:

chmod +x run.sh
LIM=900 XMX=600 ./run.sh naive   -XX:+UseZGC -XX:ZUncommitDelay=1
LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1
LIM=700 XMX=500 ./run.sh naive   -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000
LIM=700 XMX=500 ./run.sh ordered -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000

Expect OOMKilled=true exit=137 for the naive runs and running=true OOMKilled=false for the ordered ones. The exact megabyte numbers will differ a little on your machine.

FAQ

Do I need `resizePolicy: RestartContainer` for JVM memory?

Only if the heap ceiling has to move. A JVM fixes its maximum heap at startup, so growing beyond the -Xmx it launched with needs a restart. Shrinking, and growing back up to the launch ceiling, did not need one on ZGC or Shenandoah in my test, as long as the JVM released memory before the limit dropped.

Why doesn't raising the container's memory limit give the JVM more heap?

The JVM computes its maximum heap once, from -Xmx or from -XX:MaxRAMPercentage applied to the limit it saw at startup. A later in-place change to the cgroup limit does not change that number.

Can I change `-Xmx` on a running JVM?

No. What you can change on ZGC and Shenandoah is the soft ceiling, SoftMaxHeapSize, which sits at or below -Xmx.

What happens if I lower the memory limit without releasing memory first?

In my test the JVM was OOM-killed (exit code 137) every time, on both ZGC and Shenandoah. On a real cluster the kubelet makes a best-effort attempt to avoid this and may keep the resize pending while usage is above the new limit. The Kubernetes docs say this check gives no guarantees, because usage can spike right after it.

How do I make a JVM give memory back without restarting it?

On ZGC and Shenandoah, lower the soft ceiling and force a collection:

jcmd <pid> VM.set_flag SoftMaxHeapSize <bytes>
jcmd <pid> GC.run

Then check that the process's resident memory dropped before you lower the limit. The collector's uncommit delay applies: I shortened it for the test, and the defaults are longer.

Does this work with G1?

I didn't test it. G1 has no soft heap ceiling, so you cannot use the same flag. Plan for a slower, less predictable release, or use RestartContainer.

What is the default `resizePolicy`?

NotRequired for both CPU and memory, which means the change is applied without restarting the container. Setting RestartContainer for memory restarts the container, not the whole pod.

How small can the memory limit go?

The smallest limit must stay above -Xmx plus non-heap memory (metaspace, code cache, thread stacks, direct buffers) plus working-set headroom. For a typical Spring Boot service the non-heap part is commonly 100 to 150 MiB.

Where Warden fits

I maintain mnemo-jvm-warden, an open-source Kubernetes controller and sidecar that runs this order for you: lower the soft ceiling, force a GC, verify the process's resident memory dropped, and only then shrink the pod in place. To grow, it reverses the order. It does not change -Xmx, so the same sizing rule applies. There is a guide with the full reasoning, and a walkthrough on a kind cluster.

The point of this article doesn't depend on any tool. The jcmd commands above work without it.

I built a workflow that forces AI agents to teach you the code they write

BaoKhang — Sat, 18 Jul 2026 12:23:01 +0000

As a developer, I got a bit scared and fed-up of blindly merging massive pull requests and losing context on my own codebase, so I built FluencyLoop.

https://github.com/baokhang83/fluencyloop

It is an open-source, local-first workflow plugin for Claude Code and Codex that ensures your code and your codebase fluency are produced together.

Instead of letting an agent dump raw code, it tracks your technical familiarity locally and pauses to explain complex architectural choices or rejected alternatives only on topics you do not know yet. It also produces documentation and tracks the decisions it makes along the way.

Try it out ! It feels really good to get the AI caring for my understanding 🤣

DEV Community: BaoKhang

Why Kubernetes In-Place Memory Resize Gets Stuck "In Progress" on JVM Pods

The short answer

What it looks like

Why a JVM does this

PodResizeInProgress vs PodResizePending

The fix: release memory from inside the JVM

When it stays stuck

A debugging checklist

What I didn't test

Reproduce it

FAQ

Why is my Kubernetes pod resize stuck in progress?

What is the difference between PodResizePending and PodResizeInProgress?

Why won't the kubelet complete an in-place memory decrease for my Java pod?

Will the pod be OOM-killed while the resize is stuck?

Do I need to patch again after freeing memory?

How long does the kubelet take to apply the resize after the JVM releases memory?

Where Warden fits

Your G1 Pod Ignores SoftMaxHeapSize. Here's What Works Instead

The short answer

What each collector gives you

The test

What actually released memory on G1

How to make G1 release memory back to the OS

What this means for in-place resize

What I didn't test

Reproduce it

FAQ

Which garbage collector is best for Kubernetes in-place pod resize?

How do I make G1 release memory back to the OS?

Does SoftMaxHeapSize work with G1?

Why was my pod OOMKilled after an in-place memory decrease?

Is GC.run safe to use in production?

Do I still need RestartContainer?

Where Warden fits

Why Kubernetes 1.35 resizePolicy: RestartContainer Isn't Required for ZGC/Shenandoah

Why Kubernetes 1.35 resizePolicy: RestartContainer Isn't Required for ZGC/Shenandoah

The short version

What the advice says, and where it's right

Three operations, not one

The test

What the kubelet adds

When RestartContainer is still the right answer

What this test does not show

Reproduce it

FAQ

Do I need resizePolicy: RestartContainer for JVM memory?

Why doesn't raising the container's memory limit give the JVM more heap?

Can I change -Xmx on a running JVM?

What happens if I lower the memory limit without releasing memory first?

How do I make a JVM give memory back without restarting it?

Does this work with G1?

What is the default resizePolicy?

How small can the memory limit go?

Where Warden fits

I built a workflow that forces AI agents to teach you the code they write

`PodResizeInProgress` vs `PodResizePending`

Does `SoftMaxHeapSize` work with G1?

Is `GC.run` safe to use in production?

Do I still need `RestartContainer`?

Do I need `resizePolicy: RestartContainer` for JVM memory?

Can I change `-Xmx` on a running JVM?

What is the default `resizePolicy`?