Why Kubernetes In-Place Memory Resize Gets Stuck "In Progress" on JVM Pods

You lowered a JVM pod's memory with an in-place resize, and the pod now shows PodResizeInProgress, an Error reason, and an old memory limit that never changes. The kubelet is not broken. It refuses to lower a memory limit below the container's current usage, and a JVM keeps its committed heap even after the objects in it are garbage. The resize completes as soon as the JVM gives the memory back, which I confirmed on a Kubernetes 1.35.8 cluster. Below are the real outputs, the fix, and how long it took.

The short answer

• Cause: the kubelet will not apply a memory-limit decrease while the container's current usage is above the new limit. A JVM's usage stays high after garbage collection because it keeps the heap committed.
• Symptom: condition PodResizeInProgress=True with reason Error. containerStatuses[].allocatedResources shows the new value, but containerStatuses[].resources still shows the old one.
• It retries. The kubelet retried about every 11 to 15 seconds. The pod was not OOM-killed and did not restart. The check protected the JVM.
• Fix: make the JVM release memory, then let the resize finish. Releasing after the patch finished the resize on the next retry (about 20 s in my run). Releasing before the patch finished it 2 seconds after the patch, and the error never appeared.
• PodResizePending is a different problem. I saw it only when I asked for more memory than the node has (Infeasible).

What it looks like

I ran a JVM pod on a single-node kind cluster, Kubernetes v1.35.8. The pod has Guaranteed QoS with a 900Mi memory request and limit. The JVM (Temurin 21, ZGC, -Xmx600m) allocates 300 MB, holds it for 10 seconds, then drops all of it and idles. Inside the container, memory.current read 417 MB.

I shrank it to 300Mi through the resize subresource:

kubectl patch pod jvm --subresource resize -p \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"300Mi"},"limits":{"memory":"300Mi"}}}]}}'

A few minutes later, kubectl get pod jvm -o yaml (trimmed) showed:

status:
  conditions:
  - message: 'cannot decrease memory limits: [attempting to set pod memory limit (314572800)
      below current usage (435347456), attempting to set container "app" memory limit
      (314572800) below current usage (435347456)]'
    observedGeneration: 2
    reason: Error
    status: "True"
    type: PodResizeInProgress
  containerStatuses:
  - name: app
    allocatedResources:
      cpu: 500m
      memory: 300Mi        # what the kubelet accepted
    resources:
      limits:
        cpu: 500m
        memory: 900Mi      # what is actually applied
    restartCount: 0

At +10 s the condition was PodResizeInProgress=True with no reason. By +31 s the reason was Error. The cgroup never changed: memory.max stayed at 900 MB, and memory.current stayed at 417 MB for the whole minute I watched.

The events show a repeating error, and the kubelet log says why:

Normal   ResizeStarted   Pod resize started: {... "memory":"300Mi" ... "generation":2}
Warning  ResizeError     Pod resize error: {... "error":"cannot decrease memory limits: [attempting to set pod memory limit (314572800) below current usage (435347456) ...]"}
Warning  ResizeError     (repeats)

kubelet: "Allocated pod resize is not currently feasible" err="cannot decrease memory limits: ..."
kubelet: ... ResizePodInPlaceError ... (retries at 18:50:18, :33, :47, :58, 18:51:13, :25)

That matches what the Kubernetes docs say: before decreasing a container's memory limit, if memory usage exceeds the requested limit, the resize is skipped and the status remains in an "In Progress" state. They call this best-effort because usage can spike right after the check. What the docs don't show is the Error reason and the repeating ResizeError events, which look alarming but are the kubelet waiting.

Why a JVM does this

The kubelet compares the container's current memory usage with the new limit. A JVM's usage is dominated by committed heap, and the JVM does not return committed heap just because the objects in it are gone.

In my test the JVM sat idle for more than a minute with ZUncommitDelay=1 and gave nothing back. The likely reason is that no garbage-collection cycle ran after the data was dropped, so there was nothing to trigger an uncommit. I did not confirm that, and the behavior depends on the collector. Don't count on a JVM to release by itself.

PodResizeInProgress vs PodResizePending

Condition	What it means	What I saw
PodResizeInProgress	The kubelet accepted the new resources and is applying them.	With Error and "below current usage", the kubelet is retrying a memory decrease.
PodResizePending	The resize can't be started yet.	reason=Infeasible, message Node didn't have enough capacity: memory, requested: 68719476736, capacity: 8321712128, when I asked for 64Gi on an 8 GB node.

If you see Infeasible or Deferred, the problem is node capacity. If you see InProgress plus "below current usage", the problem is the container's own memory, and the fix is below.

The fix: release memory from inside the JVM

On ZGC, lower the soft ceiling and force a collection:

kubectl exec jvm -- jcmd 1 VM.set_flag SoftMaxHeapSize 104857600
kubectl exec jvm -- jcmd 1 GC.run

Releasing while the resize was pending. I ran the two commands against the stuck pod. memory.current fell from 417 MB to 96 MB. The kubelet applied the resize on its next retry, 20 seconds after I started the release. memory.max became 300 MB, restartCount stayed 0, the condition cleared, and a ResizeCompleted event appeared.

Releasing before the patch. I recreated the pod, released first (424 MB → 101 MB), then patched. The resize applied 2 seconds after the patch. For this pod, the events were ResizeStarted and ResizeCompleted only. The Error condition never appeared.

Releasing first is cleaner: no error condition, no wait for a retry. But if you are already stuck, you don't need to re-patch. Free the memory and the kubelet finishes the job.

When it stays stuck

The resize stays "In Progress" if the JVM can't get below the new limit:

• Live data is bigger than the new limit. If the heap holds more live objects than the limit, I'd expect no collection to free enough, and the resize to stay pending. I didn't test this case. Pick a limit that fits.
• The ceiling is fixed. -Xmx can't change on a running JVM. The new limit must be above -Xmx plus metaspace, code cache, thread stacks and direct buffers. If it isn't, you need a restart, which is what resizePolicy: RestartContainer is for.
• Memory outside the heap. SoftMaxHeapSize only moves the heap. Native memory and direct buffers stay.

A debugging checklist

# 1. Which condition? InProgress or Pending, and the reason
kubectl get pod <pod> -o jsonpath='{range .status.conditions[?(@.type=="PodResizeInProgress")]}{.reason}: {.message}{"\n"}{end}'
kubectl get pod <pod> -o jsonpath='{range .status.conditions[?(@.type=="PodResizePending")]}{.reason}: {.message}{"\n"}{end}'

# 2. What is accepted vs actually applied
kubectl get pod <pod> -o jsonpath='allocated={.status.containerStatuses[0].allocatedResources.memory} applied={.status.containerStatuses[0].resources.limits.memory}{"\n"}'

# 3. What the container is really using
kubectl exec <pod> -- sh -c 'cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max'

# 4. The events
kubectl get events --field-selector involvedObject.name=<pod>

If memory.current is above the new limit, the JVM is the thing to fix.

One gotcha from my own test: use the default strategic merge patch, as in the command above. With --type merge, the whole containers list is replaced, and the API server rejects it with "resource requests cannot be removed".

What I didn't test

• Collectors other than ZGC in a cluster. I tested ZGC, Shenandoah and G1 in a container separately, in another article, but not through a kubelet.
• A multi-container pod, or any version other than v1.35.8.
• Whether the retry interval changes under load, or what happens if usage spikes right after the kubelet's check.
• How long a release takes on a large heap under real traffic.

Reproduce it

You need Docker, kind and kubectl. Hold.java:

import java.util.*;
public class Hold {
  public static void main(String[] a) throws Exception {
    List<byte[]> l = new ArrayList<>();
    for (int i = 0; i < 300; i++) { byte[] b = new byte[1024*1024]; Arrays.fill(b,(byte)1); l.add(b); }
    System.out.println("allocated 300MB"); System.out.flush();
    Thread.sleep(10000);
    l.clear(); l = null;
    System.out.println("released refs"); System.out.flush();
    Thread.sleep(600000);
  }
}

pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: jvm
spec:
  containers:
  - name: app
    image: eclipse-temurin:21-jdk
    imagePullPolicy: IfNotPresent
    command: ["java","-XX:+UseZGC","-XX:ZUncommitDelay=1","-Xmx600m","/w/Hold.java"]
    volumeMounts: [{name: w, mountPath: /w}]
    resources:
      requests: {cpu: 500m, memory: 900Mi}
      limits:   {cpu: 500m, memory: 900Mi}
    resizePolicy:
    - {resourceName: memory, restartPolicy: NotRequired}
    - {resourceName: cpu, restartPolicy: NotRequired}
  volumes:
  - name: w
    configMap: {name: hold}

kind create cluster --name resize-test --image kindest/node:v1.35.8
kubectl create configmap hold --from-file=Hold.java
kubectl apply -f pod.yaml
kubectl wait --for=condition=Ready pod/jvm --timeout=240s
# wait ~15 s until the JVM logs "released refs", then:
kubectl patch pod jvm --subresource resize -p \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"300Mi"},"limits":{"memory":"300Mi"}}}]}}'
kubectl get pod jvm -o yaml        # PodResizeInProgress, reason Error
kubectl exec jvm -- jcmd 1 VM.set_flag SoftMaxHeapSize 104857600
kubectl exec jvm -- jcmd 1 GC.run  # the kubelet finishes on its next retry
kind delete cluster --name resize-test

The first image pull took about 90 seconds on my machine. The megabyte numbers will differ a little on yours.

FAQ

Why is my Kubernetes pod resize stuck in progress?

The kubelet will not lower a memory limit below the container's current usage, and it keeps retrying. Check kubectl get pod <pod> -o yaml for PodResizeInProgress with a message saying "below current usage". Free memory inside the container and the resize finishes.

What is the difference between PodResizePending and PodResizeInProgress?

PodResizePending means the resize can't start yet. I saw it with reason=Infeasible when the request exceeded the node's capacity. PodResizeInProgress means the kubelet accepted the resize and is applying it. For a memory decrease it stays that way while usage is above the new limit.

Why won't the kubelet complete an in-place memory decrease for my Java pod?

A JVM keeps its committed heap after garbage collection, so the container's usage stays above the new limit. On ZGC or Shenandoah, lower SoftMaxHeapSize and run GC.run. On G1, use GC.run or G1PeriodicGCInterval.

Will the pod be OOM-killed while the resize is stuck?

Not in my test. The pod kept running at 900Mi, with 0 restarts, until I freed memory. The Kubernetes docs say the check is best-effort, with no guarantees.

Do I need to patch again after freeing memory?

No. The kubelet retries on its own. In my run it completed 20 seconds after I released memory, with no new patch.

How long does the kubelet take to apply the resize after the JVM releases memory?

20 seconds in the run where I released after the patch (the kubelet was retrying about every 11 to 15 seconds). If you release before the patch, it applied 2 seconds after.

Where Warden fits

I maintain mnemo-jvm-warden, an open-source Kubernetes controller and sidecar that does the release-then-resize order. The sidecar lowers the soft heap ceiling where the collector supports it, forces GC.run, and checks that the process's resident memory dropped. Only then does the resize reach the kubelet. That order is the second case in this article, where the error never appears. I haven't tested that exact path in the cluster in this article.

You don't need a tool for the fix: the two jcmd commands above work on their own.