The Container Got OOMKilled at 61% Heap Usage. Here's Why Kubernetes Wasn't Lying.

The problem

A service I ran was getting OOMKilled roughly twice a week. Pod restarts, a blip in the dashboards, on-call gets paged, everyone shrugs because "it auto-heals." Except it kept happening, and every time I pulled up the JVM heap graph right before the kill, it showed the same thing: 55-65% heap utilization. Not climbing. Not a leak. A perfectly healthy-looking heap, on a process the kernel had just killed for using too much memory.

That contradiction is what actually got me to stop shrugging.

Why it happens

The mental model most people carry around is "the JVM manages its own memory, so if the heap looks fine, memory is fine." That's wrong, and it's wrong in a way that costs real debugging time, because the thing enforcing your container's memory limit has never heard of "heap."

Kubernetes sets a cgroup memory limit on your pod. The kernel tracks one number against that limit: the cgroup's total resident memory — every byte the process has touched and is holding onto. That includes the JVM heap, yes, but also:

• Thread stacks. Default stack size is often 1MB. A connection-pool-per-request pattern or an under-tuned executor that spins up 300 threads under load just quietly reserved ~300MB that no heap dashboard will ever show you.
• Metaspace (class metadata) and the JIT code cache.
• Direct/native ByteBuffers — this is the one that actually got us. We used a Netty-based client for an internal RPC path, and Netty allocates a large chunk of its buffer pool off-heap by design, specifically so GC doesn't have to touch it. Great for GC pause times. Invisible to a dashboard that only scrapes jvm.memory.heap.used.
• Whatever native libraries and mmap'd files are in play.

Heap was 60% utilized. Total process RSS was creeping past the container's limit because of everything sitting outside the heap. The kernel's OOM killer doesn't negotiate — it doesn't wait for a GC pause, doesn't throw an OutOfMemoryError the JVM can log and handle, doesn't unwind a stack. It sends SIGKILL the instant the cgroup crosses its limit. The process is just gone. Kubernetes reports exit code 137 and "OOMKilled," and if you're only watching heap, you have zero signal about what actually happened.

What to do about it

Three things fixed this for us:

1. Stop sizing the heap as a fraction of host memory. Older JVMs (pre-JDK 10, or newer ones without the right flags) can miscalculate default heap sizing when they don't correctly respect cgroup limits. Set -XX:MaxRAMPercentage explicitly, and leave real headroom — heap should be a portion of the container limit, not the whole thing. We settled on heap at roughly 60% of the container memory limit, leaving the rest for stacks, metaspace, and off-heap buffers.
2. Monitor RSS, not just heap. container_memory_working_set_bytes (cAdvisor / kube-state-metrics) tracks what the kernel actually cares about. We added it next to the heap graph so the two could be compared directly — the gap between them became a metric we alerted on.
3. Count your threads and your off-heap allocators. -XX:NativeMemoryTracking=summary plus a periodic jcmd <pid> VM.native_memory will show you where non-heap memory is actually going. In our case, a connection pool with no cap on concurrent threads was the biggest single contributor once we actually measured it — heap monitoring alone never would have pointed there.

Key takeaways

• A container gets OOMKilled based on total RSS, not JVM heap usage — these are different numbers, and only one of them is what your dashboard probably shows you.
• Off-heap memory (thread stacks, metaspace, direct buffers, native allocations) doesn't show up in heap metrics but counts fully against your cgroup limit.
• If your service gets OOMKilled with heap graphs that look calm, stop looking at the heap. Compare heap usage to container RSS — the delta is where your answer is.
• Set an explicit heap ceiling well under the container limit, and monitor RSS directly. The kernel isn't going to give you a stack trace before it kills you.