kkkevin

Posted on Nov 6 Originally published at opshubs.space

Java OOMKilled in Kubernetes: Find the Memory Pool with NMT

#kubernetes #java #devops #cloudnative

“Pile of Laptop & Desktop RAM Memory Modules” by Wilbysuffolk, CC BY-SA 4.0

A Java pod that restarts with OOMKilled is a frustrating problem, because the JVM often leaves no stack trace and no OutOfMemoryError. The kernel simply ends the process. This guide is for engineers who run Java services on Kubernetes and need to answer one question: which memory pool grew? We will use Native Memory Tracking (NMT) and jcmd to find out, then map the result to a fix.

Why Java OOMKilled is not a Java OutOfMemoryError

Two different things get confused. A Java OutOfMemoryError is thrown by the JVM when it cannot satisfy an allocation inside its own limits, for example when the heap is full. OOMKilled means the container as a whole used more memory than its Kubernetes limit. The Kubernetes documentation explains that memory limits are enforced by the kernel with out-of-memory kills, and that this enforcement is reactive: a container can briefly exceed the limit, and it is killed when the kernel detects memory pressure.

The container’s memory is more than the Java heap. It also includes class metadata, thread stacks, JIT-compiled code, GC data structures, direct buffers and memory from native libraries. A pod can be killed while the heap graph looks healthy. That is why heap dumps alone often do not explain it.

Confirm that it really was an OOM kill

Start by checking the termination reason. The Kubernetes memory tutorial shows that a killed container reports reason: OOMKilled and exitCode: 137, which is 128 + 9 (SIGKILL).

kubectl get pod my-app-7d9f -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
kubectl describe pod my-app-7d9f | grep -A6 "Last State"

Replace my-app-7d9f with your pod name. If the reason is something else (for example an error exit from your own code), this guide does not apply.

Do the budget arithmetic first

Before measuring, add up what the JVM is allowed to use. The java command reference gives these facts:

-XX:MaxRAMPercentage defaults to 25 percent, and the RAM it is a percentage of is limited by container constraints.
The default maximum code cache is 240 MB (48 MB without tiered compilation).
The default thread stack size on Linux/x64 is 1024 KB.
Metaspace is not limited by default (-XX:MaxMetaspaceSize), and direct buffer size is chosen automatically unless you set -XX:MaxDirectMemorySize.

Take a hypothetical pod with a 1024Mi limit whose JVM runs with -XX:MaxRAMPercentage=75, so the maximum heap is 768Mi. If the service has 200 threads on Linux/x64, their stacks can reserve up to 200 x 1 MB = 200 MB. Only the stack pages a thread actually touches count toward the container, so real use is usually lower, but in the worst case heap plus stacks is 968 MB. That leaves 56 MB of the 1024Mi limit for metaspace, code cache, GC structures, direct buffers and everything else. A budget this tight can explain a kill without any leak at all.

The JVM Container Memory Calculator does this addition for you with your own heap, metaspace, thread and direct memory inputs.

Enable Native Memory Tracking

NMT is off by default. According to the java command reference, -XX:NativeMemoryTracking accepts off, summary and detail. The Java troubleshooting guide states that enabling NMT costs a 5-10 percent performance drop, so turn it on in a test or canary environment first. It cannot be switched on in a running JVM; it needs a restart.

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:NativeMemoryTracking=summary"

This sets the flag through JAVA_TOOL_OPTIONS, which the JVM reads at startup, so you do not have to edit the image. Use detail only when summary points to a category and you need call sites.

Read the NMT summary with jcmd

Run jcmd inside the container. In many images the Java process is PID 1. jcmd ships with the JDK, so JRE-only or distroless images may not include it; in that case use an image variant that has the JDK tools, or an ephemeral debug container with a JDK that shares the pod’s process namespace. Run jcmd as the same user as the Java process, or it cannot attach.

kubectl exec my-app-7d9f -- jcmd 1 VM.native_memory

The output lists the total reserved and committed memory, then one block per category. The categories documented by Oracle include Java Heap, Class, Thread, Code, GC, Compiler, Internal, Symbol and Native Memory Tracking itself. The page notes that the list may change between releases. Focus on committed values, since committed memory is what counts against the container.

One limit matters here: NMT does not track allocations made by non-JVM native code, such as a JNI library. If the NMT total is far below what the container reports, suspect native libraries rather than the JVM.

Compare against a baseline

A single snapshot rarely shows growth. Take a baseline after warm-up, wait while the problem develops, then diff.

kubectl exec my-app-7d9f -- jcmd 1 VM.native_memory baseline
# ...wait until memory has grown...
kubectl exec my-app-7d9f -- jcmd 1 VM.native_memory summary.diff

The diff prints each category with a plus or minus delta. The category with the largest positive delta is your suspect. If you need allocation sites, restart with -XX:NativeMemoryTracking=detail and use detail.diff the same way.

Match the growing pool to a fix

Growing category	Typical cause	What to try
Java Heap	Heap max is too large for the limit, or real object growth	Lower `-XX:MaxRAMPercentage`, or take a heap dump with `-XX:+HeapDumpOnOutOfMemoryError` (disabled by default) and `-XX:HeapDumpPath`
Thread	Too many threads, each with a stack	Cap thread pools; reduce `-Xss` only if the code allows it
Class	Metaspace growth from class loading, for example repeated redeploys or dynamic proxies	Find the class loader leak; set `-XX:MaxMetaspaceSize` to fail early with a clear error
Code	Code cache filling toward its maximum	Check `-XX:ReservedCodeCacheSize` against your budget
GC	Collector structures that scale with heap size	Reduce heap, or leave more headroom
Nothing in NMT, but container grows	Direct buffers or native code outside NMT	Set `-XX:MaxDirectMemorySize`; review JNI or native libraries

Setting -XX:MaxMetaspaceSize or -XX:MaxDirectMemorySize does not remove a leak, but it turns a silent kernel kill into a Java OutOfMemoryError with a message, which is much easier to diagnose.

Set limits that leave headroom

Once you know the pools, size the container with a margin. A reasonable pattern is to set the heap as a percentage and leave the rest explicitly for non-heap memory:

resources:
  requests:
    memory: "1536Mi"
  limits:
    memory: "1536Mi"
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=60 -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=128m"

Line by line: the request equals the limit so the scheduler reserves what the pod can use; MaxRAMPercentage=60 gives a heap cap of about 921Mi (0.60 x 1536Mi); the metaspace and direct memory caps are bounded explicitly. These values are an illustration, not a recommendation. Replace them with the committed sizes you measured with NMT. Note that Mi is a binary unit; 1536M would be a different, smaller amount.

Key takeaways

Java OOMKilled means the whole container exceeded its limit; it is not the same as a Java OutOfMemoryError.
Confirm with reason: OOMKilled and exit code 137 before investigating.
Add up heap, stacks, code cache, metaspace and direct memory first; many kills are budget mistakes.
Enable -XX:NativeMemoryTracking=summary (5-10 percent overhead, restart needed), take a baseline, then run summary.diff.
NMT does not see memory allocated by non-JVM native code.
Cap metaspace and direct memory so failures are explicit and diagnosable.

Conclusion

Treat a Java OOMKilled as an accounting problem. Work out the budget, measure which pool grows with NMT diffs, and then either fix the growth or give the container honest headroom. Repeat the diff after each change to confirm that the growing category is the one you fixed.

DEV Community