Off-heap memory: where the JVM’s RAM goes

The heap is stable, the GC logs are clean, and yet the process RSS keeps climbing, day after day. A week later the pod gets OOMKilled. You take a heap dump, go hunting for the leak, and find nothing. There's nothing to find: the memory that's overflowing isn't in the heap.

A JVM consumes far more than its heap. Metaspace, thread stacks, direct buffers, JIT-compiled code, the GC's internal structures: all of it sits next to the heap, and none of it shows up in a heap dump. This article walks through that off-heap memory. What it's made of, how to measure it, and where to look when it grows.

Before anything else, make sure the problem is actually off-heap. If the heap fills up and collections stop freeing anything, that's a regular leak: Diagnosing a memory leak with a heap dump. And if you suspect the container limits instead, Tuning the JVM in a container covers the basics.

What lives next to the heap

The JVM process holds several memory areas, and the heap is just the biggest one. Around it:

• The Metaspace. It stores the metadata of loaded classes: structure, methods, bytecode. It grows with the number of classes, and it has no limit by default.
• The thread stacks. Each platform thread reserves its stack, 1 MB by default on 64-bit Linux. With 500 threads, that can reach 500 MB.
• The code cache. The native code produced by the JIT, up to 240 MB reserved by default.
• The GC structures. G1 maintains tables to track references between regions. Expect a few percent of the heap size, sometimes more than 10%.
• The direct buffers. The DirectByteBuffer objects allocated by NIO, Netty, or an HTTP client live entirely outside the heap.
• The native memory from libraries. Everything that goes through JNI allocates with malloc, out of the JVM's sight: compression, cryptography, native drivers.

Add it all up and you see why a 1 GB container with a 768 MB heap ends up getting killed. The heap did nothing wrong. The rest just had no room left.

Measuring with Native Memory Tracking

The JVM can break down where its memory goes. It's called Native Memory Tracking, and you enable it at startup:

-XX:NativeMemoryTracking=summary

The Oracle docs quote a 5 to 10% overhead, and summary mode costs less than that in practice. The flag does require a restart, though. Once the process is running, you query it with jcmd:

jcmd <pid> VM.native_memory summary scale=MB

The output lists each category, shortened here to the essentials:

Total: reserved=6318MB, committed=1704MB
-       Java Heap (reserved=4096MB, committed=1024MB)
-           Class (reserved=1024MB, committed=12MB)
-          Thread (reserved=250MB, committed=84MB)
-            Code (reserved=245MB, committed=52MB)
-              GC (reserved=200MB, committed=98MB)
-           Other (reserved=310MB, committed=310MB)
-       Metaspace (reserved=128MB, committed=96MB)

Two numbers per line, and the difference matters. Reserved is address space requested from the system, and it costs almost nothing. Committed is memory the JVM has actually claimed, and it's the one that tracks the RSS. When you're hunting for where the RAM goes, read the committed column and ignore the other one.

The categories speak for themselves. Metaspace holds the class metadata. Thread counts the stacks, and its committed number only counts the pages actually touched. Other includes, among other things, the direct buffers. Here, 250 threads and 310 MB of direct buffers next to a 1 GB heap: that already explains a good chunk of a container.

When you're chasing a leak, what you want is diff mode. Take a baseline, let the app run, then compare:

jcmd <pid> VM.native_memory baseline
# a few hours later
jcmd <pid> VM.native_memory summary.diff scale=MB

The output shows how each category moved since the baseline. The one that keeps climbing is your culprit, and you know right away where to dig.

The usual suspects

A growing Metaspace. It grows when classes get loaded but never unloaded. The classic causes: hot redeployments, and above all classes generated on the fly (proxies, reflection, scripting engines). Since it has no limit by default, it can push the container into OOMKilled without a sound. Setting -XX:MaxMetaspaceSize=256m doesn't fix anything, but it turns the silent death into an OutOfMemoryError: Metaspace with a stack trace, which is much easier to diagnose.

Too many threads. Each platform thread pays for its stack. An unbounded pool, or pools created per request and never shut down, show up directly on NMT's Thread line. A thread dump tells you how many there are and who created them. That's the subject of Reading a thread dump with jstack. Virtual threads don't have this problem, since their stacks live in the heap: Virtual threads in production.

Direct buffers piling up. The nastiest trap. A DirectByteBuffer allocates its memory outside the heap, but that memory is only returned to the system when the Java object itself gets collected by the GC. With a big heap and rare collections, dead buffers pile up waiting for a collection that never comes, and direct memory fills up while the heap sits half empty. The limit is set with -XX:MaxDirectMemorySize, which defaults to the max heap size. You can watch the pool over JMX, through the BufferPoolMXBean named direct, which most metrics agents expose.

The code cache and the GC. Rarely guilty. Their size settles after warmup. If they show up in an NMT diff after several days, that's unusual and worth a look, but always start with the three suspects above.

What NMT doesn't see

NMT only tracks allocations made by the JVM itself. A native library that allocates with its own malloc through JNI flies completely under the radar. The classic offender is zlib: an Inflater or Deflater that's never closed keeps its native buffer, and thousands of them add up to a real leak, invisible in NMT and in the heap dump alike.

The allocator itself can also work against you. glibc splits memory into arenas, 8 per core, and is bad at giving freed memory back to the system. If the app does a lot of native allocation, the RSS ends up well above the sum of the NMT categories, from fragmentation alone. The well-known fix in a container is a single environment variable:

MALLOC_ARENA_MAX=2

If the gap persists, the next step is to swap the allocator for jemalloc and turn on its profiling. It records the native allocation stacks and points at the library responsible. It's heavier to set up, but it's the tool that finds the leaks nothing else can see.

In short

An RSS that climbs while the heap looks healthy means off-heap memory. Enable -XX:NativeMemoryTracking=summary, read the committed column, and use summary.diff mode to see which category is growing. Metaspace, threads and direct buffers explain most cases. If NMT sees nothing, look at native allocations from libraries and at glibc fragmentation. And in a container, always keep a margin between the heap and the memory limit: it's not waste, it's the room the rest of the JVM lives in.