DEV Community

Aymen Jamaaoui
Aymen Jamaaoui

Posted on Originally published at aymenjamaaoui.dev on

Java in Docker and Kubernetes: avoid OOMKilled

The pod restarts. kubectl get pod shows OOMKilled, and the exit
code is 137. But the logs show no OutOfMemoryError, and the heap was
far from full. The JVM did not run out of memory: the container did.
The Linux kernel killed the process because it went over its limit.

This post shows what the JVM sees in a container, what adds to the
heap, how to reproduce the kill, and how to set the memory so that
everything fits. All the examples are in the
containers/
folder of the public java-gc-lab repository. They use the official
eclipse-temurin images: no image to build.

The Docker commands of this post

If you do not use Docker much, here are the options used below.

Option What it does
docker run <image> <command> Starts a container from the image and runs the command in it.
--rm Deletes the container when it stops.
--name over-limit Gives the container a name, for the next commands.
--memory=512m Memory limit of the container. This is the limit the JVM reads.
--memory-swap=512m Limit for memory + swap. Same value as --memory: no swap.
--cpus=2 CPU limit: the same as 2 CPUs of time.
-v "$PWD/containers":/app Mounts the local folder containers/ as /app in the container.
-d Runs the container in the background.
docker inspect <name> Shows the state of a stopped container: exit code, OOMKilled.
docker exec <name> <command> Runs a command in a running container.
docker logs <name> Shows what the container printed.

Watch out for swap: with --memory alone, Docker also gives the same
amount of swap. java -XshowSettings:system shows it: "Memory Limit:
1.00G", "Memory & Swap Limit: 2.00G". The container then uses swap
instead of being killed at the limit. To get what happens in
Kubernetes, I set --memory-swap to the same value.

What the JVM sees in a container

On Linux, the JVM reads the limits of the container (the cgroups): the
memory and the number of CPUs. This is the option
-XX:+UseContainerSupport, on by default. The JVM uses these limits to
size the heap, choose the GC and set the number of threads.

ShowLimits.java
prints what the JVM chose:

docker run --rm --memory=1g --cpus=2 -v "$PWD/containers":/app eclipse-temurin:25 java /app/ShowLimits.java

CPUs: 2
Max heap: 256 MiB
GC: Copy, MarkSweepCompact
Enter fullscreen mode Exit fullscreen mode

"Copy, MarkSweepCompact" is the Serial GC. With other limits, the same
results on Java 17, 21 and 25:

--memory --cpus Max heap GC
256m 1 126 MiB Serial
1g 2 256 MiB Serial
1791m 2 448 MiB Serial
1792m 2 448 MiB G1
4g 2 1024 MiB G1

The JVM gives these values itself, with no program:

docker run --rm --memory=1g --cpus=2 eclipse-temurin:25 java -XX:+PrintFlagsFinal -version | grep -E " MaxHeapSize | UseSerialGC | UseG1GC "
docker run --rm --memory=1g --cpus=2 eclipse-temurin:25 java -XshowSettings:system -version
Enter fullscreen mode Exit fullscreen mode

The first one prints the computed options (MaxHeapSize = 268435456,
UseSerialGC = true). The second one shows the limits read from the
container: "Provider: cgroupv2", "Effective CPU Count: 2", "Memory
Limit: 1.00G".

The default heap: 25 %, or 50 % for a small heap

Without -Xmx, the maximum heap is 25 % of the memory the JVM sees:
this is -XX:MaxRAMPercentage, 25 by default. In a 1 GB container,
that is 256 MB. The other 75 % are not all free: the rest of the JVM
needs them, as we will see below.

For a small heap, another option takes over: -XX:MinRAMPercentage,
50 by default. The documentation talks about a heap of "approximately
125 MB". This is what the table shows: with 256 MB, the heap is 126 MB,
not 64.

You set both options at startup:

java -XX:MaxRAMPercentage=50 -jar app.jar
Enter fullscreen mode Exit fullscreen mode

A percentage follows the container limit: if the limit goes from 1 to
2 GB, the heap follows. A fixed -Xmx does not change (see
the JVM options that matter).

Serial or G1: a question of CPUs and memory

The table has a surprise: with 2 CPUs and 1 GB, the JVM picks Serial,
not G1. Oracle's GC guide gives the rule: G1 on a "server-class"
machine, Serial otherwise. A machine is "server-class" if the JVM
detects "two or more processors and physical memory larger than or
equal to 1792 MB". The tests confirm it: Serial at 1791 MB, G1 at
1792 MB.

Serial stops the application during each GC, with one thread (the
available GCs are described in
the post on garbage collectors). For a
service with little memory, this is not always a problem. But you must
know it, and choose it. To force G1: -XX:+UseG1GC.

The number of CPUs also changes the number of GC threads. With 2 CPUs,
G1 has 2 parallel threads and 1 concurrent thread. The option
-XX:ActiveProcessorCount=<n> replaces the number of CPUs that the
JVM uses for these sizes. The documentation says it can be useful "when
running multiple Java processes in docker containers".

What about the CPU request of Kubernetes? In a pod with
requests.cpu: 1 and no CPU limit, the JVM sees all the CPUs of the
machine: 12 on my machine. The same with the Docker option
--cpu-shares (the cgroup parameter "cpu.shares"). A limit does reduce
the number of CPUs the JVM sees: with limits.cpu: "2" (or
--cpus=2), it sees 2.

Before Java 19: the JVM also used "cpu.shares" to count the CPUs,
and could see fewer than available. Java 19 ignores it
(JDK-8281181). In my tests, Temurin 17.0.20 also ignores it.

What adds to the heap

The container limit covers the whole process, not only the heap:

  • the Metaspace: the description of the loaded classes;
  • the thread stacks: by default 1024 KB per thread on Linux/x64, and 2048 KB on Linux/Aarch64 (for example Docker on an Apple Silicon Mac);
  • the code cache: the code compiled by the JIT;
  • the direct memory: the ByteBuffer.allocateDirect buffers. Without -XX:MaxDirectMemorySize, the JVM chooses the limit itself;
  • the GC data and other internal areas.

A heap at 75 % of the limit leaves 25 % for all of this. If the rest
needs more, the container is killed, even with a half-empty heap.

Reproduce the kill

OverLimit.java
keeps 250 MB in the heap, then takes direct memory by blocks of 10 MB,
with no end. A 512 MB container with no swap, max heap at 75 %
(384 MB):

docker run --name over-limit --memory=512m --memory-swap=512m -v "$PWD/containers":/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 /app/OverLimit.java

Heap: 250 MB used
Direct memory: 10 MB
...
Direct memory: 170 MB
Enter fullscreen mode Exit fullscreen mode

The output just stops. No OutOfMemoryError, no stack trace: the
process was killed. docker inspect confirms it:

docker inspect over-limit --format "exit={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}}"

exit=137 OOMKilled=true
Enter fullscreen mode Exit fullscreen mode

137 is 128 + 9: the process got signal 9 (SIGKILL), which cannot be
caught. The JVM could not write anything, and
-XX:+HeapDumpOnOutOfMemoryError does not help here: there was no
OutOfMemoryError. Depending on the version, the kill comes between
170 and 190 MB of direct memory.

See where the memory goes with NMT

Native Memory Tracking (NMT) counts the memory of the JVM by area. I
run it again with NMT, and the argument 100 stops the direct memory
at 100 MB; the program then waits 30 seconds:

docker run -d --name over-limit --memory=512m --memory-swap=512m -v "$PWD/containers":/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 -XX:NativeMemoryTracking=summary /app/OverLimit.java 100
docker exec over-limit jcmd 1 VM.native_memory summary scale=MB

Total: reserved=1875MB, committed=533MB
-                 Java Heap (reserved=384MB, committed=384MB)
-                     Class (reserved=1024MB, committed=2MB)
-                    Thread (reserved=26MB, committed=1MB)
-                      Code (reserved=245MB, committed=9MB)
-                     Other (reserved=100MB, committed=100MB)
-                 Metaspace (reserved=64MB, committed=9MB)
Enter fullscreen mode Exit fullscreen mode

(I removed some lines.) 1 is the process id of Java in the container.
The heap takes 384 MB, and the 100 MB of direct memory are in "Other".
This small program has few classes and threads; a real application has
more Metaspace, more threads and more compiled code. docker stats
--no-stream over-limit
shows the memory the container really uses:
about 430 MiB in my runs, out of 512. It is not the same measure: NMT
counts the memory "committed" by the JVM, docker stats the memory the
container uses. (A MiB, like the Mi of Kubernetes, is 1,048,576 bytes.)

Better than a kill: a Java error

With a limit on direct memory, the JVM stops before the container, with
a clear message:

docker run --rm --memory=512m --memory-swap=512m -v "$PWD/containers":/app eclipse-temurin:25 java -XX:MaxRAMPercentage=75 -XX:MaxDirectMemorySize=100m /app/OverLimit.java

Exception in thread "main" java.lang.OutOfMemoryError: Cannot reserve 10485760 bytes of direct buffer memory (allocated: 104857600, limit: 104857600)
Enter fullscreen mode Exit fullscreen mode

An error with a stack trace tells what is full. A kill tells nothing.
The other OutOfMemoryError messages are in
the post about OutOfMemoryError.

Choose MaxRAMPercentage

There is no good percentage for all applications. The rule is simple:
the maximum heap plus all the rest must fit in the limit. You measure
the rest with NMT, on the application under load.

In the example, the program needs 250 MB of heap and 200 MB of direct
memory. With 768 MB and 50 %:

docker run --name over-limit --memory=768m --memory-swap=768m -v "$PWD/containers":/app eclipse-temurin:25 java -XX:MaxRAMPercentage=50 /app/OverLimit.java 200

Done: 250 MB heap, 200 MB direct, waiting 30 s
Enter fullscreen mode Exit fullscreen mode

docker inspect gives exit=0 OOMKilled=false. The heap can grow to
384 MB (50 % of 768), and 384 MB stay for the direct memory and the
rest of the JVM. The more memory an application uses outside the heap
(threads, direct memory, many classes), the lower the percentage must
be.

In Kubernetes

In a pod, each container has requests and limits. The Kubernetes
documentation says that the request is used to choose the node for the
pod; the limit is enforced by the kernel. For memory, a container that
goes over its limit can be killed (OOM kill). For CPU, it is slowed
down.

So the memory limit is the one the JVM reads. Without limits.memory,
the JVM sees the memory of the node: in a pod with only
requests.memory: 1Gi, the max heap is 1984 MiB on my machine (25 % of
the 7.7 GB of the Docker VM), much more than the request.

For the example, the Java file goes into the pod with a ConfigMap: no
image to build. The manifest
pod-before.yaml
is the killed case:

      command: ["java", "-XX:MaxRAMPercentage=75", "/app/OverLimit.java"]
      resources:
        requests:
          memory: "512Mi"
          cpu: "1"
        limits:
          memory: "512Mi"
Enter fullscreen mode Exit fullscreen mode

The kubectl commands used:

Command What it does
kubectl create configmap over-limit-code --from-file=containers/OverLimit.java Stores the Java file in the cluster; the pod mounts it in /app.
kubectl apply -f containers/pod-before.yaml Creates the pod described in the file.
kubectl get pod over-limit Shows the status of the pod.
kubectl describe pod over-limit Shows the details: state, reason, exit code, limits.
kubectl logs over-limit Shows what the program printed.
kubectl delete pod over-limit Deletes the pod.
kubectl create configmap over-limit-code --from-file=containers/OverLimit.java
kubectl apply -f containers/pod-before.yaml
kubectl get pod over-limit

NAME         READY   STATUS      RESTARTS   AGE
over-limit   0/1     OOMKilled   0          7s

kubectl describe pod over-limit

    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
Enter fullscreen mode Exit fullscreen mode

Here, RESTARTS is 0: the pod of the example has restartPolicy: Never,
so that its state stays there to read. With a Deployment, Kubernetes
starts the container again, and RESTARTS goes up after each kill: this
is the pod that restarts at the start of this post.

pod-after.yaml
uses the size that fits: limits.memory: 768Mi,
-XX:MaxRAMPercentage=50. After kubectl delete pod over-limit and
kubectl apply -f containers/pod-after.yaml, the pod ends as
Completed, exit code 0.

What has changed

What you still read What is true today Since
-XX:+UseCGroupMemoryLimitForHeap so that Java sees the limit Nothing to add: the JVM reads the limits itself (UseContainerSupport); the old option is refused JDK 10, and Java 8u191; option removed in Java 11
-XX:MaxRAMFraction=4 -XX:MaxRAMPercentage=25 (and InitialRAMPercentage, MinRAMPercentage) JDK 10
"cpu.shares" changes the number of CPUs Java sees "cpu.shares" is ignored Java 19 (Temurin 17.0.20 too, in my tests)

Tested with Docker Desktop 29.8 (Linux VM, aarch64, cgroup v2), its
Kubernetes (v1.36), and the eclipse-temurin images 17.0.20, 21.0.12
and 25.0.4.

Sources: the documentation of the
java
command (JDK 25),
Oracle's GC guide, part "Ergonomics",
the release notes of
JDK 10,
Java 8u191
and JDK 19,
and the Kubernetes documentation
(Resource Management for Pods and Containers,
Assign Memory Resources to Containers and Pods).

Further reading

Top comments (0)