Why the Linux OOM killer killed your container when memory looked fine

Your phone buzzes at 3 AM. A pod restarted. kubectl describe pod says the last state was OOMKilled.

So you check the node. It has gigabytes free. Nothing is close to full. You stare at the two facts side by side: killed for lack of memory, and plenty of memory. One of them has to be lying.

Neither is. There are two different out-of-memory killers on that machine, and only one of them cares about the node's free memory.

The two killers

Every container on your node lives inside a cgroup, short for control group. A cgroup is the kernel's way of putting a fence around a group of processes and saying: this group gets this much CPU, this much memory, no more.

When you set resources.limits.memory: 512Mi on a container, you are setting the fence height for its cgroup. When the processes inside cross that line, the kernel steps in and kills something inside the cgroup. That is a cgroup OOM kill. The node can have 30 GB free. It does not matter. The fence is the fence.

The other killer is the system OOM killer. That one fires when the whole node is genuinely out of memory. It picks a victim from anywhere on the machine.

So the first question to answer after any OOM kill is: which one fired? The kernel tells you, in the machine's log:

dmesg | grep -i "out of memory"

You will see a line like this:

Out of memory: Killed process 18421 (java) total-vm:2147484kB ...

Right above it, the log names the cgroup the victim belonged to. If it names your container's cgroup, the container hit its own limit, not the node's. That is the 3 AM answer nine times out of ten.

How the killer picks its victim

Once the kernel decides a kill has to happen, it does not kill at random, and it does not simply kill the biggest process. It scores every candidate and kills the highest scorer.

The score lives in a file you can read right now:

cat /proc/self/oom_score

Mine says 666. Yours will say something else. The number is roughly 0 to 1000 and it is mostly this: how much memory the process uses, as a share of the memory it is allowed to use.

That "allowed to use" part is the whole trick. Inside a container, your allowance is the cgroup limit, not the node's memory. A process using 400 MB inside a 512 MB container scores far higher than a process using 2 GB on a 64 GB node, because the first one is at 78 percent of its allowance.

The kernel adds small adjustments on top. Root-owned processes get a small discount. And then there is the knob.

The one knob that changes the score

Every process carries a second file:

cat /proc/self/oom_score_adj

This one says 0 by default, and you (or your orchestrator) can set it anywhere from -1000 to +1000. The kernel simply adds it to the score. Set it to -1000 and the process is unkillable by the OOM killer. Set it to +1000 and it is first in line.

This is the mechanism behind one of the most confusing behaviors in Kubernetes. Kubernetes sets oom_score_adj for you based on the pod's quality of service class, a label Kubernetes derives from your resource settings:

• Guaranteed (every container sets equal memory requests and limits): the kernel is told -998. Almost never killed.
• BestEffort (no requests or limits at all): the kernel is told 1000. First in line, always.
• Burstable (anything in between): somewhere from 2 to 999, scaled from how big your memory request is compared to the node.

So two identical processes can have wildly different survival odds, purely because of the QoS class on the pod spec. Your BestEffort batch job is not unlucky. It volunteered.

What to actually check

When a container dies with OOMKilled and the node looks healthy:

1. Read the kernel log. dmesg | grep -i "out of memory" names the cgroup. If it is your container's cgroup, the limit was the ceiling, not the node.
2. Compare limit to reality. kubectl top pod shows actual usage. If usage was brushing the limit, the limit was too small, or the app leaks.
3. Check the QoS class. kubectl describe pod shows it. A BestEffort pod on a busy node is living on borrowed time.
4. If it is Java, check the heap. The classic trap: the JVM sizes its heap from the node's memory unless told otherwise, then the container's cgroup fence kills it. Size the heap to fit inside the container limit, with room for the JVM's own overhead.

None of this requires new tools. The score, the knob, and the log are all plain files and one shell command. The killer is not mysterious once you know it scores against the fence, not the field.

What is the strangest OOM kill you have ever chased? I want to hear the one where the numbers made no sense until they did.