DEV Community

Florient DOGBE
Florient DOGBE

Posted on

Why Your Pods Get OOMKilled While Memory Looks Free: Decoding cgroups v2

#kubernetes #openshift #cgroups #sre

You get paged. A pod restarts with Last State: Terminated, Reason: OOMKilled, Exit Code: 137. You exec into the replacement container, run top, and see 12 GB free on the node. The app team swears the heap never went above 400 MB. The limit is 1 Gi. Nothing adds up.

This is one of the most common "unexplained" incidents on Kubernetes and OpenShift clusters, and it almost always comes down to the same thing: the memory your process thinks it uses and the memory the kernel charges to your cgroup are two different numbers. Since Kubernetes 1.25 (and RHEL 9 / OpenShift 4.12+ by default), that accounting lives in cgroups v2, with its own files, its own semantics, and its own traps.

Let's take it apart.

The illusion: top inside a container lies

top, free, and most runtimes read /proc/meminfo. In a container, /proc is the host's procfs (there is no memory namespace in Linux). So free -m inside a pod reports the node's 64 GB of RAM, not your 1 Gi limit. Unless you ship lxcfs or run with a modern runtime that masks these files, every tool based on /proc/meminfo is structurally wrong inside a container.

The kernel does not care about /proc/meminfo when it kills you. It cares about one counter: memory.current in your container's cgroup, compared against memory.max.

# Inside a pod, with cgroup v2 mounted (systemd-managed nodes, OpenShift 4.12+)
cat /sys/fs/cgroup/memory.max      # the limit the kernel enforces
cat /sys/fs/cgroup/memory.current  # what the kernel charges you
cat /sys/fs/cgroup/memory.peak     # high-water mark (kernel >= 5.19)
cat /sys/fs/cgroup/memory.events   # low/high/max/oom/oom_kill counters
Enter fullscreen mode Exit fullscreen mode

If memory.max says max instead of a number, you have no limit and your pod is a node-level OOM candidate instead — a different and worse failure mode.

cgroups v2 memory accounting in five minutes

memory.current is not RSS

memory.current is the sum of every page charged to the cgroup, which includes far more than your process' anonymous memory:

  • anon — heap, stacks, anonymous mmaps. This is roughly what you call "RSS".
  • file — page cache for files your container read or wrote.
  • shmem — tmpfs, /dev/shm, emptyDir: {medium: Memory}.
  • kernel_stack, pagetables, percpu, slab — kernel structures allocated on your behalf.
  • sock — TCP/UDP socket buffers. A proxy holding 20k connections pays for this.

Dump it:

# Node-side, find the container cgroup (CRI-O naming; containerd uses cri-containerd-*.scope)
POD_UID=$(oc get pod myapp-0 -o jsonpath='{.metadata.uid}')
find /sys/fs/cgroup/kubepods.slice -type d -name "*${POD_UID//-/_}*"

cat /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/\
kubepods-burstable-pod${POD_UID//-/_}.slice/crio-<cid>.scope/memory.stat
Enter fullscreen mode Exit fullscreen mode

Typical output on a pod that "uses 400 MB":

anon 419430400
file 1810432000
kernel 58720256
slab_unreclaimable 12582912
pagetables 7340032
sock 2097152
shmem 268435456
inactive_file 1476395008
active_file 334036992
inactive_anon 0
active_anon 419430400
file_dirty 83886080
file_writeback 12582912
Enter fullscreen mode Exit fullscreen mode

memory.current here is ~2.3 GB even though the process' own data is 400 MB. Most of it is page cache from reading files — and that page cache is charged to your cgroup, because the kernel charges the first cgroup that touches a page.

What actually triggers the OOM killer

The important part: page cache is reclaimable. When memory.current approaches memory.max, the kernel does direct reclaim — it drops clean page cache instead of killing you. The OOM killer fires only when reclaim fails to free enough, i.e. when the unreclaimable fraction alone exceeds the limit.

So the real OOM equation is roughly:

anon + unreclaimable_slab + pagetables + sock + shmem(no swap) + dirty_pages_pending_writeback > memory.max
Enter fullscreen mode Exit fullscreen mode

Three consequences engineers consistently miss:

  1. Dirty pages are not instantly reclaimable. A container writing 500 MB/s to a slow PVC accumulates file_dirty + file_writeback. Reclaim must wait for writeback. Under pressure, that stall is enough to trigger an OOM kill even though the data is "just cache".
  2. tmpfs/shmem cannot be reclaimed without swap. Kubernetes nodes usually run swapoff. Shmem pages sit on the file LRU lists, so they look like cache in memory.stat — but the kernel cannot evict them. This is the single most under-diagnosed OOMKill cause.
  3. Kernel memory counts. cgroups v2 accounts kernel memory unconditionally (no kmem switch like v1). Page tables for a process with huge mmaps, or socket buffers for a busy ingress, can be hundreds of MB.

Working set: what kubelet, kubectl top and VPA actually measure

The kubelet does not report memory.current. It computes:

working_set = memory.current - inactive_file   (floored at 0)
Enter fullscreen mode Exit fullscreen mode

That value is container_memory_working_set_bytes in cAdvisor, what kubectl top pod and oc adm top pod display, what the eviction manager compares against thresholds, and what VPA uses as its input signal.

Now the gap becomes obvious:

kubectl top pod myapp-0 --containers
# NAME      CPU(cores)   MEMORY(bytes)
# app       120m         820Mi         <- working set

# but on the node:
cat .../memory.current   # 2415919104  (2.3Gi)
Enter fullscreen mode Exit fullscreen mode

Working set subtracts inactive_file only. active_file stays in the working set, so a read-heavy container looks like it's leaking. And because shmem pages land on file LRUs, part of your unreclaimable memory may be subtracted out — meaning working set can simultaneously over-report (active page cache) and under-report (shmem) the memory that actually matters for OOM.

This is why autoscalers and dashboards disagree with reality. VPA will happily recommend a limit based on a working set inflated by page cache, or a limit too low for a pod whose shmem usage is invisible.

Debugging an unexplained OOMKill, step by step

Step 1: confirm who killed what 

oc get pod myapp-0 -o jsonpath='{.status.containerStatuses[*].lastState.terminated}' | jq
# {"exitCode":137,"reason":"OOMKilled","startedAt":"...","finishedAt":"..."}

oc get events --field-selector reason=OOMKilling -A
Enter fullscreen mode Exit fullscreen mode

Exit code 137 = 128 + SIGKILL. But OOMKilled on a container can mean two different things: a cgroup OOM (your limit) or a node OOM (the kernel killed you to save the node). Distinguish them from the node:

oc debug node/worker-3 -- chroot /host dmesg -T | grep -i "out of memory"
# "Memory cgroup out of memory: Killed process 12345 (java) ..."  -> cgroup limit
# "Out of memory: Killed process ..." (no "Memory cgroup")        -> node pressure
Enter fullscreen mode Exit fullscreen mode

Step 2: read memory.events 

cat /sys/fs/cgroup/.../memory.events
# low 0
# high 0
# max 148239        <- number of times allocation hit memory.max and triggered reclaim
# oom 12
# oom_kill 12
Enter fullscreen mode Exit fullscreen mode

A non-zero and growing max counter with oom_kill 0 means you are thrashing in reclaim: the workload is surviving, but every allocation pays a direct-reclaim penalty. Latency degrades long before the kill. Alert on this, not just on restarts.

Step 3: isolate the unreclaimable part

Trigger a reclaim and compare, on a canary pod only:

# inside the container cgroup (node-side, root): ask the kernel to reclaim 256M
echo "256M" > /sys/fs/cgroup/.../memory.reclaim   # kernel >= 5.19
cat /sys/fs/cgroup/.../memory.current
Enter fullscreen mode Exit fullscreen mode

Whatever will not shrink is your true footprint. Compare anon + shmem + slab_unreclaimable + pagetables + sock against memory.max. If that sum is above ~80 % of the limit in steady state, you are one burst away from an OOMKill.

The usual suspects

Memory-backed emptyDir and /dev/shm. Every byte is charged to the pod cgroup and is unreclaimable without swap. A log buffer, a Spark shuffle dir, or PostgreSQL's shared memory will silently eat the limit.

apiVersion: v1
kind: Pod
metadata:
  name: shm-aware
spec:
  containers:
    - name: app
      image: registry.access.redhat.com/ubi9/ubi-minimal:latest
      command: ["sleep", "infinity"]
      resources:
        requests:
          memory: 1Gi
          cpu: 250m
        limits:
          memory: 1536Mi   # app budget + 256Mi tmpfs + 256Mi headroom
          cpu: "2"
      volumeMounts:
        - name: cache
          mountPath: /var/cache/app
        - name: dshm
          mountPath: /dev/shm
  volumes:
    - name: cache
      emptyDir:
        medium: Memory
        sizeLimit: 256Mi   # always set this
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 256Mi
Enter fullscreen mode Exit fullscreen mode

JVM and glibc. -XX:MaxRAMPercentage reads memory.max correctly on cgroups v2, but it bounds only the heap. Metaspace, code cache, thread stacks, direct ByteBuffers and glibc's per-thread malloc arenas live outside it. A 64-core node gives glibc up to 8 × ncpu arenas; cap it with MALLOC_ARENA_MAX=2 and budget 25–30 % above max heap.

Network-heavy sidecars. Envoy/HAProxy with large buffers and tens of thousands of connections: watch sock in memory.stat.

Calibrating requests and limits

Stop averaging. Use percentiles of the working set, and add a term for the unreclaimable extras:

# P95 working set over 7 days, per container
quantile_over_time(0.95,
  container_memory_working_set_bytes{namespace="prod", container!="", container!="POD"}[7d]
)

# How close are we to the limit?
max by (namespace, pod, container) (
  container_memory_working_set_bytes{container!=""}
  / on(namespace, pod, container)
  kube_pod_container_resource_limits{resource="memory"}
)
Enter fullscreen mode Exit fullscreen mode

A workable rule: requests.memory = P95(working set) × 1.1, limits.memory = P99(working set) + tmpfs sizeLimit + 20 % headroom. Never set limits more than ~2× requests unless you accept node overcommit.

Alert before the kill:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: memory-pressure-rules
  namespace: prod
spec:
  groups:
    - name: cgroup-memory
      rules:
        - alert: ContainerNearMemoryLimit
          expr: |
            max by (namespace, pod, container) (
              container_memory_working_set_bytes{container!=""}
              / on(namespace, pod, container)
                kube_pod_container_resource_limits{resource="memory"}
            ) > 0.85
          for: 10m
          labels: {severity: warning}
          annotations:
            summary: "{{ $labels.pod }}/{{ $labels.container }} above 85% of its memory limit"
        - alert: ContainerOOMKilled
          expr: |
            increase(kube_pod_container_status_restarts_total[15m]) > 0
            and on (namespace, pod, container)
            kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
          labels: {severity: critical}
Enter fullscreen mode Exit fullscreen mode

Let VPA recommend, not decide

VPA consumes the same working-set metric, so treat its output as a starting point and keep updateMode: "Off" until you have validated it against memory.stat.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myapp-vpa
  namespace: prod
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  updatePolicy:
    updateMode: "Off"          # recommendations only; flip to Auto after validation
  resourcePolicy:
    containerPolicies:
      - containerName: app
        controlledResources: ["memory"]
        controlledValues: RequestsOnly   # keep your hand-tuned limits
        minAllowed:
          memory: 256Mi
        maxAllowed:
          memory: 4Gi
      - containerName: istio-proxy
        mode: "Off"
Enter fullscreen mode Exit fullscreen mode 
oc get vpa myapp-vpa -o jsonpath='{.status.recommendation.containerRecommendations}' | jq
Enter fullscreen mode Exit fullscreen mode

Note the conflict: VPA on memory and HPA on memory must never target the same workload. And if you use VPA with controlledValues: RequestsAndLimits, it scales limits proportionally to requests — which breaks the "limit must cover tmpfs + off-heap" logic above.

memory.high and MemoryQoS

cgroups v2 adds memory.high, a throttling threshold: crossing it forces reclaim and slows the cgroup instead of killing it. Kubernetes exposes this through the alpha MemoryQoS feature gate, which sets memory.high from requests and memoryThrottlingFactor (default 0.9). It converts hard kills into graceful degradation — but verify the gate's status in your Kubernetes/OpenShift version before relying on it in production.

Security best practices

  • Always set memory limits, and enforce them. A container without a limit can drive the node into global OOM, where the kernel picks victims by oom_score — potentially kubelet or unrelated tenants. Enforce defaults with a LimitRange and a cap with ResourceQuota: 
apiVersion: v1
kind: LimitRange
metadata:
  name: default-mem
  namespace: prod
spec:
  limits:
    - type: Container
      default:          {memory: 512Mi, cpu: 500m}
      defaultRequest:   {memory: 256Mi, cpu: 100m}
      max:              {memory: 8Gi}
      maxLimitRequestRatio: {memory: "2"}
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-quota
  namespace: prod
spec:
  hard:
    limits.memory: 64Gi
    requests.memory: 32Gi
Enter fullscreen mode Exit fullscreen mode
  • Always set sizeLimit on memory-backed emptyDir. Without it, a compromised or buggy container can fill tmpfs up to half the node's RAM, turning a write loop into a node-wide denial of service. The same applies to /dev/shm.
  • Treat repeated OOMKills as a security signal, not just a capacity one. Unbounded request-body buffering, decompression bombs, or regex/alloc amplification all surface first as memory kills. Correlate kube_pod_container_status_last_terminated_reason with ingress traffic spikes.
  • Do not grant cgroup access via hostPath. Reading /sys/fs/cgroup from the host should go through an ephemeral oc debug node/<node> session (cluster-admin, audited), not a long-lived privileged pod. On OpenShift, keep workloads under the restricted-v2 SCC; diagnostics belong in oc debug or an ephemeral container.
  • Avoid hostPID for "just reading dmesg". It exposes every host process' /proc to the container. Use node debug pods instead, and keep node kernel logs in your log pipeline so nobody needs shell access during an incident.
  • Reserve node resources. Configure system-reserved and kube-reserved (via KubeletConfig on OpenShift) and enforce hard eviction thresholds, so pressure evicts pods predictably instead of letting the kernel OOM-kill critical system daemons.
  • Guaranteed QoS for anything in the control path. Equal requests and limits put the pod in the Guaranteed class with oom_score_adj = -997, making it the last victim under node pressure.

Key takeaways

  • top and free inside a container read the host's /proc/meminfo. They are irrelevant to OOM decisions.
  • The kernel kills on memory.current vs memory.max, and memory.current includes page cache, tmpfs, socket buffers, page tables and slab — not just your heap.
  • The OOM killer fires only when reclaim fails. Your real budget is anon + shmem + unreclaimable slab + pagetables + sock + dirty pages.
  • kubectl top and VPA show working_set = memory.current - inactive_file: it over-counts active page cache and under-counts shmem. Never size limits from that number alone.
  • Debug with memory.stat, memory.events and memory.peak on the node; alert on the max counter in memory.events to catch reclaim thrashing before the kill.
  • Size limits as P99(working set) + tmpfs sizeLimit + 20 % headroom, cap maxLimitRequestRatio, and always set sizeLimit on memory-backed volumes.

The memory on your node really is free. It is just not free for that cgroup.

Top comments (2)

Collapse
 
dhruonweb profile image
Dhruv Malaviya •

The sock line surprises people most and deserves emphasis: a proxy holding tens of thousands of connections can be charged hundreds of megabytes in socket buffers alone, with the heap never moving. Nothing in the app's own metrics shows it.

Two additions. memory.peak is the single most useful file here and most people don't know it exists , it answers "how close did we get" without needing to have been watching.

And the runtime side is usually where this originates. A JVM sizing its heap from /proc/meminfo reads the node's 64 GB, not the 1 Gi limit. Container-aware flags fixed that for the JVM; Go still needs GOMEMLIMIT set explicitly.

Collapse
 
florient_4262fab5463a691 profile image
Florient DOGBE •

Thanks Dhruv, great additions.

Agreed on sock: it never shows up in app metrics, so the OOMKill is often the first sign.

One caveat on memory.peak: it resets when the container restarts, since a new cgroup is created. Export it continuously if you need it for post-mortems.

On runtimes: cgroups v2 support only landed in JDK 15. I’d set GOMEMLIMIT to about 90% of the limit, since it’s a soft limit.