When a container shows OOMKilled while the node dashboard still has gigabytes of free memory, engineers often assume Kubernetes made a mistake.
Usually it did not.
The container exceeded the memory boundary assigned to its cgroup.
Node memory and container memory are related, but they are not the same limit.
That distinction is fundamental to Kubernetes resource management.
Memory limits are enforced below Kubernetes
Kubernetes accepts memory requests and limits in the Pod specification.
The kubelet and container runtime translate those settings into operating system resource controls.
On Linux, cgroups are responsible for enforcing those boundaries.
If a container exceeds its memory limit, the kernel can terminate it even while the node itself still has free memory.
That behavior is intentional.
Without container-level enforcement, one workload could consume memory intended for other Pods.
I confirm the termination reason
Before adjusting anything, I inspect the previous container state and events.
I want to confirm OOMKilled rather than assuming every exit under load is an out-of-memory event.
Then I compare the container's configured limit with observed memory usage.
I also check whether the workload experienced a sudden spike rather than a gradual leak.
AceCloud's C*rashLoopBackOff troubleshooting guide* is useful here because repeated OOM kills often eventually surface as restart loops rather than as an obvious memory-sizing incident.
Requests and limits solve different problems
A memory request influences scheduling.
A memory limit constrains runtime consumption.
If a container requests 512 MiB and has a 1 GiB limit, the scheduler reserves capacity based on the request while the container can grow beyond it until enforcement becomes relevant.
That gap can be perfectly reasonable.
It can also be dangerous when every workload regularly consumes far more than requested.
I therefore right-size both values based on observed behavior.
I look for workload memory patterns
Not every OOM event needs a larger limit.
I want to know why memory increased.
Possible causes include
- Memory leaks
- Large request payloads
- Unbounded caches
- Excessive concurrency
- JVM heap configuration
- Python worker counts
- Model loading
- Large in-memory buffers
- Temporary data processing
Increasing the limit can postpone a leak without fixing it.
If memory scales with traffic, I may need different concurrency limits or horizontal scaling.
Sidecars deserve attention
A Pod can contain several containers.
The application may look healthy while a sidecar is being killed.
Logging agents, proxies, and service mesh components consume memory too.
I inspect resource settings per container.
I do not assume the main application process is always responsible.
Node-level OOM is a different incident
The reverse situation also matters.
A node under severe memory pressure can experience broader eviction or OOM behavior even when an individual container has not crossed the limit I expected.
That investigation requires node memory, requests, limits, kubelet eviction thresholds, kernel logs, and workload priority.
I keep container OOM and node memory pressure conceptually separate until evidence connects them.
cgroup v2 matters increasingly
Modern Kubernetes is standardizing around cgroup v2.
That brings a more consistent resource management model and supports newer memory management capabilities.
For teams operating their own node lifecycle, Kubernetes upgrades should therefore include cgroup compatibility checks.
For teams that would rather reduce that infrastructure maintenance surface, AceCloud Kubernetes provides managed control plane and worker lifecycle capabilities while still allowing application teams to configure Kubernetes resource requests and limits.
I do not remove limits blindly
Some teams respond to OOMKilled by deleting every memory limit.
I think that trades one problem for another.
Limits provide isolation.
Without them, a badly behaving workload can threaten neighboring Pods and the node.
I prefer understanding normal usage, peak usage, application concurrency, and failure behavior before setting the boundary.
For certain workloads a generous limit makes sense.
For others, memory growth should trigger scaling or backpressure rather than unlimited allocation.
My mental model
I think of Kubernetes memory at three levels.
The application consumes memory.
The container has a cgroup boundary.
The node has finite physical or virtual memory shared across workloads.
An OOM at one level does not require exhaustion at another.
Once that model is clear, the apparently impossible incident makes sense.
The node can have free RAM.
The container can still exceed the amount it was allowed to use.
That is not contradictory.
That is resource isolation working.
Top comments (0)