A service started getting OOMKilled in production. Not under load — randomly, a few times a day, restarting cleanly, so alerting mostly stayed quiet. The dev team swore the code hadn't changed. They were right, and that made it worse, because a problem with no code change feels like a ghost.
Here's what was actually happening. The container had a memory limit of 512Mi. The JVM inside it saw the host's total memory — 32 gigabytes — because older JVMs don't read cgroup limits by default. So it happily sized its heap for a machine that had 64x more memory than the pod was allowed to use. Everything was fine until the heap grew past 512Mi, at which point the kernel's OOM killer stepped in and reaped the process. Kubernetes restarted it, the heap started small again, and the cycle repeated.
The fix was one line: tell the JVM to respect container limits. But the lesson was bigger than one flag.
A container is not a small machine. It's a process with a cgroup accounting boundary wrapped around it, and a lot of software running inside was written assuming it owns the whole box. Runtimes that autosize thread pools to CPU count, garbage collectors that size to total RAM, database clients that pick connection-pool defaults from nproc — all of them read the host, not the limit, unless you configure them or run a version that knows better.
Now, whenever I containerize something, I check three things before it ships. Does the runtime respect the memory limit? Does it respect the CPU quota? And are my requests and limits set from real observed usage, not a number someone typed once? I'd rather set a limit that's a little generous and tighten it with data than guess low and spend Fridays reading crash loops.
Also: set memory requests equal to limits for anything that matters. Bursting past your request into node memory that isn't there is how you turn one sick pod into a cascading node eviction.
The container told the truth. We just weren't listening to the runtime inside it.
– Sergey Shinder
Top comments (0)