DEV Community

Parsa
Parsa

Posted on

Exit code 137 doesn't mean it ran out of memory

137 just means the process got SIGKILL. the OOM killer sends that, but so does a failed liveness probe, a node draining, or someone running kubectl delete at the wrong moment. so before chasing a leak, run kubectl describe pod and look at the last state. OOMKilled is spelled out there when it's real.

when it really is memory, the pattern is almost always the same. something grows per task and never shrinks. a dict keyed by job id that nobody prunes, an lru cache with no maxsize, an event log kept in memory "for debugging". each one looks harmless in review, the tests run for seconds so they never see it grow, and it shows up days after the deploy when the pod has been up long enough.

Celery had a textbook case in events/state.py. memory growth that was first reported in 2018, and the maintainers shipped a bound for it in 2026. we work on Tomosu, and we ran a blind scan on the file as it was before the fix, no issue, no hint of what to look for. it flagged the unbounded growth and recommended a bound that matches the one the maintainers shipped.

that's the part of what we build that i find most useful, catching the slow ones from the code instead of from a pod restart graph. it doesn't replace watching memory in prod, and the case study is honest about the setup and its limits.

https://tomosu.ai/case-studies/celery-memory-leak.html?utm_source=devto&utm_medium=post&utm_campaign=oom-137

Top comments (0)