OOMKill in Kubernetes and Linux (Exit Code 137)

When a container(process) is terminated due to an Out Of Memory (OOM) manager condition, Kubernetes marks it as OOMKilled. In this case , the Linux kernel’s OOM Killer terminates the process and the container exits with exit code 137. This exit code is an important troubleshooting signal in Kubernetes.

Why OOMKill Happens

Linux uses an over-commitment policy for memory management. Applications can request more memory than is physically available. When the actual demand exceeds system capacity, the kernel must decide which process to kill in order to reclaim memory. How OOM Kill works ?

• The OOM Killer selects and terminates processes based on their OOM score.
• In Kubernetes, this usually happens when a container exceeds its memory limit.
• Kubernetes then records the container’s state as OOMKilled, and logs exit code 137. OOM Killer Sends What Signal to the Process ?

Exit codes 1 - 2, 126 - 165, and 255 have special meanings, and should therefore be avoided for user-specified exit parameters. Ending a script with exit 127 would certainly cause confusion when troubleshooting (is the error code a "command not found" or a user-defined one?).

Resources

• Exit Status
• Exit Codes
• Out Of Memory Management
• Linux Memory Overcommitment and the OOM Killer
• Kill signal linux kernel
• OOM Dashboard
• Kubernetes OOMKilled Error
• Virtual Memory in Linux
• Go Memory Management
• Virtual Memory
• Memory Segments in C
• Tasks
  • Kill Process Group on OOM Event
  • Make the Container Exit When One of Its Processes Runs Out of Memory
  • Make a Kubernetes Pod Survive an OOM Event Without Restarting