Docker exit code 137: telling the causes apart from the logs

A container has stopped with exit code 137. docker ps -a shows Exited (137), and docker inspect shows "ExitCode": 137.

This value does not tell you whether the container hit its memory limit, the host as a whole ran out of memory, or Docker or some other process stopped it.

I reproduced this on a test VM, changed how the container was stopped, and checked what each case left in the kernel log and the Docker daemon log.¹

Which log points to which cause

The OOM killer is the mechanism the kernel uses when memory runs short: it picks a process and kills it. For each way the container stopped, the kernel log and the Docker daemon log contained the following.²

What happened	Kernel log	Docker daemon log
The container hit its memory limit and the OOM killer killed it	Memory cgroup out of memory: Killed process	none
The host as a whole ran out of memory and the OOM killer killed it	Out of memory: Killed process	none
The container did not exit on docker stop, so Docker killed it	none	using the force
Someone ran docker kill	none	hasBeenManuallyStopped=true (if the container has a restart policy; no using the force)
Cannot be told from the logs	none	none

The last row covers a process on the host running kill -9, the app returning 137 itself, and a container without a restart policy being stopped with docker kill.

For a container with a restart policy, docker stop also logs hasBeenManuallyStopped=true. If a using the force line comes before it, it was docker stop. If not, it was docker kill.

When a container without a restart policy is stopped with docker kill, the Docker daemon log has no line that identifies it as docker kill. The only trace is the kill event in docker events, and the Docker daemon keeps only the most recent 256 events in memory, counted across all containers on the host.³

Read the kernel log with journalctl -k. A line starting with oom-kill: appears just before the Killed process line. The task_memcg= value on that line contains the container ID, which tells you which container it was.⁴ When the host as a whole runs out of memory, the container that gets killed is not necessarily the one that used up the memory. In the test, the OOM killer killed the app container, which held more memory than the container whose memory kept growing.

journalctl -k | grep -E 'Killed process|oom-kill:'

Read the Docker daemon log with journalctl -u docker. The string after container= is the container ID.

journalctl -u docker | grep -E 'using the force|hasBeenManuallyStopped=true'

Using the table in production

In production, there are situations where the values in docker inspect cannot be relied on. The logs in the table still tell the causes apart in those situations.

The container has restarted

If RestartCount in docker inspect is 1 or more, the container has restarted. When a container with a restart policy restarts, ExitCode in docker inspect goes back to 0 and OOMKilled back to false, and nothing there shows that it stopped with 137.

After a restart, FinishedAt in docker inspect still holds the time of the previous stop. If the kernel log has a Killed process line at about the same time, the previous stop was a kill by the OOM killer. FinishedAt is in UTC, so use journalctl -k --utc to line up the timestamps.

OOMKilled is false

State in docker inspect has a value called OOMKilled, which Docker sets to true when a process in the container is killed by the OOM killer. But it sometimes stays false even though the OOM killer did the kill. Decide whether the OOM killer killed it from the Killed process line in the kernel log (journalctl -k).

What exit code 137 indicates

PID 1 is the first process started in the container, the one given in ENTRYPOINT or CMD. When PID 1 exits, the container exits, and the exit code is determined by how PID 1 ended.

When PID 1 is ended by a signal, the exit code is 128 plus the signal number. SIGKILL is 9, so the code is 137.⁵

The exit code can be 137 even when PID 1 itself did not receive SIGKILL. If an init is started ahead of the app and runs the app as its child, as with docker run --init, then when the app is ended by SIGKILL, init exits with 137.⁶ With init in place, the logs for a kill at the container's memory limit and for kill -9 from a host process looked the same as without init.

SIGKILL is sent to a container's processes by one of the following:

• The kernel's OOM killer
• Docker (when the container does not exit on docker stop, and on docker kill)
• Another process on the host

Every kill in the table is this SIGKILL. The exit code does not tell you who sent it, so use the logs in the table to tell the senders apart.

What was not checked

I did not check mechanisms outside the kernel that watch for memory shortage and kill processes (systemd-oomd and earlyoom).

Appendix: each way of stopping and what it left

--memory 128m, app allocates 256MiB

• Kernel log: Memory cgroup out of memory
• Docker daemon log: none
• docker events: oom, die
• OOMKilled: true (false in 0 to 1 of 3 tries per full test run)

Another container with no limit uses up the host's memory

• Kernel log: Out of memory
• Docker daemon log: none
• docker events: oom, die (order varies between tries)
• OOMKilled: true (false in 0 to 2 of 10 tries per full test run)

docker stop -t 2

• Kernel log: none
• Docker daemon log: using the force
• docker events: kill (15), kill (9), stop, die
• OOMKilled: false

docker stop -t 2, with a restart policy

• Kernel log: none
• Docker daemon log: using the force, hasBeenManuallyStopped=true
• docker events: kill (15), kill (9), stop, die
• OOMKilled: false (does not restart)

docker kill

• Kernel log: none
• Docker daemon log: none
• docker events: kill (9), die
• OOMKilled: false

docker kill, with a restart policy

• Kernel log: none
• Docker daemon log: hasBeenManuallyStopped=true
• docker events: kill (9), die
• OOMKilled: false (does not restart)

kill -9 from the host to the app

• Kernel log: none
• Docker daemon log: none
• docker events: die
• OOMKilled: false

The app returns 137 itself

• Kernel log: none
• Docker daemon log: none
• docker events: die
• OOMKilled: false

--memory 128m, with a restart policy

• Kernel log: Memory cgroup out of memory
• Docker daemon log: none
• docker events: oom, die, start
• OOMKilled: false after the restart (ExitCode is 0)

---

1. I tested on a VM started with KVM inside a GitHub Actions runner (Ubuntu 24.04, kernel 6.8, 512MiB memory, no swap, cgroup v2), using Docker 29.1.3 from Ubuntu's docker.io package (cgroup driver: systemd). The scripts and results are in show-your-work and in the GitHub Actions run. ↩

2. The kernel log strings come from kernel 6.8: the call to oom_kill_process() and the line that prints Killed process. The Docker daemon log lines come from Docker 29.1.3: the line that logs using the force and the lines that log the warning containing hasBeenManuallyStopped=true. ↩

3. eventsLimit in the Docker 29.1.3 source. ↩

4. Kernel 6.8: where the oom-kill: line is printed and where task_memcg= is printed. ↩

5. runc's ExitStatus(). ↩

6. The init that --init starts is tini, which returns 128 plus the signal number when its child is ended by a signal (tini v0.19.0). ↩