Docker Exit Code 137: Prove OOMKilled Before Raising Memory

Adapted from Docker Exit Code 137: Prove OOMKilled Before Raising Memory, published on TechFuelHQ.

A container has stopped with exit code 137. Before restarting it or increasing its memory limit, capture its identity, state and failure window. Those records help distinguish a container memory limit from host pressure, a stop timeout, or an external kill.

The number alone cannot tell you which happened. Bash reports fatal signals as 128 + the signal number, and Linux assigns SIGKILL number 9. But Docker returns the container command's exit status, so the command can return 137 itself. Docker also documents two non-memory routes to SIGKILL: docker kill sends it by default, while docker stop escalates to it after the timeout.

Capture one container and one time window

Replace these example values. Keep an explicit timezone: Docker events otherwise interprets timestamps in the client's local timezone.

CONTAINER='replace-with-name-or-id'
SINCE='2026-10-03T18:20:00-05:00'
UNTIL='2026-10-03T18:30:00-05:00'

First, read the stopped container's state and configured limits with Docker inspect. Save the full container ID with the output.

docker inspect --type=container \
  --format 'id={{.Id}} name={{.Name}} image={{.Config.Image}} exit={{.State.ExitCode}} oom_killed={{.State.OOMKilled}} started={{.State.StartedAt}} finished={{.State.FinishedAt}} error={{json .State.Error}} memory={{.HostConfig.Memory}} reservation={{.HostConfig.MemoryReservation}} memory_swap={{.HostConfig.MemorySwap}} oom_kill_disable={{json .HostConfig.OomKillDisable}} restart_count={{.RestartCount}}' \
  "$CONTAINER"

Next, collect the event sequence. Docker exposes separate oom, kill, die, stop and restart events. Its retained history covers only the last 256 events. Collect it promptly.

docker events \
  --since "$SINCE" \
  --until "$UNTIL" \
  --filter type=container \
  --filter container="$CONTAINER" \
  --format '{{json .}}'

docker logs --timestamps --since "$SINCE" --until "$UNTIL" "$CONTAINER"

Application logs can preserve the last completed operation or a shutdown request. A missing final message does not establish a kernel kill.

On a systemd host, collect kernel and daemon logs too. The UTC timestamps below cover the same example interval. These queries select the current boot; journalctl documents time and boot filtering.

JOURNAL_SINCE='2026-10-03 23:20:00 UTC'
JOURNAL_UNTIL='2026-10-03 23:30:00 UTC'

journalctl -k --boot=0 --since "$JOURNAL_SINCE" --until "$JOURNAL_UNTIL" --no-pager \
  | grep -Ei 'out of memory|oom-kill|killed process|memory cgroup'

journalctl -u docker.service --boot=0 --since "$JOURNAL_SINCE" --until "$JOURNAL_UNTIL" --no-pager

If the incident preceded this boot, run journalctl --list-boots. Choose the boot containing the incident and replace --boot=0 in both commands with --boot=THE_32_CHARACTER_BOOT_ID. On other hosts, use their actual kernel and daemon log mechanism.

Identify the memory boundary

State.OOMKilled=true means Docker recorded an OOM-killed state. It does not establish whether the container's own limit, a parent cgroup, or host pressure was responsible. Correlate it with the event timeline and kernel record.

A false flag is not an all-clear. Without matching evidence, keep the cause unresolved.

The inspect field memory is the container's hard limit in bytes. Zero means Docker configured no per-container hard limit; it says nothing about a parent boundary or the host's capacity. Docker distinguishes hard limits from soft reservations and warns about host OOM when memory is exhausted.

Where the host retains the container's cgroup, inspect that too. Locate the path through the host's runtime configuration. Do not borrow another machine's path.

For cgroup v2:

CGROUP_PATH='/replace/with/verified/container-cgroup-path'
cat "$CGROUP_PATH/memory.max"
cat "$CGROUP_PATH/memory.high"
cat "$CGROUP_PATH/memory.events"
cat "$CGROUP_PATH/memory.events.local"

The kernel's memory-controller documentation gives these files distinct meanings:

• memory.max is the hard limit.
• memory.high applies reclaim pressure without itself invoking the OOM killer.
• An increase in the exact cgroup's local oom counter indicates allocation was about to fail at that cgroup's limit.
• An oom_kill increase counts processes in the cgroup killed by an OOM killer. It does not identify the limiting boundary or prove the victim was the container's PID 1.
• memory.events includes descendant events; memory.events.local stays local to that cgroup.

Record unavailable counters as unavailable. A replacement container's counters cannot fill the gap. Hosts using cgroup v1 need its different memory interfaces, covered in the original guide.

Choose a change the evidence supports

A matching Docker oom event and a nonzero limit still need corroboration. A kernel record naming the limiting cgroup, or a same-window increase in that exact cgroup's local oom counter, helps establish the boundary. Then compare workload peaks with the limit and investigate the application's memory configuration or a leak.

For host pressure, inventory the other workloads and host reserve. Raising one container's limit can move pressure onto the host.

For a stop-timeout escalation, investigate shutdown handling. Docker notes that shell-form ENTRYPOINT and CMD run below /bin/sh -c, which does not pass signals to the executable. Investigate signal handling or the grace period before changing memory.

For an external kill, trace the actor through daemon, service or scheduler records. If no branch is proven, retain the evidence and improve capture for the next occurrence.

Keep a short incident record:

Container ID and image:
Incident timestamp and timezone:
ExitCode / OOMKilled / RestartCount:
Configured container and proven parent limits:
Same-window Docker events:
Kernel victim and limiting cgroup, if identified:
cgroup version, exact path and counter changes:
Unavailable evidence:
Current classification, including unknown:
One change to test / result / timestamp:

The commands above collect evidence. They do not restart the container or change its limits. A restart may restore service while the cause remains unknown.

Disclosure: AI agents adapted this article from TechFuelHQ's sourced guide. The original documents a Docker and Linux documentation review on October 3, 2026; no Docker-host OOM experiment was performed for it, and this adaptation claims no new testing.