Exit code 137: was it really out of memory? Find who sent the SIGKILL

docker ps -a says Exited (137) or Restarting (137), the app's logs stop mid-line with no error, and the box is a small 1–2 GB VPS running several services. It looks like "out of memory", and often it is. But 137 only tells you the main process received SIGKILL. The real question is who sent it. Everything below is read-only; run it only on servers you're authorized to access, and replace YOUR_CONTAINER with the real name or ID. Kernel log checks need sudo.

Why 137, and why the logs are silent

137 is 128 + 9: the process was ended by signal 9. SIGKILL can't be caught or delayed, so the app never gets to log a goodbye. A log that just stops fits a 137. It doesn't clear the app.

Several senders produce the same number:

Who sent SIGKILL	Where the evidence usually is	OOMKilled
Kernel OOM killer, container's own memory limit reached	Kernel: Memory cgroup out of memory: Killed process …; an oom event in docker events	Usually true
Kernel OOM killer, whole host out of memory	Kernel: Out of memory: Killed process …	true or false, depending on cgroup version and Docker/containerd versions
Userspace OOM daemon (systemd-oomd, earlyoom)	That daemon's journal; nothing from the kernel	false
docker kill / docker rm -f	kill event with signal=9	false
docker stop, compose down or a redeploy, app ignored SIGTERM for the whole grace period (10 s default)	kill signal=15, then signal=9 about 10 s later	false
A script, agent or person running kill -9 on the host	Only a die event; maybe that tool's own logs	false

Memory is the first suspect, not the answer.

1. Capture the exit state before a restart wipes it

docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Image}}'
docker inspect --format 'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} started={{.State.StartedAt}} finished={{.State.FinishedAt}} restarts={{.RestartCount}} policy={{.HostConfig.RestartPolicy.Name}}' YOUR_CONTAINER

• A restart resets the evidence. When the container starts again, Docker sets ExitCode back to 0 and OOMKilled back to false. With a restart policy you'll often see status=running exit=0 oom=false restarts=5, which looks clean but isn't. The previous exit then lives only in docker events (step 2) and the kernel log (step 4).
• exited / 137 / oom=true: the container's own limit is the lead hypothesis. Still confirm with steps 3 and 4.
• exited / 137 / oom=false: memory isn't ruled out, but check Docker's events first.
• oom=true on a running container, or with another exit code: in recent Docker versions the flag is set when the kernel kills any process in the container. A killed worker can leave the container up, or the main process may exit later with its own code.
• Memory trouble doesn't always end in 137. If a runtime hits its own heap limit first, it logs an error and exits another way: Node.js prints JavaScript heap out of memory and often exits 134; a JVM logs java.lang.OutOfMemoryError.
• finished is UTC (ends in Z). Convert before comparing with local host logs.

2. Ask Docker whether it sent the signal

docker events --since 24h --until "$(date +%s)" --filter container=YOUR_CONTAINER --filter event=oom --filter event=kill --filter event=die --filter event=start
docker inspect --format 'stop_signal={{.Config.StopSignal}} stop_timeout={{.Config.StopTimeout}}' YOUR_CONTAINER

Repeated event= filters are OR'd. Patterns to look for:

• oom → die exitCode=137: kernel killed a process under the container's limit, main process died.
• oom with no die: something inside was killed, main process survived. Look for a dead worker in the app logs.
• kill signal=15 → ~10 s → kill signal=9 → die exitCode=137: a stop that ran out of grace period. Not memory.
• kill signal=9 → die: someone ran docker kill or docker rm -f.
• die exitCode=137 with no kill before it: the signal came from outside Docker, usually the kernel. Go to step 4.
• A start after every die is the restart policy. Newer Docker adds execDuration to die; if the exits are memory kills, similar values each time suggest something that grows until it's killed.

Docker keeps only recent events in memory (the last 256), and they're gone after a daemon restart, so check soon.

For the stop pattern: empty stop_signal and stop_timeout=<nil> mean the defaults (SIGTERM, 10 s). If PID 1 is a shell wrapper that doesn't forward SIGTERM, or the app has no SIGTERM handler, every stop ends in SIGKILL.

3. Check the limit and who is using memory

docker inspect --format 'memory_limit_bytes={{.HostConfig.Memory}} memory_swap_bytes={{.HostConfig.MemorySwap}}' YOUR_CONTAINER
docker stats --no-stream --format 'table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}'
free -h
ps -eo pid,user,rss,comm --sort=-rss | head -n 10

• memory_limit_bytes=0: no limit, so the container can eat host memory until the whole host runs short. A host-wide OOM is then the likelier path.
• memory_swap_bytes is memory + swap. Equal to the limit means no swap for this container; -1 means unlimited.
• docker stats only covers running containers. Without a limit, the limit column shows host RAM. Several unlimited containers on a 2 GB host are fighting over the same memory.
• In free -h, read available, not free, and check whether there's any swap.
• In a host-wide OOM the kernel picks its victim mostly by size and oom_score_adj, so the process that got killed isn't always the one that caused the pressure. A neighbour can grow and get your container killed.

These are snapshots of now, not of the kill. To tell a spike from steady growth, sample after the container comes back:

for i in 1 2 3 4 5 6 7 8 9 10; do date '+%H:%M:%S'; docker stats --no-stream --format '{{.Name}} {{.MemUsage}}' YOUR_CONTAINER; sleep 60; done

Steady climb that never levels off → leak, or an unbounded cache or queue. Jumps during specific requests or jobs → peak workload. Ten minutes is a hint; a real trend needs hours or days of monitoring.

4. Read the kernel log, and know when OOMKilled lies

sudo dmesg -T | grep -i -E 'oom-kill|out of memory|killed process' | tail -n 20
sudo journalctl -k --since "2026-09-29 14:00" --until "2026-09-29 14:30" --no-pager | grep -i -E 'oom-kill|out of memory|killed process'
docker inspect --format '{{.Id}}' YOUR_CONTAINER

• Pick a window around finished (converted to local time) or a die time. journalctl timestamps are usually more reliable than dmesg -T's converted ones.
• dmesg only covers the current boot. After a reboot, sudo journalctl -k -b -1 shows the previous one, if the journal is persistent.
• Memory cgroup out of memory: Killed process 2314 (node) → a cgroup limit, normally the container's. Out of memory: Killed process … without "Memory cgroup" → the whole host.
• Newer kernels add an oom-kill: line first: constraint=CONSTRAINT_MEMCG for a cgroup limit, CONSTRAINT_NONE plus global_oom for host-wide. task_memcg= usually contains the container ID (e.g. /system.slice/docker-<id>.scope); search for its first 12 characters.
• The PID is the host PID. anon-rss is roughly what the process held when killed.

No matching kernel line? Check for a userspace OOM daemon, which kills without the kernel logging an OOM:

systemctl is-active systemd-oomd earlyoom
sudo journalctl -u systemd-oomd -u earlyoom --since "2026-09-29 14:00" --no-pager

Memory was the cause but OOMKilled is false. How Docker learns about an OOM kill depends on the kernel, cgroup version and containerd/Docker versions. Things to check, not rules:

• The container restarted before you inspected it.
• A userspace daemon did the killing, not the kernel.
• The whole host ran out. On cgroup v1 the event Docker listens for is tied to the container's own limit, so host-wide kills often leave the flag false. On cgroup v2 the kernel counts OOM kills per cgroup, so the flag is more likely set, though older containerd/Docker versions had gaps.
• On Docker Desktop, memory runs out inside Docker's VM, invisible to Mac/Windows tools.

docker info --format 'cgroup={{.CgroupVersion}} driver={{.CgroupDriver}}'
docker exec YOUR_CONTAINER cat /sys/fs/cgroup/memory.events   # cgroup v2, if the image has cat

A nonzero oom_kill means the kernel killed a process in this container since it last started.

5. Pick a next step that matches the evidence

Each of these is a change to plan, with a way back:

• Container limit hit, workload really needs more. Raise mem_limit / deploy.resources.limits.memory / --memory, after checking the host has room for all limits plus the OS. docker update --memory changes a running container, but the next Compose recreate reverts to the file.
• No limit, host ran out. Limits keep one container's growth inside that container, so it dies instead of the database next door. If the services together need more than the host has, it's a bigger instance or fewer services.
• Memory climbs after every start. Looks like a leak or an unbounded cache/queue; a higher limit only delays the kill. Compare with the last deploy and use the runtime's profilers. Recycling workers is a stopgap.
• Runtime heap settings. Keep the runtime's own limit below the container's so failure shows up as a logged error, not a silent 137. JVM (JDK 10+, 8u191+) reads the container limit and typically caps heap at a quarter of it; set -XX:MaxRAMPercentage or -Xmx, leaving headroom for non-heap memory. Node.js: --max-old-space-size=<MB>. Multi-worker servers multiply per-worker memory.
• Swap, with caveats. Absorbs short peaks, but under sustained pressure everything slows instead of failing fast, and leaks hide longer.
• It was a stop, not memory. Handle SIGTERM, use exec-form CMD/ENTRYPOINT or exec in entrypoint scripts so the app is PID 1, or add --init / init: true. Raise the grace period only if clean shutdown genuinely takes longer.
• Something else on the host. Find the script, agent or person from the event timestamp. Memory settings won't fix this.

Before touching production, write down the evidence, the change, how you'll verify it and how you'll roll it back.

Illustrative example

Illustrative example (not a real case): a 2 GB VPS runs api (Node.js), postgres and nginx with Compose. The API drops out for a few seconds several times a day. docker ps -a shows api as Up 12 minutes; inspect says status=running exit=0 oom=false restarts=5, clean only because the last start reset it. docker events --since 24h shows five die exitCode=137, each followed by start, no kill before any of them, so not docker kill or docker stop. memory_limit_bytes=0. One die is at 06:05:41 UTC, 14:05:41 host time (UTC+8). journalctl -k at that minute has an oom-kill: line with global_oom and a task_memcg path containing the api container ID, then Out of memory: Killed process 2314 (node) … anon-rss:1398212kB. Ten docker stats samples after restart show api rising about 15 MB a minute while postgres stays flat.

Together: the host ran out, the kernel picked the Node process in api, and api grows steadily after each start, which fits a leak or unbounded cache better than a one-off peak. Not proven: which code path grows. Next steps are a person's call: give api a limit, set the Node heap below it so failures get logged, and hunt the growth starting from the last deploy.

137 is a clue and OOMKilled is evidence, not proof. Conclude from the exit state, Docker events and kernel lines with matching timestamps.

---

I work on OpsMate, which puts an SSH terminal and an AI assistant on one page, so you can run the commands above yourself or describe the exit in plain language and check against the raw output. Each time you ask the AI for help, recent terminal output is sent to the cloud AI along with your message, so remove sensitive information first. Every action is sorted into one of three risk tiers. Low-risk actions (such as a confident service restart or log rotation) run automatically. Medium-risk actions (such as changing config or recreating a container) are sent to you for approval first, and run automatically if no one responds within 30 minutes. High-risk actions (database schema, kernel parameter, network and firewall changes) only raise an alert and are never executed. When the system isn't sure, it treats the action as the next tier up; dangerous commands are always blocked. The full guide is on itops.sh.

Drafted with AI assistance and reviewed before publishing.