Exit 0, Green Healthchecks, and the OOM Killer That Lied

Exit code 0 means success. A healthcheck is Docker's way of asking a container whether it can still do its job. When a container reports both while a process inside it was just killed by the kernel's OOM killer, every monitoring board you own is lying to you.

Here is the punchline before the story: Docker only tracks PID 1. When the kernel kills a child process, the shell wrapper that is PID 1 keeps running, the healthcheck keeps returning 0, and when the shell finally exits it exits clean. The container's story is exit 0, healthy, running, and the kernel's story is a body count. Both are on the same machine, and they disagree because they answer different questions.

I reproduced the whole failure, the two fixes that do not work, and the one that does, on a fresh Docker 29.1.3 host with cgroup v2. Every command and every line of output below is from that run. If you want the short version: a green healthcheck plus a restart count that never moves is a memory investigation until proven otherwise, and the evidence lives in docker events, /sys/fs/cgroup/memory.events, and dmesg.

Key Takeaways

• Exit code 0 and OOMKilled: true in the same docker inspect is not a contradiction: the code belongs to PID 1, the kill happened to a child.
• A healthcheck only reports whether its command exited 0. Process-existence probes (kill -0, pgrep) stay green over an OOM-killed worker because the corpse is a zombie until its parent reaps it.
• Three signals catch the lie: the oom event in docker events, the oom_kill counter in the container's /sys/fs/cgroup/memory.events, and the Memory cgroup out of memory line in dmesg.
• The fix that works is a functional probe, like a heartbeat file whose staleness fails the check. --init does not fix the probe, and the honest architecture is exec form with one main process per container.

In this post:

• What does the dying container actually look like?
• Why does exit code 0 lie?
• Why does the healthcheck stay green?
• What did the kernel see that Docker missed?
• Why does the obvious fix fail?
• What kind of healthcheck catches a dead worker?
• What does an honest container look like?

What does the dying container actually look like?

The shape of the incident is always the same. Somewhere in your stack there is an image that starts a worker process from a shell wrapper. The CMD uses shell form, or an entrypoint script backgrounds a crawler, a transcoder, a Celery worker, a PHP-FPM pool. It has a memory limit because you set one. Traffic spikes, the worker allocates past the limit, and the kernel's OOM killer picks it.

Here is the exact container I ran, trimmed to the parts that matter:

docker run -d --name liar2 \
  -m 120m --memory-swap 120m \
  --health-cmd='sh -c "exit 0"' \
  --health-interval=5s \
  python:3.12-alpine \
  sh -c 'python3 -c "x = b\"x\" * 400_000_000" & sleep 300'

A Python worker grabs 400 MB inside a 120 MB cgroup limit, and a shell sits as PID 1 holding the container open. This is the pattern behind half the "my container says running but nothing works" threads on r/docker, where the process inside dies to a memory kill and the container stays up as if nothing happened.

Twelve seconds after start, Docker reports:

$ docker ps --filter name=liar2 --format '{{.Status}}'
Up 12 seconds (healthy)

$ docker inspect liar2 --format 'ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}} Status={{.State.Status}}'
ExitCode=0 OOMKilled=true Status=running

$ docker logs liar2
(empty)

Exit code 0. Healthcheck green. Status running. Logs empty. Meanwhile the worker has been dead for ten seconds, and it will stay dead. Nothing will restart it, because as far as every Docker signal you are watching is concerned, nothing happened.

The part that makes this dangerous is not the crash. Crashes are fine; crash loops are loud. This is a silent degradation: the container runs for hours with a dead worker and a healthy badge. If a process supervisor inside the container respawns the worker, you get mystery 502s instead, with no restart recorded anywhere.

Why does exit code 0 lie?

Exit code 0 is the exit status of PID 1, nothing more. In my repro, PID 1 is /bin/sh, which runs sleep 300 in the foreground after backgrounding the Python worker. The kernel OOM-kills the Python child, and the shell does not care, because a background job dying is not the shell's problem. When sleep finishes, the shell exits with status 0.

Docker's job is to report how PID 1 left, and it does that job correctly. The problem is that we read "exit 0" as "everything inside went fine", and those are very different claims once more than one process lives in the container.

There is a variant that is even sneakier, and it is the one that shows up in real incidents. The wrapper script that "handles" failures looks like this:

sh -c 'python worker.py || echo "worker died"; sleep 3600'

If the worker gets OOM-killed here, the shell prints worker died to logs nobody reads, then sleeps for an hour. If you use docker logs --follow on a Friday you will see the message scroll past once and never again. The one-word trace it leaves, Killed, is a shell message that only appears when the killed process was in the foreground of that shell.

Why does the healthcheck stay green?

A healthcheck is a command Docker runs inside the container on an interval. The contract, from the Dockerfile reference: the command's "exit status indicates the health status of the container" (Docker docs). Exit 0 means healthy. Exit 1 means unhealthy. That is the entire protocol.

So the healthcheck answers exactly one question: did my command exit 0? It does not answer: is the process that does the actual work alive? That is the gap the OOM killer walks through.

Most healthchecks in the wild are one of these three, and all three stayed green while my worker was dead:

1. The placeholder: exit 0 or true. Common in base images and inherited compose files. It checks nothing at all.
2. The port probe: wget -q -O /dev/null http://localhost:8080/health. If the worker held the port, the socket stops listening when the process dies, so this one sometimes catches the kill. Unless a supervisor or the shell keeps the port open, or the port is held by a sidecar. Then it lies too.
3. The process probe: kill -0 $(cat /tmp/worker.pid) or pgrep worker. This looks correct. It is not, and the reason why is the best part of this story.

There is a fourth kind that actually works, and I will show it after we watch the obvious one fail.

One more thing the healthcheck does not do: it is not a restart trigger. An unhealthy status shows in docker ps and in docker inspect, and Compose's depends_on: service_healthy gate consumes it at startup, but a healthcheck going red does not itself restart anything. I covered that startup gate in the previous post in this series, where pg_isready plus condition: service_healthy makes Compose actually wait for Postgres. A healthcheck is evidence, not a supervisor. You still need restart: unless-stopped, on-failure, or an orchestrator watching the state.

The sandbox post from the same series is the other half of this pattern: when you hand a whole VM to an AI agent in a Docker Sandbox, you cannot trust a green badge either, you trust the boundary. For example, consider a crawler container on a 512 MB limit: same shell wrapper, same silent worker death, and the same three receipts to collect. Memory forensics is the same discipline aimed at your own containers.

What did the kernel see that Docker missed?

This is where the story turns. Docker missed the kill, but the kernel did not, and it left receipts in three places. This trio is the lie detector, and it costs nothing to run.

Receipt one: the oom event. Docker's event stream emits an oom action for the container the moment the cgroup kill happens (docker system events reference):

$ docker events --since ... --filter event=oom
2026-10-27T08:07:09.720667678Z container oom 3c73a70784e0 \
  (image=python:3.12-alpine, name=liar2)

$ docker events --since ... 2>&1 | grep liar2
...T08:07:09.50Z container start 3c73a70784e0 (name=liar2)
...T08:07:09.72Z container oom   3c73a70784e0 (name=liar2)   <- the kill
...T08:07:14.55Z container health_status: healthy 3c73a70784e0 (name=liar2)

Read that timeline carefully, because it is the whole article in three lines. The oom event fires five seconds before the health_status: healthy event. The healthcheck turned green while the corpse was still warm. docker events has a ring buffer of the last 256 events, so for forensics after the fact use --since with a time window, or run it in the background to a file.

Receipt two: the cgroup counter. On cgroup v2 hosts, every container has a memory.events file inside its cgroup. The kernel documentation defines the oom_kill field as "The number of processes belonging to this cgroup killed by any kind of OOM killer" (kernel cgroup v2 docs):

$ docker exec liar2 cat /sys/fs/cgroup/memory.events
low 0
high 0
max 39
oom 1
oom_kill 1
oom_group_kill 0
sock_throttled 0

oom_kill 1 is the body count. max 39 means the cgroup bumped its ceiling 39 times before that, each bump a round of kernel memory reclaim that shows up as latency before it shows up as a crash. A climbing max with oom_kill 0 is the squeezed-container signature: slow, not dead. The squeeze is not unique to long-running services, either. A 2026 arXiv characterization of sandboxed AI coding agents measured tool-call memory bursts reaching a 15.4x peak-to-average ratio, exactly the workload shape that turns a fixed limit into a kill lottery (AgentCgroup, arXiv 2602.09345). This counter only counts up for the life of the cgroup, so take a snapshot, let it run, then take another and diff them.

Receipt three: the kernel log. If you can read the host's dmesg, the kill line is there with the memory numbers:

Memory cgroup out of memory: Killed process 7 (python3) ...

Two details matter when you read it. The phrase Memory cgroup out of memory means the container's own limit was hit. A line starting with plain Out of memory: instead means the host itself ran out, and your container may have been the victim simply because it had the biggest score. Different problems, different fixes: raise the limit in the first case, fix node capacity in the second.

Put the three receipts together and the diagnosis writes itself: docker events gives you the moment, memory.events gives you the count, and dmesg gives you the anatomy.

Here is every signal from my liar container, side by side. The left column is what your monitoring sees, and the right column is what the kernel knows.

[ORIGINAL DATA] Test setup: one container on this host, Docker 29.1.3, cgroup v2, seven timed runs of the same failure. The 12-second snapshot below is one representative run.

Signal	What it showed 12s after the kill	Truth
docker ps status	Up 12 seconds (healthy)	worker dead for 10s
docker inspect ExitCode	0	exit code belongs to PID 1 (a shell)
docker inspect OOMKilled	true (set for the cgroup, ignored by the eye because everything else looks fine)	a process in this cgroup was OOM-killed
docker logs	empty	the killed process was backgrounded, shell prints nothing
docker events	container oom at T+0.7s, then health_status: healthy at T+5s	the kill, timestamped, before the green badge
memory.events	oom_kill 1, max 39	one kill, 39 ceiling bumps before it
ps aux in the container	PID 7 shown as [python3]	a zombie: the corpse the healthcheck probes

The row that surprises people is OOMKilled true. Docker does record the cgroup kill in the state. The lie is not a missing signal, it is the combination: nobody treats a true there as an alarm when the status line says running and the badge says healthy. If you take one habit from this post, make it this: OOMKilled: true under a green status is the kernel screaming into a pillow.

Why does the obvious fix fail?

Knowing that the healthcheck checked the wrong thing, the obvious move is to check the right thing: is the worker process alive? Record its PID at startup, probe it in the healthcheck:

healthcheck:
  test: ["CMD", "sh", "-c", "kill -0 $(cat /tmp/worker.pid)"]
  interval: 5s
  retries: 2

I ran it. Twenty seconds after the kill: Up 20 seconds (healthy). Still green. Still lying.

The reason is a Unix joke with a body count. When the kernel kills the worker, the process does not fully disappear. It becomes a zombie: it stays in the process table, holding its PID and its exit status, until its parent calls wait() to collect it. The shell that backgrounded it never will while it is busy running sleep in the foreground. And kill -0 checks process existence, which a zombie has. Process existence is literally the one thing a zombie still has.

$ docker exec liar2 ps aux
PID   USER     TIME  COMMAND
    1 root      0:00 sleep 300
    7 root      0:00 [python3]        <- brackets = zombie, and kill -0 still loves it
   28 root      0:00 ps aux

The bracketed [python3] is the tell. A process in brackets has no userland memory left. It is a receipt waiting to be picked up, and your process-existence healthcheck just asked whether the receipt exists.

You might think, like I did, that docker run --init solves this, since the flag runs "an init inside the container that forwards signals and reaps processes" (docker run reference). I ran that too. Still healthy. Still a zombie in the table. docker-init reaps the processes it parents, the shell in this case, but the shell was never the problem. The worker is the shell's child, and only the shell can reap it. The zombie survives until the shell exits, and by then your healthcheck has told a room full of dashboards that everything is fine.

pgrep does not save you either. It finds zombies the same way kill -0 does, by walking the process table. The whole family of "is there a process with this name" checks is blind to the difference between a worker and its ghost.

What kind of healthcheck catches a dead worker?

If the process table cannot be trusted, stop asking about processes and start asking about work. A functional probe tests something the worker does on a schedule, so that its death has a side effect the healthcheck can see.

The heartbeat pattern: the worker thread writes a timestamp to a file every two seconds. The healthcheck fails when the timestamp is stale:

healthcheck:
  test: ["CMD", "sh", "-c",
    "test $(( $(date +%s) - $(cat /tmp/alive 2>/dev/null || echo 0) )) -lt 10"]
  interval: 5s
  retries: 2

The worker-side code is three lines:

import threading, time

def beat():
    while True:
        open("/tmp/alive", "w").write(str(int(time.time())))
        time.sleep(2)

threading.Thread(target=beat, daemon=True).start()
x = b"x" * 400_000_000

I killed the worker the same way as before. Sixteen seconds later:

$ docker ps --filter name=beat6 --format '{{.Status}}'
Up 16 seconds (unhealthy)

$ docker inspect beat6 --format 'FailingStreak={{.State.Health.FailingStreak}}'
FailingStreak=2

oom_kill 1 in the cgroup, unhealthy on the badge, with a stale heartbeat to explain it. The zombie is still there, and it does not matter, because the healthcheck never asked about the zombie. It asked whether the work is happening, and the work stopped.

Three caveats from doing this for real. The staleness window must be longer than one write interval, or a single slow tick fails the check: ten seconds against a two second beat gives four missed ticks before failure, which is a reasonable margin. Timestamps come from the container's clock, so a stopped or skewed clock breaks the math. And if the healthcheck command itself needs the worker to respond, like hitting a /health endpoint that serves from memory, make sure the response cannot be cached by anything between the probe and the app.

For multi-worker setups, the same idea scales: each worker updates its own heartbeat file, and the check requires the newest of them all to be fresh. Supervisord users get this pattern almost free with its event listeners; the Celery equivalent is the celery inspect ping probe, which asks the broker-side workers to answer, not merely to exist.

What does an honest container look like?

The deepest fix is architectural, and it is the Docker way whether or not it is convenient: one main process per container, with that process as PID 1. Exec form, not shell form:

CMD ["python", "worker.py"]

not:

CMD python worker.py

When the worker is PID 1 and the kernel kills it, nothing is left to lie. My last repro, with --restart=on-failure:3:

$ docker inspect honest7 --format \
  'ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}} Status={{.State.Status}} RestartCount={{.RestartCount}}'
ExitCode=137 OOMKilled=true Status=exited RestartCount=3

Exit 137, OOMKilled: true, restart count climbing. Every signal agrees with the kernel, and the restart policy actually has something to act on. Same failure, zero lies.

Shell form is sometimes unavoidable, and backgrounded workers are a fact of life in real images. For those, combine the layers. Keep the functional healthcheck so the lie is visible. Alert on the oom event and the memory.events counters so the kill is counted. Let the wrapper, not your on-call rotation, handle the respawn. --init belongs in that toolbox for signal forwarding to backgrounded children, which is its actual job, and I have stopped expecting it to fix problems it was never aimed at.

There is one more knob worth knowing about even though you will almost never touch it: --oom-kill-disable. It stops the kernel from OOM-killing inside the container. The resource constraints documentation describes it and its companion soft limit --memory-reservation (Docker docs). Disable the killer and the container's memory pressure becomes somebody else's problem, usually the host OOM killer's, which picks victims with worse aim. Know it exists; use it about as often as you use dd.

What is the field kit when it happens in production?

When a container says healthy and the world says otherwise, run these commands in this order.

docker events --since 24h --filter event=oom
docker exec <container> cat /sys/fs/cgroup/memory.events
dmesg -T | grep -iE "invoked oom-killer|Killed process"
docker exec <container> ps aux
docker inspect <container> --format \
  'ExitCode={{.State.ExitCode}} OOMKilled={{.State.OOMKilled}} Status={{.State.Status}}'

The oom event gives you the moment. The cgroup counter gives you the body count, the kernel log the anatomy, the process table the zombies, and the inspect line what Docker is prepared to admit.

For the pipeline side, one line of defense beats all five: a docker events listener that pages on event=oom. The event fires the moment the kernel pulls the trigger, no polling, no math, no waiting for a human to notice a pattern. The cgroup counter is your audit trail; the event is your pager.

The lesson I keep from this one: when exit code and healthcheck agree that everything is fine, and something in production clearly is not, believe the kernel first. Its records are not pretty, but they are honest.

A question I am still turning over: what does your shop alert on? I have met teams who page on OOMKilled: true, teams who only page on restarts, and teams whose first signal is a customer ticket. If a container can sit there, green and dead, for a week, something in that alerting chain is measuring the wrong thing, and I am collecting examples of which signal actually caught it first. Tell me yours in the comments, especially if the answer is "the customer".

Verified against Docker 29.1.3 on cgroup v2 (kernel 7.0.0-gcp), 2026-10-27, with a fact check against every linked source. Every command and output block in this post is from a live run on that host. Claims about Docker and kernel behavior are cited to the Docker docs and to the Dockerfile, docker run, and kernel cgroup v2 references. Anything I could not reproduce or cite, I cut.

Sources

Every claim above traces to these primary sources. Where I could not verify something hands-on, the post says so. Links were fact-checked against each page on 2026-10-27.

• Dockerfile reference, HEALTHCHECK instruction.
• docker system events reference.
• docker run reference.
• Docker resource constraints documentation.
• Kernel cgroup v2 documentation.
• AgentCgroup: Understanding and Controlling OS Resources of AI Agents, arXiv 2602.09345, the measurement of tool-call memory bursts behind the 15.4x peak-to-average figure.
• Docker Compose healthchecks: make depends_on actually wait for Postgres, the earlier post in this series on the startup gate.
• Your AI Agent Can Run rm -rf. Put It in a Sandbox., the isolation half of the same series.

About the author

Kunal Das writes operations engineering notes as he learns them, always against primary sources. Contact: GitHub and Dev.to. Follow the Docker tag for the next posts in this series.