Docker Reported Exit 0. The Kernel Had Already OOM-Killed It.

Originally published on my blog — read it there with the full context, code and updates.

The notification I expected to be a config mistake

My NAS sent the notification every Docker user learns to skim: container
stopped unexpectedly. This one is the edge proxy in front of my
self-hosted wiki, and it had been up for two weeks. Nobody had deployed
anything that morning and nothing in the compose file had been touched, so
the two available stories were "it crashed" and "something restarted it".

Both were wrong, and the fact that I could not tell from Docker's own report
is the part worth writing down.

Docker's report says nothing happened

State:
  Status:      running
  Running:     true
  ExitCode:    0            <- a clean exit
  OOMKilled:   false        <- not killed by the kernel
  FinishedAt:  2026-10-15T19:39:33.63057879Z
  StartedAt:   2026-10-15T19:39:38.769326672Z   <- restarted 5 s later
RestartCount: 6
RestartPolicy: always

Read that as written and the conclusion is: the process exited cleanly, six
times, and the restart policy gave it a new life each time. A long-running
proxy does not exit with status 0 on its own — and a clean exit that repeats
six times is a contradiction, not an answer. ExitCode and OOMKilled are
exactly the two fields I would have trusted here. On this host, both were
wrong.

The kernel keeps its own record

On Linux, a cgroup memory kill is logged by the kernel even when the
container runtime's bookkeeping does not carry it forward:

[1737797.044856] traefik invoked oom-killer: gfp_mask=0x2400040, order=0, oom_score_adj=0
[1737797.123736] Memory cgroup out of memory: Kill process 615 (traefik) score 970 or sacrifice child
[1737797.132800] Killed process 615 (traefik) total-vm:2525800kB, anon-rss:478480kB, file-rss:0kB

The uptime stamp 1737797 corresponds to 19:39:33 UTC — to the second, the
container's FinishedAt. That is the whole diagnosis, and it took one
dmesg to get it. The container did not exit; it was killed inside its own
memory limit and restarted 5 seconds later.

dmesg holds six such events, and RestartCount was 6. Every other
container in that stack has a restart count of zero and a start time from
weeks ago, so the six kills pin to this one container:

dmesg uptime	local time	anon-rss at kill
1406539	Oct 12, ~07:37	491 MB
1407759	Oct 12, ~07:57	494 MB
1432574	Oct 12, ~14:49	486 MB
1734793	Oct 16, ~02:45	487 MB
1735495	Oct 16, ~02:58	485 MB
1737797	Oct 16, 03:39 ← the notification	478 MB

Every kill happens at 478–494 MB against a limit of 512 MiB (536,870,912
bytes). That is a ceiling being touched, not a runaway.

Two more readings made the mechanism clear. The cgroup's recorded peak — the
counter that survives the restart — was already back at exactly
536,870,912 18 minutes after the restart, i.e. the container touched
its ceiling again in the same cycle without being killed. And the usage
curve oscillated (176 → 308 → 361 MB) instead of climbing monotonically, so
this is allocation churn under load hitting a small cap, not a provable slow
leak. The prime suspect is a geoblock plugin doing an external HTTPS lookup
for every client IP it has not seen before.

A seventh kill landed at 05:33 that morning, before the fix went in.

On this DSM build the truth was only in dmesg plus the cgroup's
memory.max_usage_in_bytes. If you are chasing the same shape on a normal
host, docker inspect -f '{{.State.OOMKilled}}' is the field to read first
— it is supposed to be authoritative. Here it read false five seconds
after the kernel had logged the kill, which is the reason this post exists.

What was actually breaking: a crawler flood

Host memory was never the problem — 64 GB installed, ~39 GB available. One
container's 512 MiB ceiling was.

The driver was traffic, and the numbers came out of the container's own
access log. In the last 50,000 lines:

distinct client IPs      25,976
403 responses            33,636   (geoblock / deny-list doing its job)
429 responses            16,082   (inflight and rate limiting doing their job)
container CPU            118–230%
host load average        26–32

Each restart refilled the heap in 12 to 40 minutes. Nothing in the
configuration was wrong, and neither were the 403s and 429s — they are the
filter rejecting crawlers exactly as designed. The problem is who is
knocking: this container is the public edge for a multi-terabyte Wikipedia
corpus served from a home NAS, behind a country allowlist and a bot filter.
That is an extremely attractive target for a residential-proxy crawler
cloud, and the responses telling it to go away are still work.

The fix, and the sentence that has to come with it

 deploy:
   resources:
     limits:
-      memory: 512M
+      memory: 2g
 restart: always
+memswap_limit: 2g

Verification, because "I raised the limit and it feels better" is not a
verification:

• Only that container was recreated. The wiki, the proxy core and the robots container still reported two weeks of uptime — the gap users saw was seconds, not a stack restart.
• HostConfig.Memory = 2147483648, MemorySwap = 2147483648, and the cgroup's memory.limit_in_bytes agreed with both; RestartCount went back to 0.
• Peak usage after 18 minutes: 210 MB, against a ceiling it had been pinning at 512 MB. By midday it sat at 94 MiB.
• The mount list still matched the file, including the plugin directory mount — losing that one had once disabled every router in this stack.
• Both plugins were loaded and active (country allowlist, bot filter), and the routes returned 200 once the filter's score window expired.

And what it is not: this is a stopgap. The flood is the driver, so an
order-of-magnitude bigger flood reaches 2 GB too. The real fix is shrinking
the public entry surface, not enlarging the cage around it — and a limit
bump that is not labelled as a stopgap becomes a permanent explanation.

I chose to disable swap for this container rather than let it swap:
memswap_limit equal to the memory limit does exactly that. The host
already had 16 GB of its 40 GB swap in use with committed memory at 92 GB,
and an edge proxy that starts swapping slows the entire NAS — the disk array
is the bottleneck here, so swapping buys a slower failure rather than a
longer life.

The trap that nearly made me report a self-inflicted outage

During verification, my first bare curl probes against the site returned
403, and it looked exactly like a regression I had just caused by
restarting the proxy. It was not: the bot filter had scored my own client
IP (blocked user-agent, score 80) and placed it in a 15-minute score
window, because I had been hammering the host.

The tell was in the same test run — /robots.txt returned 200 and the
internal /cdn path returned 400 at the same moment, i.e. the router was
healthy and only my own requests were being filtered. On a host where you
are also a client, never diagnose with your own bare curl: probe a path the
filter ignores, or read a 200 out of the access log.

What I would do differently

1. Treat "stopped unexpectedly" plus ExitCode 0 as an instruction to read dmesg, not as evidence of a clean exit. A restart count above zero is a memory investigation until proven otherwise.
2. Alert on the cgroup peak, not only on restarts. The post-restart reading that was already pinned at the ceiling told me a seventh kill was coming about eighteen minutes before the notification arrived.
3. Get the "who is hitting me" number early. 25,976 distinct IPs in one log sample is the entire story, and I found it after spending time on the memory curve.
4. Question the container, not the server. "The server is out of memory" and "one container has a 512 MiB ceiling on a 64 GB machine" need completely different responses, and the second one is invisible unless you look at the limit.

The result

No OOM kill since the change, with a peak at 210 MB after 18 minutes against
a ceiling that used to be touched in every single cycle — and the seven
kills that had been silently restarting the public edge for four days
stopped appearing in dmesg. The crawler flood did not stop, which is why I
recorded the failure mode in my ops notes next to the stack: "container
stopped unexpectedly" plus exit 0 / OOMKilled=false is a misleading
signature, and on this platform the honest evidence lives in the kernel log
and the cgroup's high-water mark.

---

I write about self-hosted infrastructure, DevOps and the failures behind them at blog.hoelee.com.

Comments

[Launch Gate]

The exit-code-0 lie has a monitoring fix that doesn't require reading dmesg by hand: alert on RestartCount going up, not on the container being down. A proxy with restart-always heals itself in 5 seconds, so uptime checks stay green through the whole incident - the only signal that moved is the restart counter. One more field worth watching alongside it: the cgroup's memory.events oom_kill counter, which increments even when Docker's inspect bookkeeping misses the kill, and can be scraped without root access to logs. The pair (restart count rising, oom_kill counter rising) tells you it's a memory limit problem before you open a single log.

[Odinaka Robert Nnamani]

The detail I'd promote to the headline is the 18-minute one: the cgroup peak was already back at the ceiling 18 minutes after the restart, which means the seventh kill was knowable before it happened — RestartCount tells you after the fact, but memory.max_usage_in_bytes trending at the ceiling turns 'it'll OOM again' from a hunch into a timer. And the 'and what it is not' section is the part I'd underline twice — I've watched limit bumps quietly become permanent architecture because nobody wrote down the stopgap sentence. Did the 2g limit hold through the next crawler wave, or did the flood scale up to meet it?