Exit code 137 is 128 + 9 — your process was killed with SIGKILL, and in a container that almost always means the cgroup memory limit was hit and the kernel OOM killer picked your process. The reason your heap graph looks innocent is that the limit applies to the container's working set (RSS: V8 heap + native buffers + thread stacks + mapped code + allocator overhead), while heapUsed only measures one slice of that. Debugging this starts by measuring the right number, not by raising the limit.
I have burned entire afternoons on this, twice, because the dashboard I was staring at was accurate and irrelevant at the same time.
How do I confirm it was actually an OOM kill and not a crash?
A Node process that runs out of heap dies loudly. You get a stack trace and a nonzero-but-not-137 exit:
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
An OOM kill is silent. There is no stack trace, no uncaughtException handler firing, no last log line — SIGKILL cannot be trapped. That silence is the diagnostic signal. Confirm it from the outside:
# Kubernetes
kubectl describe pod my-api-7d9f | grep -A3 'Last State'
# Last State: Terminated
# Reason: OOMKilled
# Exit Code: 137
# Plain Docker
docker inspect my-api --format '{{.State.OOMKilled}} {{.State.ExitCode}}'
# true 137
# The kernel's side of the story, if you can reach the host
dmesg -T | grep -i 'memory cgroup out of memory'
If Reason: OOMKilled is absent and you still see 137, someone or something sent SIGKILL — a docker kill, a failed liveness probe escalating past its grace period, or a node draining. Those are different bugs with different fixes.
Takeaway: a clean stack trace means you ran out of heap; total silence plus 137 means you ran out of container.
Why does heapUsed stay flat while RSS climbs?
process.memoryUsage() returns five numbers, and most dashboards graph the wrong one:
setInterval(() => {
const m = process.memoryUsage();
const mb = (n) => Math.round(n / 1024 / 1024);
console.log(JSON.stringify({
rss: mb(m.rss), // what the kernel counts against your limit
heapTotal: mb(m.heapTotal), // V8 heap reserved
heapUsed: mb(m.heapUsed), // what most dashboards graph
external: mb(m.external), // C++ objects bound to JS objects
arrayBuffers: mb(m.arrayBuffers), // Buffers, TypedArrays — off-heap
}));
}, 15_000);
Every byte you read from a socket, a file stream, an image pipeline, or a database driver's binary protocol lands in arrayBuffers/external, not in old space. A service that buffers uploads or concatenates response bodies can hold hundreds of megabytes off-heap while heapUsed sits at a tidy 90 MB. V8's garbage collector is also under no pressure to run, because from its point of view there is plenty of room — the memory it cares about is not the memory that is running out.
Native modules make this worse in a way that is easy to misread. sharp, canvas, grpc, and similar addons allocate through their own libraries; leaked handles there never appear in a heap snapshot at all.
Takeaway: if RSS and heapUsed diverge, stop reading heap snapshots — the memory you are losing is not on the JavaScript heap.
What limit is Node actually enforcing inside a container?
Two limits exist, they are set in different places, and nothing makes them agree:
| Limit | Set by | Covers | What happens when you hit it |
|---|---|---|---|
| V8 old-space max |
--max-old-space-size / NODE_OPTIONS
|
JS heap only | Aggressive GC, then a fatal heap out of memory error with a stack trace |
| cgroup memory max | Docker --memory, k8s resources.limits.memory
|
Everything: RSS, page cache, allocator overhead | SIGKILL, exit 137, no output |
Read the real limit from inside the container rather than trusting the manifest:
# cgroup v2 (the default on current distros as of mid-2026)
cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current
# cgroup v1
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
Node tries to size its default heap from the memory it detects, but that detection depends on the runtime version and on which cgroup version the host uses, so I no longer rely on it in either direction. Set it explicitly, below the container limit, leaving room for everything off-heap:
# container limit 1024Mi → heap ceiling 768Mi, ~256Mi for buffers, stacks, allocator
ENV NODE_OPTIONS="--max-old-space-size=768"
The point of that gap is not politeness. A heap ceiling under the container ceiling converts a silent SIGKILL into a loud, debuggable JavaScript heap out of memory crash with a stack trace — you trade an unexplained restart for an actual error message.
Takeaway: set --max-old-space-size to roughly 70–80% of the container limit so leaks surface as crashes you can read instead of kills you can't.
Why does memory keep growing when there is no leak?
Three causes I hit far more often than an actual leak:
Allocator fragmentation. glibc's malloc creates per-thread arenas, and a process with many threads (Node's libuv pool, native addons) can hold on to a lot of freed-but-unreturned memory. Capping arenas with MALLOC_ARENA_MAX=2 or switching the image to jemalloc is worth testing — it flattened RSS growth in one image-processing service of mine and did nothing measurable in two others, so treat it as an experiment, not a fix.
Page cache counted as usage. memory.current includes file cache, which inflates the number without being a real problem. Kubernetes evaluates the working set, not raw usage, so graph container_memory_working_set_bytes — Prometheus with cAdvisor metrics exposes exactly this series, and it is the one that lines up with an OOM kill. Its drawback is granularity: at a 15–30 second scrape interval it will miss a spike that kills you in under a scrape.
Unbounded concurrency. Memory tracks in-flight requests. A queue consumer that pulls 500 messages at once, or an endpoint with no upload size cap, is not leaking — it is just allowed to use more memory than exists. If you want a managed alternative to burning a week on this, Datadog's container memory views correlate working-set spikes with the specific request traces that caused them, at the cost of another per-host bill.
For genuine leaks, capture a snapshot from a live process without a debugger attached:
node --heapsnapshot-signal=SIGUSR2 server.js
kill -USR2 <pid> # writes a .heapsnapshot into cwd; diff two of them in Chrome DevTools
Take one snapshot after warmup and one an hour later, and compare — a single snapshot tells you almost nothing.
Takeaway: raising the memory limit is a legitimate fix only after you have ruled out fragmentation, page cache, and unbounded concurrency.
FAQ
What does exit code 137 mean in Docker?
It means the process received SIGKILL (128 + signal 9). In containers, this is nearly always the kernel OOM killer enforcing the cgroup memory limit. Check docker inspect --format '{{.State.OOMKilled}}' — if it prints true, it was memory, not your code.
Why is my Node.js container OOMKilled when heap usage is low?
Because the container limit counts RSS, which includes off-heap memory: Buffers and TypedArrays (arrayBuffers), native addon allocations, thread stacks, and allocator overhead. Log process.memoryUsage().rss alongside heapUsed; if they diverge, the growth is off-heap and heap snapshots will not show it.
Should I set --max-old-space-size in a container?
Yes. Set it explicitly at roughly 70–80% of the container's memory limit rather than relying on Node's automatic detection, which varies by version and cgroup setup. The gap leaves headroom for off-heap memory and turns silent OOM kills into readable heap-exhaustion errors.
Bottom line
If you see 137 with no logs, confirm OOMKilled first — that single check separates a memory problem from a lifecycle problem and saves you from debugging the wrong thing. Then graph RSS and working set, not heapUsed, and set --max-old-space-size under the container limit so future failures arrive with a stack trace attached. Reach for heap snapshots only once RSS and heapUsed are growing together; if they have diverged, look at buffers, native addons, and concurrency limits instead. Raising the limit is a valid last step, but it is a decision you should make with the working-set graph in front of you, not as a reflex.
Top comments (0)