How to debug memory leaks in production without a zero downtime migration going sideways

Your memory leak fix can cause the outage your leak never did

Here's a scenario that plays out more often than anyone likes to admit: a service that ran fine at 400MB for months starts getting OOM-killed every day and a half, nobody can point to the deploy that caused it, and the on-call engineer's first instinct is to just restart things and ship a patch fast. That instinct is exactly how a slow leak turns into a self-inflicted outage. This is a process for finding the leak, proving the fix, and rolling it out without dropping a single request.

Full reference: debugging memory leaks in production without breaking your zero downtime migration

What you need before starting

• Shell access to the host or container, with permission to attach a profiler to a live process
• A load balancer or reverse proxy in front of the service (Nginx, HAProxy, cloud LB); without this, zero downtime isn't really on the table
• Memory, CPU, and latency metrics scraped at 30 second intervals or tighter. No metrics, no trend, no debugging. Set up node_exporter or equivalent first.
• The ability to spin up a second instance for canary testing

Examples below use Node.js and systemd, but the process transfers directly to Ruby, Python, Java, or Go. Only the tooling changes.

Step 1: Make sure it's actually a leak

Caches warming up look like leaks for the first hour. A real leak climbs indefinitely and correlates with uptime, not traffic volume.

watch -n 5 'ps -o pid,rss,etime,cmd -p 1234'

If RSS climbs linearly regardless of traffic and only drops on restart, that's a leak. If it climbs with traffic and then flattens, that's cache sizing, a different problem entirely.

Step 2: Pull evidence before you touch the process

Restarting first is the number one mistake here. You lose the exact evidence you need to diagnose anything.

Node.js, heap snapshot without stopping the process:

kill -USR2 1234   # if a SIGUSR2 handler is wired up to write snapshots
# or with --inspect already running:
ssh -L 9229:127.0.0.1:9229 user@server

Java:

jmap -dump:live,format=b,file=heap.bin 1234

Python, no process interruption:

py-spy dump --pid 1234
py-spy record -o profile.svg --pid 1234 --duration 60

Take two snapshots, 30 to 60 minutes apart, under normal load. The diff is the actual signal. The absolute numbers aren't.

Step 3: Read the diff

Load both snapshots into Chrome DevTools (Memory tab > Load) and use comparison view. Check for object counts that keep growing without a matching drop. Usual suspects:

• Event listeners added without a matching removeListener
• Unbounded Maps or caches with no TTL, no max size, often keyed by user or request ID
• Closures holding onto large objects past the life of the request that created them
• Connection pools that grow without a ceiling under retry logic

For Java, run the dump through Eclipse MAT's Leak Suspects report. For Python:

import tracemalloc
tracemalloc.start()
# ... later ...
snapshot1 = tracemalloc.take_snapshot()
# ... wait ...
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
for stat in top_stats[:10]:
    print(stat)

Step 4: Reproduce it locally, don't just guess

Load test it:

autocannon -c 50 -d 300 http://localhost:3000/your-endpoint
# in another terminal
while true; do ps -o rss= -p $(pgrep -f server.js); sleep 10; done

If local memory climbs the same way production did, you've confirmed it, and you've got a repeatable test to validate the fix.

Step 5: Patch it, then re-run the exact same test

Add the TTL, remove the listener, cap the pool, whatever the fix is. Memory should plateau within a few minutes of steady load, not keep trending up.

Working locally proves the code is right. It says nothing about whether the rollout is safe. Don't skip the next part.

Step 6: Ship it as a zero downtime migration, not a restart

The instinct after finding a leak is to restart everything right now. Resist that. Treat it like any other production change.

Behind a load balancer with multiple instances:

ansible-playbook -i inventory deploy.yml --limit app-node-1

curl -X POST http://localhost:8080/-/drain   # app-specific drain endpoint
# or for Nginx: mark backend down, reload, then stop the old process
nginx -s reload

Single instance behind systemd, use blue-green on the same box:

# Start the patched version on a new port
PORT=3001 systemctl start app-patched.service

# Health check before switching traffic
curl -f http://localhost:3001/health || echo "not ready"

# Flip the upstream
sed -i 's/3000/3001/' /etc/nginx/conf.d/upstream.conf
nginx -s reload

# Only now stop the old process
systemctl stop app.service

Order matters: start new, verify healthy, switch traffic, stop old. Do it out of order and your memory leak fix becomes the outage.

On Kubernetes, same principle via rolling update:

kubectl set image deployment/app app=registry/app:patched
kubectl rollout status deployment/app

Make sure the readiness probe actually checks application health, not just that the port is listening. A pod that accepts connections before it's actually warmed up will drop requests mid-rollout, which defeats the point of doing this carefully.

How to know it actually worked

Don't close the ticket right after deploy. Watch it for a full business cycle, ideally 24-48 hours.

• RSS trend: should plateau over 48 hours, not just slow its climb
• Restart/OOM count: journalctl -u app.service | grep -i "oom" or kubectl get events --field-selector reason=OOMKilling
• 5xx rate during rollout: should stay at baseline; a spike means your drain or readiness logic has a gap
• GC pause times (for GC'd runtimes): a healthier heap usually means shorter, more predictable p95 GC pauses

A reasonable target: RSS variance over 48 hours should stay within 10-15% of its post-restart baseline.

Originally published on binadit.com