Fix Python Memory Leaks in Production: Debugging, Monitoring, and Prevention

#python #memory #debugging #production

Introduction

Memory leaks in long‑running Python services can silently degrade performance and eventually cause crashes. This guide walks you through identifying, reproducing, and fixing Python memory leaks in production environments.

1. How Python Leaks Happen

Reference cycles involving objects with __del__ methods.
Unreleased C extensions (e.g., numpy, pandas buffers).
Global caches that grow unchecked.
Improper use of multiprocessing queues.

2. Profiling the Leak

import tracemalloc, psutil, os

tracemalloc.start()

def log_memory():
    current, peak = tracemalloc.get_traced_memory()
    rss = psutil.Process(os.getpid()).memory_info().rss / (1024**2)
    print(f"RSS: {rss:.2f} MB | Traced: {current/1024:.1f} KiB (peak {peak/1024:.1f} KiB)")

# Call `log_memory()` at regular intervals or inside a health‑check endpoint.

Use tracemalloc.take_snapshot() and snapshot.statistics('lineno') to pinpoint the hot‑spot.

3. Step‑by‑Step Fix Workflow

Reproduce the leak locally with a workload that mimics production traffic.
Capture a baseline snapshot before the suspect code runs.
Run the workload, then capture a second snapshot.
Compare snapshots:

python -m pip install memray
memray flamegraph -o leak.svg your_script.py

Identify the growing objects (e.g., list of dicts, numpy.ndarray).
Patch the code:
- Break reference cycles (weakref).
- Explicitly close() C‑extension resources.
- Limit cache size with functools.lru_cache(maxsize=…).
Validate that memory usage stabilises.

4. Real‑World Example

# Before: a global list that never gets cleared
leaked_data = []

def handle_request(payload):
    # ... processing ...
    leaked_data.append(payload)   # ← memory grows indefinitely

Fix:

from collections import deque

# Bounded buffer – old entries are discarded automatically
leaked_data = deque(maxlen=1000)

def handle_request(payload):
    # ... processing ...
    leaked_data.append(payload)

5. Deploying the Fix Safely

Blue‑Green rollout: Deploy the patched version alongside the old one.
Canary monitoring: Track process_resident_memory_bytes via Prometheus.
Graceful restart: Use systemd Restart=on-failure and a low StartLimitInterval.

6. Ongoing Monitoring

Add a Prometheus exporter:

from prometheus_client import start_http_server, Gauge
import psutil, os, time

mem_gauge = Gauge('python_process_rss_bytes', 'Resident set size')
def update_metrics():
    mem_gauge.set(psutil.Process(os.getpid()).memory_info().rss)

start_http_server(8000)
while True:
    update_metrics()
    time.sleep(30)

Set an alert when RSS exceeds a threshold for more than 5 minutes.

7. Ready‑to‑Use Patch Toolkit

We've packaged the profiling snippets, a reusable MemoryGuard context manager, and deployment scripts into a single repository. Download the pre‑configured script here, or Get the complete patch tool. For the full source, Access the full repository fix.

By following this systematic approach, you can eradicate hidden memory leaks before they impact your users and keep your Python services running smoothly in production.

DEV Community