Your Python Service Grows 40 MB a Day and Then OOMs. Here's How to Find the Leak in an Hour

A gunicorn worker with a 512 MB --max-requests-less config starts at 180 MB RSS. Three days later it's 480 MB. On day four the OOM killer takes it, the health check flaps, and your pager fires at 3 AM. Restarting the worker fixes it for exactly three days. gc.collect() in a debug endpoint reclaims 2 MB — which tells you the objects aren't garbage, they're referenced. Something is holding them.

The reflex is to reach for gc.get_objects() and dump counts. Don't. That gives you a 200,000-line wall of <class 'dict'> and tells you nothing. The workflow below is what actually works: tracemalloc to find the allocation site, objgraph to find the referrer chain, and a repeatable before/after diff so you're not chasing noise.

Step 1: Confirm it's a real leak, not fragmentation

RSS growing is not proof of a leak. glibc's arena allocator can hold freed memory indefinitely, and Python's pymalloc holds onto arenas after free. Before you spend an hour on this, rule out fragmentation:

import gc, resource, os

def rss_mb():
    # ru_maxrss is in KB on Linux, bytes on macOS. Normalize.
    return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024

def live_objects():
    gc.collect()
    # Count only objects the GC tracks + the ones it doesn't.
    # len(gc.get_objects()) alone misses untracked types like str/bytes.
    return len(gc.get_objects())

Run your suspected workload in a loop. If live_objects() climbs monotonically and rss_mb() climbs with it, it's a leak. If live_objects() is flat but RSS climbs, it's fragmentation or a C extension holding memory outside the GC's view — different problem, different fix (usually MALLOC_ARENA_MAX=2 in the environment, or PYTHONMALLOC=malloc to bypass pymalloc and confirm).

In our case live_objects() went from 412,000 to 1,204,000 over 200 requests. Real leak.

Step 2: tracemalloc, filtered aggressively

tracemalloc records the traceback of every allocation. Unfiltered it's slow and noisy. Start it after warmup, take two snapshots across a known workload, and diff:

import tracemalloc
import linecache

def leak_report(before, after, top=15):
    stats = after.compare_to(before, 'lineno')
    for stat in stats[:top]:
        # stat.size_diff is bytes; stat.count_diff is object count.
        # A leak shows up as both growing together.
        if stat.size_diff <= 0:
            continue
        frame = stat.traceback[0]
        # Resolve the actual source line — traceback[0] gives the
        # allocation site, not the call site, which is what you want.
        line = linecache.getline(frame.filename, frame.lineno).strip()
        print(f"{stat.size_diff/1024:8.1f} KB  "
              f"{stat.count_diff:+7d} objs  "
              f"{frame.filename}:{frame.lineno}  {line}")

Wire it up:

tracemalloc.start(25)  # 25 frames of traceback; default 1 is useless for callers
warmup()               # let caches, connection pools, lazy imports settle
snap1 = tracemalloc.take_snapshot()
for _ in range(100):
    handle_request(fixture_payload)
snap2 = tracemalloc.take_snapshot()
leak_report(snap1, snap2)

Real output from the service in question:

  4820.3 KB    +1204 objs  /app/serializers.py:88  return cls._cache[key]
  4811.0 KB    +1201 objs  /app/handlers.py:142    ctx = RequestContext(payload)
   120.4 KB      +30 objs  /usr/lib/python3.11/json/encoder.py:257

Two lines, near-identical counts. _cache[key] and RequestContext(payload). That's your smoking gun — the allocation site is a dict write that never gets a matching delete.

Step 3: objgraph to find who holds the reference

tracemalloc tells you where memory was allocated. It does not tell you why it's still alive. That's objgraph's job. Install it (pip install objgraph, and graphviz if you want the PNG output).

import objgraph

# After the workload, before you restart:
objgraph.show_growth(limit=10)
# Prints types whose count grew since the last call. Call it once
# early to set the baseline, then again after the workload.

# Now find *instances* of the leaking type and see what refs them:
import gc
ctxs = [o for o in gc.get_objects() if type(o).__name__ == 'RequestContext']
print(len(ctxs))  # 1204 — matches the tracemalloc count
objgraph.show_backrefs(ctxs[0], max_depth=4, filename='refs.png')

The show_backrefs graph is where it clicks. In our case the chain was:

RequestContext
  <- dict (value in _cache)
    <- _cache (module-level dict in serializers.py)

A module-level dict keyed by something that was per-request unique. Let me show you the actual code, because the bug is subtle and common:

# serializers.py — the buggy version
class Serializer:
    _cache = {}  # module-level, shared across all requests

    @classmethod
    def for_payload(cls, payload):
        key = hash(json.dumps(payload, sort_keys=True))
        if key not in cls._cache:
            cls._cache[key] = cls(payload)
        return cls._cache[key]

The intent was a memoization cache. The bug: key is a hash of the entire request payload, and payloads are user-supplied and near-unique (they contained a request ID). So every distinct payload added a new entry, and nothing ever evicted. Over 100 requests, 1204 entries — because some payloads repeated. Over a day, hundreds of thousands.

This is the canonical Python long-running-service leak: an unbounded cache keyed on something that's effectively unique per request. You will see it in functools.lru_cache without a maxsize, in @cached_property on objects that outlive the request, in class-level dicts, in functools.cache (which is lru_cache(maxsize=None) — unbounded by design).

The fix

Three options, in order of preference:

# Option A: bound it. lru_cache with a real maxsize.
from functools import lru_cache

class Serializer:
    @staticmethod
    @lru_cache(maxsize=1024)
    def _serialize(payload_json: str):
        return Serializer(json.loads(payload_json))

If the cache key can't be a hashable primitive, use cachetools:

# Option B: TTL + size bound, for caches where entries go stale.
from cachetools import TTLCache
import threading

class Serializer:
    _cache = TTLCache(maxsize=2048, ttl=300)
    _lock = threading.Lock()  # TTLCache is not thread-safe

    @classmethod
    def for_payload(cls, payload):
        key = hash(json.dumps(payload, sort_keys=True))
        with cls._lock:
            if key not in cls._cache:
                cls._cache[key] = cls(payload)
            return cls._cache[key]

Option C — and this is the one people forget — is don't cache at all. If the cache hit rate is low (and it will be, if keys are near-unique), the cache is pure overhead: it costs memory, costs the json.dumps to compute the key, and saves nothing. Profile the hit rate before adding a cache. In our case the hit rate was 0.3%, and removing the cache entirely was the fix. The "optimization" had been added speculatively two years earlier.

Gotchas that will waste your afternoon

tracemalloc.start() must be called before the allocations you want to track. If you start it mid-request, you get a partial view. In production, start it at process boot behind an env flag, or accept that you need to reproduce in staging.

tracemalloc adds ~30-50% overhead and holds tracebacks for every live allocation. Do not leave it on. In a 512 MB worker it will itself push you into OOM. Enable via signal, take snapshots, disable.

gc.get_objects() returns objects tracked by the GC. str, bytes, int, and tuples of untracked types are not tracked. If your leak is a giant str (e.g. accumulating log lines), objgraph will miss it. Use tracemalloc for those.

sys.getsizeof lies about container size. It returns the size of the container object, not its contents. A dict with 100,000 entries reports ~5 MB via getsizeof but holds far more. Use tracemalloc or pympler.asizeof for real numbers.

Reference cycles with __del__. In Python < 3.4 these were uncollectable; modern CPython handles them, but if you have a C extension holding a reference (a common pattern with some ORMs and HTTP clients), gc.collect() won't free it and neither will objgraph. That's when you reach for gc.get_referrers on the suspect object and read the raw gc.garbage list.

objgraph.show_backrefs on a large object graph is slow and can hang. Set max_depth=4 and pass a specific object, not a type.

When this workflow is the wrong tool

If live_objects() is flat and RSS still climbs, you have a C-extension leak or allocator fragmentation. tracemalloc and objgraph are blind to memory allocated by malloc inside a C extension that doesn't use Python's allocators. Check with PYTHONMALLOC=malloc valgrind --tool=memcheck on a short run, or heaptrack for a lower-overhead view. The Python-level tooling will show nothing, and you'll waste hours.

If the leak only manifests under production load and not in your fixture loop, the leak is likely concurrency-dependent — a lock held across an exception path, a queue that grows faster than it drains, a thread-local that never gets cleaned. tracemalloc snapshots from a single thread won't show the other threads' allocations. Use faulthandler.dump_traceback_later() and per-thread snapshots.

And if the service is short-lived (a cron job, a Lambda), stop. A leak that grows 2 MB over a 30-second invocation is not a leak, it's a working set. Don't fix what the process lifetime already bounds.

The one-line version

tracemalloc finds the allocation site; objgraph.show_backrefs finds the referrer. Leaks in long-running Python services are almost always an unbounded cache — a dict, an lru_cache without maxsize, or a functools.cache — keyed on something per-request-unique. Bound it, TTL it, or delete it. Then verify with the same before/after snapshot that found it, because if you don't, you'll be back here in three days.

Docs worth bookmarking: tracemalloc, gc, objgraph, cachetools, functools.lru_cache.