Python Memory Profiling: tracemalloc, scalene, py-spy for Long-Running Pipelines

python memory profiler is the tool a Python data engineer eventually needs at 3 a.m. when an Airflow task starts getting OOM-killed after two hours of running fine, a streaming Kafka worker's RSS grows 100 MB/hour without any input volume change, a Pandas DataFrame transform mysteriously peaks at 40 GB when the input is only 2 GB, or a FastAPI service's RSS creeps upward over days until the pod restarts itself. Every Python engineer eventually debugs a memory leak; knowing which of the three canonical tools (tracemalloc, scalene, py-spy/memray) fits which failure mode and having the leak-pattern vocabulary in your head is what separates a mid-level operator from a senior one. This guide walks the three tools that cover 95% of memory investigations in production DE pipelines.

The tour walks the five pillars — (1) tracemalloc for in-process allocation snapshots and compare_to() diffs that pinpoint the exact file:line where memory grew, (2) scalene for line-level CPU + memory + GPU profiling with sampling-based low overhead suitable for CI or short prod attaches, (3) py-spy for live inspection of an already-running production process with zero code changes and flame-graph output, plus memray (Bloomberg) for allocation-level live sampling, (4) the eight common memory-leak patterns every Python codebase eventually hits (reference cycles, unbounded caches, pandas chained assignment, SQLAlchemy session growth, Kafka consumer buffers, iterator materialisation, logging with heavy payloads, big pickle load), and (5) production monitoring via psutil.Process().memory_info().rss inline in Airflow / Prefect tasks with alerts on growth trends. Every section ships a Solution-Tail interview answer — code, trace, output, why-this-works with __concept__ underlines.

Practice on SQL library →, SQL optimization drills →, and SQL indexing drills →.

---

On this page

• Why memory profiling matters in 2026
• tracemalloc for allocations
• scalene for CPU + memory
• py-spy for live processes
• Patterns + gotchas + monitoring
• Cheat sheet — memory profiler recipe list
• Frequently asked questions
• Practice on PipeCode

---

1. Why memory profiling matters in 2026

The python memory profiler mental model — measure before optimising, know which tool for which failure mode

The one-sentence invariant: Python memory bugs come in three shapes — leaks (memory grows unbounded until OOM), spikes (peak usage during a specific operation exceeds container limit), and steady-state waste (RSS 3× larger than expected) — and each has its own diagnostic tool: tracemalloc for leaks (snapshot diff pinpoints growing allocations), scalene for spikes (line-level CPU+memory profile shows the hot line), py-spy/memray for live processes (attach to running production without restart). Reaching for the wrong tool wastes hours; reaching for the right one solves most cases in 15 minutes.

Where memory bugs actually show up in DE pipelines.

• Airflow / Prefect tasks OOM after N hours. Steady memory growth; container's 4 GB limit hit at hour 3. Almost always a leak or unbounded cache.
• Streaming Kafka workers grow RSS steadily. aiokafka's poll buffer accumulates if downstream is slow; SQLAlchemy session grows if not scoped; unbounded lru_cache.
• Batch ETL peaks at 40 GB and gets killed. Peak is a spike — usually a materialised DataFrame or list; Polars streaming or chunked processing fixes.
• Jupyter notebook accumulates through cells. Each cell's local variables retained; kernel memory grows until restart. Restart the kernel periodically.
• FastAPI service RSS creeps up over days. Slow leak — often connection pool growth, unbounded cache, or reference cycle.
• CI test suite runs out of memory. Test isolation broken; each test accumulates state; use pytest-forked or memory limits.
• Data platform Kubernetes pod restarts every N hours. Steady RSS growth; alerts on memory-limit approach; use tracemalloc + memray.

The three-tool matrix.

Tool	Use case	Overhead	Attaches to running?
tracemalloc	In-process leak detection via snapshot diff	Moderate (2-5× slowdown)	No (restart needed)
scalene	Full profile (CPU + memory + GPU)	Sampling; low (~5%)	Yes (via CLI attach in newer versions)
py-spy	Live CPU profiling of running process	Very low (< 1%)	Yes
memray	Live memory allocation sampling	Low	Yes

What senior interviewers actually probe.

• Which tool first? py-spy on the live process to see what it's doing; then tracemalloc or memray for allocation-level detail.
• RSS vs heap. RSS is OS view; heap is Python interpreter view; RSS ≥ heap.
• When to force gc. gc.collect() after freeing a large object — occasionally useful; not for correctness.
• How reference cycles form. Two objects referring to each other; GC eventually collects but not immediately. Fix — weakref or restructure.
• Pandas chained assignment. df[cond]["col"] = val creates copies that leak.
• @lru_cache(maxsize=None). Unbounded; grows forever.
• gc.get_stats(). Generation counts; frequent gen-2 collections = cycles suspect.
• objgraph.show_growth(). Object type growth between calls.

The five-step memory triage workflow.

• Step 1 — is it a leak or a spike? Monitor RSS over time. Leak = monotonic growth. Spike = transient peak during one operation.
• Step 2 — attach py-spy to see the running stack. py-spy top --pid X — what functions are hot right now?
• Step 3 — sample allocations with memray or tracemalloc. Find the file:line that grew the most since baseline.
• Step 4 — read the code around that line. Common patterns: list append forever, dict grow forever, closure capturing large data.
• Step 5 — fix + verify. Bounded cache, explicit del, chunked processing, session scoping. Verify with re-profile.

Worked example — the OOM after two hours

Detailed explanation. An Airflow task processes Kafka messages and writes to Postgres. Works fine for 90 minutes; then RSS climbs from 500 MB to 4 GB and the pod is OOM-killed. Every 90 minutes, same crash. What's leaking?

Question. Diagnose without code changes, then fix.

Input. A Python worker running in Kubernetes; you have py-spy installed in the container.

Code — diagnostic.

# Step 1: find the running PID inside the container
kubectl exec -it worker-pod-xyz -- bash
$ ps aux | grep python
python  1234  ...

# Step 2: attach py-spy top to see what's running
$ py-spy top --pid 1234

# Sample output:
%CPU  Function
20%   sqlalchemy.orm.session.Session.query
15%   confluent_kafka.Consumer.poll
10%   pandas.core.frame.DataFrame.__init__

# Step 3: attach memray to sample allocations
$ memray attach --live --pid 1234
# Watch top allocators by bytes; see SQLAlchemy Session accumulating

Code — fix.

# BEFORE — session held across whole task; accumulates
from sqlalchemy.orm import Session
session = Session(engine)
for msg in consumer:
    obj = MyModel(**msg.value)
    session.add(obj)
    if msg.offset % 1000 == 0:
        session.commit()

# AFTER — scoped session per batch; freed each iteration
from sqlalchemy.orm import Session
batch = []
for msg in consumer:
    batch.append(msg.value)
    if len(batch) >= 1000:
        with Session(engine) as session:   # scoped
            session.execute(insert(MyModel), batch)
            session.commit()
        batch.clear()

Step-by-step explanation.

1. py-spy top — shows SQLAlchemy's Session.query is hot.
2. memray attach --live — shows Session identity-map bytes growing linearly with time.
3. Root cause — long-lived session accumulates every added-but-not-flushed object.
4. Fix — scoped session per batch via with Session(engine) context.
5. Post-fix: RSS stays flat at ~500 MB indefinitely.

Output.

Metric	Before	After
RSS after 2 hours	4 GB (OOM)	500 MB (stable)
Time to detect	15 min	—
Time to fix	5 min	—

Rule of thumb. Long-lived SQLAlchemy Session = leak. Scope to batch.

Worked example — the peak during Pandas transform

Detailed explanation. A batch ETL peaks at 40 GB RSS while transforming a 2 GB DataFrame — a 20× amplification. Not a leak (RSS drops after); a spike during a specific transform.

Question. Diagnose the peak.

Code — diagnostic.

# Restart the process with scalene attached
$ scalene --html --outfile profile.html etl_script.py

# Open profile.html — look for lines with high MEM% column

Sample scalene output.

% CPU   NET MEM   CODE
 15%    +2 GB   df = pd.read_csv("in.csv")
 40%   +38 GB   df["combined"] = df["a"] + df["b"] + df["c"]   # 20× copies
 20%    +1 GB   df.to_parquet("out.parquet")

Fix — the 20× amplification came from chained temp DataFrames.

# BEFORE — allocates temp for each intermediate
df["combined"] = df["a"] + df["b"] + df["c"]

# AFTER — Polars streams; no temp DataFrames
import polars as pl
(
    pl.scan_csv("in.csv")
      .with_columns((pl.col("a") + pl.col("b") + pl.col("c")).alias("combined"))
      .sink_parquet("out.parquet")
)

Output.

Approach	Peak RSS	Wall clock
Pandas chained	40 GB	4 min
Polars streaming	3 GB	2 min

Rule of thumb. Pandas peaks come from intermediate DataFrames. Polars streaming or careful .copy() avoidance fixes.

Worked example — reference cycle with closure

Detailed explanation. A closure captures a large DataFrame; the closure is stored in a list that's retained; the DataFrame can't be freed even after del.

Code — the bug.

def make_callback(df):
    def callback(x):
        return df.loc[x]   # closure captures df
    return callback

callbacks = []
for i in range(1000):
    df = pd.DataFrame({"x": range(1_000_000)})   # 8 MB
    callbacks.append(make_callback(df))

# 1000 DataFrames × 8 MB = 8 GB captured

Fix — pass by copy or use weakref.

def make_callback(row_dict):   # smaller capture
    def callback(x):
        return row_dict[x]
    return callback

callbacks = []
for i in range(1000):
    df = pd.DataFrame({"x": range(1_000_000)})
    row_dict = df.set_index("x").to_dict()["x"]   # extract needed data only
    callbacks.append(make_callback(row_dict))

Rule of thumb. Closures capture. Extract only what the closure actually uses.

Common beginner mistakes

• Guessing without profiling — always measure.
• Assuming del obj frees memory immediately — refcount may hold.
• gc.collect() as a fix — masks the root cause.
• Not distinguishing leak vs spike vs steady-state.
• Blaming Python garbage collector — usually your code, not the GC.
• Increasing memory limit as fix — postpones OOM, doesn't solve.

python memory profiler interview question on triage workflow

A senior interviewer often opens with: "You get paged: an Airflow worker's RSS is climbing at 100 MB/hour. Walk me through your triage."

Solution Using the 5-step triage workflow

# Step 1: confirm leak vs spike
kubectl exec -it airflow-worker -- bash
$ for i in $(seq 10); do ps -o rss= -p 1234; sleep 60; done
# If monotonically growing → leak.

# Step 2: attach py-spy top
$ py-spy top --pid 1234
# See what's hot right now.

# Step 3: attach memray for allocations
$ memray attach --live --pid 1234
# Sort by bytes; find growing type.

# Step 4: correlate to code
# Which module owns that type? What creates them?

# Step 5: fix + verify
# Deploy patch; monitor for 24h; confirm flat RSS.

Step-by-step trace.

Step	Command	Signal
1	ps monitoring	Leak vs spike
2	py-spy top	Hot code path
3	memray	Type + bytes growing
4	code review	Root cause
5	patch + monitor	Confirmed fix

Output: Structured triage; average time to root cause 20-40 min.

Why this works — concept by concept:

• ps monitoring — cheapest first check; distinguishes leak from spike.
• py-spy top on live PID — zero code changes; no restart; shows current work.
• memray for allocations — reveals what type is growing.
• Code review with the signal — narrow search space; usually one specific pattern.
• 24h re-monitor — confirms fix on production behavior.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL Topic — SQL SQL practice library

Practice →

---

2. tracemalloc for allocations

tracemalloc.take_snapshot() + compare_to() — the built-in snapshot-diff tool that pinpoints the leak's file:line

The mental model in one line: tracemalloc is a built-in Python module that instruments the memory allocator to record every allocation's traceback; take a snapshot before the suspicious operation, take another after, and .compare_to(...) produces a sorted list of file:line locations by size difference — the top entry is almost always the leak's origin. Overhead is 2-5× slowdown, so you use it in dev / CI / debug attaches, not in production hot paths.

Slot 1 — enable + snapshot.

import tracemalloc

tracemalloc.start(25)   # 25 = traceback depth

# ... run workload ...
snapshot1 = tracemalloc.take_snapshot()

# ... more work ...
snapshot2 = tracemalloc.take_snapshot()

# Diff
top_diff = snapshot2.compare_to(snapshot1, 'lineno')
for stat in top_diff[:10]:
    print(stat)

Slot 2 — output format.

/path/to/mymodule.py:42: size=45.3 MiB (+40.1 MiB), count=100 (+95), average=463 KiB
/path/to/other.py:15: size=12.1 MiB (+8.2 MiB), count=50 (+42), average=248 KiB

• size = current total.
• (+40.1 MiB) = change since snapshot1.
• count = number of live allocations.
• average = average size per allocation.

Slot 3 — group by traceback.

top_diff = snapshot2.compare_to(snapshot1, 'traceback')
for stat in top_diff[:5]:
    print(stat.traceback.format())

Shows the full stack that allocated; often more helpful than file:line alone.

Slot 4 — filter.

snapshot = snapshot.filter_traces([
    tracemalloc.Filter(True, "*/mypackage/*"),
    tracemalloc.Filter(False, "*/pytest/*"),
])

• True = include; False = exclude.
• Focuses the report on your code, hiding stdlib and test framework noise.

Slot 5 — get current stats without diff.

snapshot = tracemalloc.take_snapshot()
top = snapshot.statistics('lineno')
for stat in top[:10]:
    print(stat)

Shows all live allocations by file:line.

Slot 6 — memory usage totals.

current, peak = tracemalloc.get_traced_memory()
print(f"Current: {current/1024/1024:.1f} MB")
print(f"Peak: {peak/1024/1024:.1f} MB")

Slot 7 — reset peak counter.

tracemalloc.reset_peak()
# ... work ...
current, peak = tracemalloc.get_traced_memory()   # peak resets from here

Slot 8 — pytest integration.

# pytest fixture
import tracemalloc, pytest

@pytest.fixture
def track_memory():
    tracemalloc.start()
    before = tracemalloc.take_snapshot()
    yield
    after = tracemalloc.take_snapshot()
    diff = after.compare_to(before, 'lineno')
    for s in diff[:5]:
        print(s)
    tracemalloc.stop()

def test_no_leak(track_memory):
    process_batch()

Slot 9 — overhead.

• Depth 1: ~2× slowdown, ~1.5× memory overhead.
• Depth 25 (default when enabling explicit): ~3-5× slowdown, ~2× memory overhead.
• Don't leave enabled in production hot paths.

Slot 10 — when tracemalloc alone isn't enough.

• Very fine-grained allocations (billions of small objects) — objgraph or guppy3 for object-type view.
• Native/C extension allocations — tracemalloc misses these; use memray.
• Production live attach — tracemalloc requires restart; memray attaches live.

Common beginner mistakes

• Forgetting tracemalloc.start() — no data.
• Using tracemalloc in production hot paths — 5× slowdown.
• Small snapshot depth — misses deeper stack context.
• Not filtering — noise from stdlib/tests dominates.
• Assuming tracemalloc catches C-extension leaks — it doesn't; use memray.

Worked example — finding the growing list

Detailed explanation. A worker appends to a list forever without pruning. tracemalloc's snapshot diff pinpoints the file:line.

Question. Reproduce + diagnose + fix.

Code — bug.

# leaky.py
import tracemalloc

_cache = []   # module-global; never pruned

def process(x):
    _cache.append(x)   # LEAK
    return x * 2

def main():
    tracemalloc.start()
    snapshot1 = tracemalloc.take_snapshot()
    for i in range(1_000_000):
        process(i)
    snapshot2 = tracemalloc.take_snapshot()
    diff = snapshot2.compare_to(snapshot1, 'lineno')
    for s in diff[:5]:
        print(s)

Output.

leaky.py:7: size=40.2 MiB (+40.2 MiB), count=1000000 (+1000000)

The tuple (file:7, +40 MB, +1M allocations) pinpoints line 7 as the source. The _cache.append is unmistakable.

Fix.

from functools import lru_cache

@lru_cache(maxsize=1024)   # BOUNDED
def process(x):
    return x * 2

Rule of thumb. Snapshot diff + top-N report finds most leaks in under a minute of analysis.

Worked example — filtering to your own package

Code.

snapshot = tracemalloc.take_snapshot()
snapshot = snapshot.filter_traces([
    tracemalloc.Filter(True, "*/mypipeline/*"),
    tracemalloc.Filter(False, "*/pandas/*"),
    tracemalloc.Filter(False, "*/numpy/*"),
])
for s in snapshot.statistics('lineno')[:10]:
    print(s)

Rule of thumb. Always filter to your code before reading the report — reduces noise 10-100×.

Worked example — peak vs current

Code.

import tracemalloc
tracemalloc.start()

# Baseline
tracemalloc.reset_peak()
huge_temp = list(range(50_000_000))   # 400 MB
del huge_temp

current, peak = tracemalloc.get_traced_memory()
print(f"Current: {current/1024**2:.1f} MB")   # small (temp freed)
print(f"Peak: {peak/1024**2:.1f} MB")          # ~400 MB

Rule of thumb. Peak reveals transient allocations that current doesn't. Use both.

tracemalloc interview question on diagnosing a slow leak

A senior interviewer asks: "A worker's RSS grows 50 MB per hour. In dev, you can't reproduce. Show me how you'd instrument to catch it in staging."

Solution Using periodic snapshots + logging

import tracemalloc, time, threading, logging

tracemalloc.start(15)
_baseline = None

def snapshot_loop():
    global _baseline
    _baseline = tracemalloc.take_snapshot()
    while True:
        time.sleep(300)   # every 5 min
        snap = tracemalloc.take_snapshot()
        diff = snap.compare_to(_baseline, 'lineno')
        top = diff[:5]
        logging.info("Memory diff top 5:")
        for s in top:
            logging.info(f"  {s}")
        # Rotate baseline periodically to catch new leaks
        if time.time() % 3600 < 300:   # once per hour
            _baseline = snap

threading.Thread(target=snapshot_loop, daemon=True).start()

Why this works — concept by concept:

• Background thread — snapshots don't interfere with main work.
• 5-minute interval — catches slow leaks without spamming logs.
• Baseline rotation — hourly refresh so old leaks don't dominate.
• Top 5 diff — focuses on biggest growers.
• Logged to structured logs — searchable in production.
• Cost — tracemalloc overhead ~2-3×; acceptable for staging.

SQL
Topic — optimization
SQL optimization drills

Practice →

---

3. scalene for CPU + memory

scalene — line-level CPU + memory + GPU profile with sampling; the "why is this line hot" tool

The mental model in one line: scalene samples the running process periodically to build a line-by-line report of CPU time, memory allocation, and GPU usage — the killer feature is that it distinguishes native (C library) memory from Python-object memory, and shows a per-line percentage of both, so you can see df["combined"] = df["a"] + df["b"] allocated 38 GB in a single line and CPU-was-40%-hot on the same line, giving you both bottlenecks at once.

Slot 1 — basic run.

pip install scalene
scalene my_script.py

Opens interactive HTML report on completion.

Slot 2 — output columns.

• % CPU (Python) — time spent in Python code.
• % CPU (native) — time in C extensions (NumPy, Pandas, etc.).
• % CPU (system) — kernel time (I/O syscalls).
• MEM (%) — memory allocated at this line as % of total.
• NET MEM (avg) — net memory change per call (allocation minus deallocation).
• PEAK MEM — peak during this line.
• GPU % — GPU utilisation (if CUDA available).

Slot 3 — HTML output.

scalene --html --outfile profile.html my_script.py

• Sortable table.
• Line-by-line source annotation.
• Sparklines for time-series.

Slot 4 — command-line options.

• scalene --cpu-only script.py — skip memory profile.
• scalene --memory-only script.py — skip CPU.
• scalene --gpu script.py — include GPU.
• scalene --profile-only mypackage script.py — filter to your package.
• scalene --outfile out.json --json script.py — machine-readable output.

Slot 5 — CI integration.

# In CI
scalene --json --outfile perf.json --profile-all pytest tests/

# Check that no line exceeds threshold
python -c "import json; d = json.load(open('perf.json')); ..."

Slot 6 — memory attribution.

• Python-object memory — attributed to line that created the object.
• Native memory (NumPy arrays, Pandas DataFrames) — attributed to line that called the constructor.
• This distinguishes — a NumPy array allocated in one line vs Python objects created elsewhere.

Slot 7 — sampling vs deterministic.

• Scalene uses sampling — negligible overhead (< 5%).
• Doesn't capture 100% of allocations; but captures the important ones.
• Deterministic profilers (cProfile) are slower and don't include memory.

Slot 8 — comparison with cProfile.

Tool	CPU	Memory	Overhead
cProfile	✅	❌	~30%
scalene	✅	✅	~5%
py-spy	✅	❌	< 1%
memray	❌	✅	~5%
tracemalloc	❌	✅	~100-500%

Slot 9 — hot-path finding workflow.

• Run scalene on the workload.
• Sort by % CPU (Python) descending.
• Top 5 lines are the CPU bottlenecks.
• Sort by NET MEM descending.
• Top 5 lines are the memory bottlenecks.
• Cross-reference — a line hot on both is the highest-leverage optimisation.

Slot 10 — GPU profiling.

• Requires CUDA + nvidia-ml-py.
• scalene --gpu enables.
• Attributes GPU time to Python line that launched the kernel.

Common beginner mistakes

• Not filtering to own package — noise from Python stdlib dominates.
• Sorting by wrong column — memory vs CPU are different axes.
• Running scalene in production — meant for dev / CI attach.
• Ignoring native memory — Pandas DataFrames show up under native.
• Missing GPU option — CUDA workloads underprofiled.

Worked example — finding the CPU-hot line

Code being profiled.

# etl.py
import pandas as pd

def main():
    df = pd.read_csv("data.csv")   # 500 MB CSV
    for i in range(100):
        df["c"] = df["a"] + df["b"]   # in a loop — 100× redundant
    df.to_parquet("out.parquet")

if __name__ == "__main__":
    main()

Scalene output (simplified).

Line 4:  15%  Python    +500 MB     df = pd.read_csv("data.csv")
Line 5:   1%  Python       0 MB     for i in range(100):
Line 6:  80%  native   ~50 GB peak   df["c"] = df["a"] + df["b"]
Line 7:   4%  Python    +50 MB      df.to_parquet("out.parquet")

Line 6 dominates both CPU (80%) and memory (peak 50 GB). Fix — move outside the loop.

Rule of thumb. Scalene surfaces the hot line in one glance; usually a fix is obvious once found.

Worked example — native vs Python memory

Code.

import numpy as np
big = np.zeros((10000, 10000), dtype=np.float64)   # 800 MB — native
py_list = list(range(10_000_000))                   # 80 MB — Python

Scalene shows big under native (NumPy) and py_list under Python — helpful when tuning memory-heavy workloads.

Rule of thumb. Native memory is often larger; check both axes.

Worked example — CI performance gate

Bash.

scalene --json --outfile ci_profile.json my_pipeline.py
python -c "
import json
data = json.load(open('ci_profile.json'))
peak_mb = data['peak_memory_bytes'] / 1024**2
if peak_mb > 4000:
    print(f'FAIL: peak memory {peak_mb} MB > 4000 MB limit')
    exit(1)
"

Rule of thumb. Automated regression check — CI fails if peak memory or CPU time crosses a threshold.

scalene interview question on memory attribution

A senior interviewer asks: "Show me how you'd use scalene to distinguish a memory bug in your Python code vs a bug in a NumPy operation."

Solution Using scalene's native vs Python breakdown

# Script
import numpy as np
import pandas as pd

def user_bug():
    huge = []
    for i in range(1_000_000):
        huge.append({"i": i, "v": [0.0] * 100})   # Python-object leak
    return huge

def numpy_ok():
    return np.zeros((10000, 10000))   # Native NumPy allocation

# scalene shows:
# user_bug: 40% Python memory + 40% CPU Python
# numpy_ok: 20% native memory + 5% CPU native

Why this works — concept by concept:

• Python-column shows list/dict allocations — user_bug is 40% Python-object memory.
• Native-column shows NumPy allocations — numpy_ok is under native.
• CPU breakdown — Python CPU vs native (NumPy release GIL).
• Combined — identifies user_bug as the leak (Python column high); numpy_ok is intentional.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — optimization SQL optimization drills

Practice →

---

4. py-spy for live processes

py-spy top --pid X + py-spy record — attach to a running Python process with zero code changes, get flame graphs

The mental model in one line: py-spy is a Rust-based sampling profiler that attaches to an already-running Python process via OS-level introspection (no code changes, no restart, no library import) and produces either a real-time top-view of hot functions or a flame graph SVG — it's the tool you reach for first when a production process is misbehaving and you need to see what it's doing without disturbing it.

Slot 1 — installation.

pip install py-spy
# Or via cargo (Rust): cargo install py-spy

Slot 2 — top view (live).

py-spy top --pid 12345

• Terminal top-view of hot functions.
• Updates in real time.
• Shows Python function names, files, %CPU, samples.

Slot 3 — flame graph.

py-spy record -o profile.svg --pid 12345 --duration 60

• 60-second recording.
• Output SVG opens in browser.
• Shows time distribution as a hierarchical flame graph.

Slot 4 — stack dump.

py-spy dump --pid 12345

• One-shot snapshot of every thread's stack.
• Useful when process is hung; shows exactly where.

Slot 5 — Python native features.

• --native — include C code frames (requires debug symbols).
• --gil — highlight GIL-holding frames.
• --threads — per-thread breakdown.
• --subprocesses — follow child processes.

Slot 6 — docker / kubernetes attach.

# Inside container (SYS_PTRACE capability needed)
kubectl exec -it pod-name -- py-spy top --pid 1

# Or, run py-spy as sidecar with hostPID

Slot 7 — comparison with cProfile / scalene.

Tool	Overhead	Live attach	Memory profile
cProfile	~30%	No	No
scalene	~5%	Yes (newer)	Yes
py-spy	< 1%	Yes	No
memray	~5%	Yes	Yes

Slot 8 — memray for live memory.

py-spy doesn't do memory; memray does:

pip install memray
memray attach --live --pid 12345

• Live top view of allocations.
• Attaches to running process.

Slot 9 — flame graph reading.

• Y-axis: stack depth.
• X-axis: time spent (width = %).
• Wider = hotter.
• Look for wide plateaus — that's your bottleneck.
• Icicle vs flame (top-down vs bottom-up) — same data, different visualisation.

Slot 10 — automating in CI or alerts.

# CI: profile a test run
py-spy record -o test-profile.svg -- pytest tests/

# Alert: attach when memory is high
if [ $(ps -o rss= -p $PID) -gt 3000000 ]; then
    py-spy dump --pid $PID > /tmp/high-mem-stack.txt
fi

Common beginner mistakes

• Missing ptrace capability in container — SYS_PTRACE needed.
• Attaching to wrong PID — verify with ps first.
• Not recording long enough — 5 sec sample may miss the bottleneck.
• Flame graph interpretation — width matters, height doesn't.
• Using py-spy when memory is the issue — use memray instead.

Worked example — attach to production for live diagnostics

Question. A production worker is CPU-pegged. Attach py-spy.

Code.

kubectl exec -it worker-abc -- bash
$ py-spy top --pid 1

# Output (updates live):
Collecting samples from '/usr/bin/python worker.py' (python v3.11.6)
Total Samples 500

%Own   %Total  OwnTime  TotalTime  Function (filename)
40.0%  40.0%    2.00s     2.00s   process_event (worker.py:42)
30.0%  30.0%    1.50s     1.50s   json.loads (json/__init__.py:346)
20.0%  20.0%    1.00s     1.00s   dumps_dict (worker.py:78)

process_event and json.loads are the CPU bottlenecks. dumps_dict third. Fix — batch JSON parsing or use orjson.

Rule of thumb. py-spy top for triage — see what's hot in 30 seconds.

Worked example — flame graph for a slow request

Code.

# Attach for 60 seconds while a slow request is in progress
py-spy record -o slow-request.svg --pid 12345 --duration 60

# Open in browser
open slow-request.svg

Flame graph shows entire call tree; wide bars at the top are the hot leaves.

Rule of thumb. Flame graph for detailed analysis; top for quick triage.

Worked example — hung process

Code.

$ py-spy dump --pid 12345

# Output:
Thread 12345 (idle): "MainThread"
    _wait (threading.py:346)
    wait (threading.py:626)
    join (worker.py:100)
    main (worker.py:200)

Process stuck in Thread.wait. Look for deadlock — is there a paired notify?

Rule of thumb. py-spy dump on hung process reveals the exact deadlock.

py-spy interview question on triaging a memory + CPU mystery

A senior interviewer asks: "Your worker pod has 90% CPU and 3.9 GB RSS on a 4 GB limit. Explain your triage sequence using py-spy and memray."

Solution Using py-spy for CPU then memray for memory

# Step 1: attach py-spy top
kubectl exec -it worker -- py-spy top --pid 1
# See CPU hot path — likely a tight loop.

# Step 2: py-spy record for detailed flame
py-spy record -o flame.svg --pid 1 --duration 30

# Step 3: attach memray for allocations
kubectl exec -it worker -- memray attach --live --pid 1
# See top allocators by bytes.

# Step 4: correlate
# CPU hot in loop that also allocates big; that's the target.

# Step 5: patch — batch, cache, or rewrite.

Why this works — concept by concept:

• py-spy for CPU (live, zero overhead) — first tool because it's cheap.
• memray for memory (live, low overhead) — second tool for allocation data.
• Both live-attach — no restart needed.
• Correlate — line hot on both is highest-leverage fix.
• Cost — < 2% overhead combined; safe for production.

SQL
Topic — SQL
SQL practice library

Practice →

SQL Topic — optimization SQL optimization drills

Practice →

---

5. Patterns + gotchas + monitoring

The eight common memory leak patterns + psutil monitoring inline in tasks

The mental model in one line: most Python memory leaks in DE pipelines fall into eight recognisable patterns — reference cycles, unbounded caches, pandas chained assignment, SQLAlchemy session growth, Kafka consumer buffers, iterator materialisation, logging with heavy payloads, big pickle deserialisation — and every senior operator recognises them on sight and applies the specific fix (weakref, lru_cache(maxsize=N), .loc[], scoped session, backpressure, streaming iteration, log summaries, chunked reads); combine with psutil.Process().memory_info().rss sampled inline in every Airflow / Prefect task for early alerting.

Slot 1 — reference cycles.

# Bug — two objects reference each other; GC eventually collects but not immediately
class Node:
    def __init__(self):
        self.child = None
        self.parent = None

a = Node()
b = Node()
a.child = b
b.parent = a   # cycle

• Fix — weakref.proxy(a) for the back-reference.
• Or restructure: children own parents but not vice versa.
• Detect: gc.get_objects() + type counts.

Slot 2 — unbounded caches.

# Bug — grows forever
_cache = {}
def get(key):
    if key not in _cache:
        _cache[key] = expensive(key)
    return _cache[key]

# Fix
from functools import lru_cache
@lru_cache(maxsize=1024)
def get(key):
    return expensive(key)

Slot 3 — pandas chained assignment.

# Bug — creates copies that leak
df[df.x > 0]["y"] = 1

# Fix
df.loc[df.x > 0, "y"] = 1

Slot 4 — SQLAlchemy session growth.

# Bug — session held forever; identity map grows
session = Session(engine)
for msg in stream:
    session.add(build(msg))
    session.commit()

# Fix — scoped session per batch
def process_batch(msgs):
    with Session(engine) as session:
        session.add_all([build(m) for m in msgs])
        session.commit()

Slot 5 — Kafka consumer buffers.

# Bug — poll returns big batches; process is slow; buffer grows
consumer = KafkaConsumer(...)
for msg in consumer:   # iterator; consumer poll buffer accumulates
    slow_process(msg)   # 10 ms

# Fix — bounded queue + backpressure
q = queue.Queue(maxsize=1000)
def consume():
    for msg in consumer:
        q.put(msg)   # blocks if queue full
def process():
    while True:
        msg = q.get()
        slow_process(msg)

Slot 6 — iterator materialisation.

# Bug — materialises 10 M rows
rows = list(cursor.fetchall())   # all in RAM

# Fix — iterate directly
for row in cursor:   # streaming
    process(row)

Slot 7 — logging with heavy payloads.

# Bug — log format keeps reference
logger.info(f"Processed: {big_dataframe}")   # holds df reference

# Fix — log summaries
logger.info(f"Processed rows={len(big_dataframe)} cols={list(big_dataframe.columns)}")

Slot 8 — big pickle deserialisation.

# Bug — materialises full pickle
with open("big.pkl", "rb") as f:
    data = pickle.load(f)   # 10 GB in RAM

# Fix — chunked pickle, or use parquet
import pyarrow.parquet as pq
for batch in pq.ParquetFile("big.parquet").iter_batches(batch_size=100000):
    process(batch)

Slot 9 — psutil inline monitoring.

import psutil, os

def get_rss_mb():
    return psutil.Process(os.getpid()).memory_info().rss / (1024 ** 2)

# In Airflow task
from airflow.decorators import task

@task
def etl_task():
    logger.info(f"RSS at start: {get_rss_mb():.1f} MB")
    df = load_data()
    logger.info(f"RSS after load: {get_rss_mb():.1f} MB")
    transformed = transform(df)
    logger.info(f"RSS after transform: {get_rss_mb():.1f} MB")
    save(transformed)
    logger.info(f"RSS after save: {get_rss_mb():.1f} MB")

Slot 10 — production alerts.

• Alert on RSS > 80% of container limit.
• Alert on RSS growth rate > 100 MB/hour.
• Alert on OOM-kill event.
• Pageduty / Slack integration via Prometheus + Alertmanager.

Common beginner mistakes

• gc.collect() as a fix — masks root cause.
• Increasing container limit as fix — postpones OOM.
• Ignoring log format holds — big objects retained by logger.
• Not scoping SQLAlchemy sessions.
• Materialising iterators (list, tuple) without need.

Worked example — building a memory-monitoring context manager

Code.

import psutil, os, contextlib, logging

@contextlib.contextmanager
def memory_track(label: str):
    proc = psutil.Process(os.getpid())
    rss_start = proc.memory_info().rss
    try:
        yield
    finally:
        rss_end = proc.memory_info().rss
        delta_mb = (rss_end - rss_start) / (1024 ** 2)
        logging.info(f"[MEMORY] {label}: delta={delta_mb:+.1f} MB (now {rss_end/(1024**2):.0f} MB)")

# Use in any pipeline step
with memory_track("load_kafka_batch"):
    batch = poll_kafka()

with memory_track("transform"):
    result = transform(batch)

with memory_track("write_postgres"):
    write(result)

Rule of thumb. Every pipeline task should wrap major steps in memory_track.

Worked example — Airflow task memory alarm

Code.

from airflow.decorators import task
from airflow.exceptions import AirflowFailException

@task
def etl_with_limit():
    proc = psutil.Process(os.getpid())
    for chunk in load_chunks():
        result = transform(chunk)
        write(result)
        rss_mb = proc.memory_info().rss / (1024 ** 2)
        if rss_mb > 3500:   # 87.5% of 4 GB
            raise AirflowFailException(f"Memory too high: {rss_mb:.0f} MB")

Rule of thumb. Fail fast at 85% of limit — better a caught fail than OOM-kill (which loses task state).

Worked example — objgraph for type-level growth

Code.

import objgraph

# Baseline
objgraph.show_growth(limit=10)

# ... run workload ...

# See what grew
objgraph.show_growth(limit=10)
# Output:
# dict           +50000
# list           +10000
# str            +5000
# UserRow        +100000   <- suspect

Rule of thumb. objgraph shows type-level growth; complements tracemalloc's file:line view.

python memory profiler interview question on choosing the tool

A senior interviewer asks: "You have three memory issues: (1) worker RSS climbs 100 MB/hour over 10 days, (2) batch job peaks at 40 GB briefly, (3) hard-to-reproduce production OOM once a week. Which tool for each?"

Solution Using tool-per-shape matrix

Issue	Tool	Reason
Slow leak (100 MB/hr)	tracemalloc periodic snapshots + logging	Long-running; need diff over time
Peak spike (40 GB)	scalene on batch run	Line-level attribution needed
Rare production OOM	memray attach + core dump	Attach when high; capture pre-OOM stack

Why this works — concept by concept:

• tracemalloc for slow leaks — periodic snapshots build a growth timeline.
• scalene for spikes — line-level shows the offending transform.
• memray for rare events — live attach + core dump on OOM.
• psutil monitoring in all — early detection triggers alert-based deeper profiling.

SQL
Topic — optimization
SQL optimization drills

Practice →

SQL
Topic — indexing
SQL indexing drills

Practice →

---

Cheat sheet — memory profiler recipe list

• psutil.Process().memory_info().rss — OS-view RSS in bytes.
• tracemalloc.start(depth) — enable allocation tracking.
• snap = tracemalloc.take_snapshot() — capture.
• diff = snap2.compare_to(snap1, 'lineno') — top growers.
• snap.filter_traces([Filter(True, '*/mypkg/*')]) — narrow scope.
• current, peak = tracemalloc.get_traced_memory() — totals.
• scalene script.py — line-level CPU + memory.
• scalene --html --outfile out.html — HTML report.
• scalene --profile-only mypkg — filter to your code.
• py-spy top --pid X — live CPU top.
• py-spy record -o flame.svg --pid X — flame graph.
• py-spy dump --pid X — one-shot stack dump.
• memray attach --live --pid X — live memory sampling.
• memray run --output out.bin script.py — recorded profile.
• objgraph.show_growth() — type-level growth.
• gc.get_objects() — all live objects.
• gc.get_stats() — GC generation counts.
• weakref.proxy(obj) — break reference cycles.
• @lru_cache(maxsize=N) — bounded cache.
• .loc[...] — non-chained pandas assign.
• Scoped SQLAlchemy session — with Session(engine).
• Bounded Queue — backpressure.
• for row in cursor: — streaming (not list(fetchall())).
• Log summaries — never log big objects.
• Parquet iter_batches — chunked reads instead of pickle load.
• Airflow task guards — check RSS mid-task; fail early.
• Alert on 80% container limit.

Frequently asked questions

Which memory profiler should I reach for first?

Start with py-spy top --pid X on the live process to see what it's doing at CPU level. Then memray attach --live --pid X for allocation-level detail. Both attach without restart or code changes — safest on production. Only if you can restart the process, use tracemalloc.start() in the code for snapshot-diff analysis, or scalene script.py for a full CPU+memory line-level profile. Rule of thumb — py-spy for triage (30 sec), memray for allocations (5 min), scalene for planned analysis (needs restart), tracemalloc for deep dive with snapshots.

How do I profile an already-running Airflow task?

Exec into the worker container: kubectl exec -it worker -- py-spy dump --pid $(pgrep -f airflow_task_id) gives you the stack. For live memory, memray attach --pid X. If you need Airflow-level metrics, log psutil.Process().memory_info().rss at task start/end in your task code — Airflow captures those logs. For deeper analysis, restart the task locally with tracemalloc or scalene on the same data.

What's the difference between RSS and heap?

RSS (Resident Set Size) is the OS-level view — the physical memory pages held by the process. Python heap is what the interpreter has allocated via PyObject_Malloc internally. RSS can be higher than heap (page-cache, fragmentation, C libraries) or occasionally lower (swap). For debugging, focus on RSS growth as the leak signal — that's what the container's OOM killer measures. Python's sys.getsizeof(obj) gives object-level size but doesn't include referenced objects.

How do I fix a pandas memory leak?

Check for chained assignment (df[cond][col] = val instead of df.loc[cond, col] = val) — creates copies that don't free predictably. Verify you're not holding old DataFrame references in closures or globals. Use del df; gc.collect() explicitly between large transforms. For large data, migrate to Polars or DuckDB. Rule — Pandas leaks usually come from chained ops or long-held references; both fixable with discipline.

Do I need to explicitly call gc.collect()?

Rarely for correctness — Python's garbage collector runs automatically. Occasionally useful after freeing a huge DataFrame to force immediate RSS reclaim. In long-running services, monitor gc.get_stats() — high generation-2 collection frequency suggests reference cycles worth investigating. Rule — gc.collect() as a diagnostic tool (see if RSS drops immediately) not as a fix.

How much overhead does tracemalloc add?

Depth 1: ~2× slowdown, ~1.5× memory overhead. Depth 25 (typical for good tracebacks): ~3-5× slowdown, ~2× memory overhead. Fine for dev runs and short prod attaches; not for production hot paths. Use scalene (< 5% overhead) or memray (~5% overhead) for lower-cost long attaches. Rule — tracemalloc for debugging; scalene / memray for continuous monitoring.

Practice on PipeCode

• Drill the SQL practice library → — 450+ DE-focused questions covering resource-conscious pipeline design.
• Sharpen SQL optimization drills → — memory-efficient query patterns.
• Layer SQL indexing drills → — reducing IO reduces memory.
• Warm up with SQL join drills → — join memory footprint analysis.
• For the broader SQL interview surface, take the SQL for Data Engineering course →.

Pipecode.ai is Leetcode for Data Engineering — every `python memory profiler` recipe above ships with hands-on practice rooms where you attach `py-spy top` to a running Airflow worker without code changes, use `tracemalloc.take_snapshot().compare_to()` to diff two moments and pinpoint the leaking file:line, walk `scalene`'s line-level CPU+memory annotated HTML output to find the offending transform, wrap pipeline steps in a `psutil` memory-track context manager, and finally fix the eight canonical leak patterns from reference cycles to unbounded caches to pandas chained assignment — the exact memory-profiling fluency that senior Python DE interviews probe.

Practice SQL now →
Optimization drills →

Comments

[Luis]

I particularly appreciated the section on tracemalloc for in-process allocation snapshots, as it's often the first step in identifying memory leaks in long-running pipelines. The example use of compare_to() to pinpoint the exact file and line where memory grew is especially useful, as it can save a lot of time in debugging. Have you found that using tracemalloc in conjunction with py-spy or scalene provides a more comprehensive understanding of memory usage patterns, or are there specific scenarios where one tool is more suitable than the others?