DEV Community

Ayi NEDJIMI
Ayi NEDJIMI

Posted on

How to Profile and Optimize Python Code for 10x Performance

Python is fast enough for most tasks — until it isn't. When a data pipeline that takes 3 seconds starts handling 100x more data, that 3 seconds becomes 300 seconds, and suddenly you have a production incident. The real mistake isn't writing slow Python; it's optimizing blind without first understanding where time is actually going.

Measure Before You Touch Anything

The cardinal rule: never optimize without profiling first. Human intuition about bottlenecks is wrong roughly 80% of the time. The function you think is slow rarely is — the actual culprit is usually somewhere you weren't looking.

Python ships with two useful built-in profilers:

  • cProfile: deterministic, low overhead, good for finding the top time consumers
  • line_profiler: line-by-line timing, surgical precision when you know roughly where the problem is

Start with cProfile:

import cProfile
import pstats
import io
from pstats import SortKey

def run_with_profiling(func, *args, **kwargs):
    pr = cProfile.Profile()
    pr.enable()
    result = func(*args, **kwargs)
    pr.disable()

    stream = io.StringIO()
    stats = pstats.Stats(pr, stream=stream)
    stats.sort_stats(SortKey.CUMULATIVE)
    stats.print_stats(20)
    print(stream.getvalue())
    return result
Enter fullscreen mode Exit fullscreen mode

The cumulative sort shows functions consuming the most total time, including subcalls — more useful than tottime for initial triage. It tells you which call chain is burning time, not just which individual function.

If you want something with less friction, py-spy is a sampling profiler that attaches to a running process without code changes:

pip install py-spy
py-spy top --pid 12345
py-spy record -o profile.svg --pid 12345
Enter fullscreen mode Exit fullscreen mode

The SVG flamegraph makes hot paths immediately obvious.

The Most Common Python Performance Mistakes

Once you have a profile, you're usually staring at one of these patterns:

Quadratic loops: a nested loop over a list of n items is O(n²). Convert the inner lookup to a dict or set.

Unnecessary object creation: string concatenation with += in a loop creates a new string object on every iteration. Use "".join(list_of_parts) instead.

Calling the same function repeatedly inside a loop: if you call len(my_list) inside a loop condition, Python evaluates it every iteration. Cache it outside.

# Before — O(n³) worst case
def find_duplicates(items):
    duplicates = []
    for i in range(len(items)):
        for j in range(len(items)):
            if i != j and items[i] == items[j]:
                if items[i] not in duplicates:
                    duplicates.append(items[i])
    return duplicates

# After — O(n)
def find_duplicates_fast(items):
    seen = set()
    duplicates = set()
    for item in items:
        if item in seen:
            duplicates.add(item)
        seen.add(item)
    return list(duplicates)
Enter fullscreen mode Exit fullscreen mode

On 10,000 items, the difference is seconds vs microseconds. The algorithmic fix always outperforms any amount of micro-optimization on the slow version.

Leveraging Built-ins and Standard Library Tools

CPython's built-in functions run at C speed. Using them instead of pure Python loops is usually the cheapest performance win available, and it requires no dependencies.

Compare sorting:

import timeit

data = list(range(10000, 0, -1))

def bubble_sort(arr):
    n = len(arr)
    for i in range(n):
        for j in range(0, n - i - 1):
            if arr[j] > arr[j + 1]:
                arr[j], arr[j + 1] = arr[j + 1], arr[j]
    return arr

t1 = timeit.timeit(lambda: bubble_sort(data.copy()), number=10)
t2 = timeit.timeit(lambda: sorted(data), number=10)

print(f"Bubble sort: {t1:.3f}s")
print(f"sorted():    {t2:.5f}s")
Enter fullscreen mode Exit fullscreen mode

The result: sorted() runs 1000x faster because Timsort is implemented in C. No surprises there, but the same principle applies in less obvious places:

  • collections.Counter over manual counting dicts
  • itertools.chain over manual list concatenation
  • functools.lru_cache for memoizing expensive pure functions
  • bisect for maintaining sorted lists instead of re-sorting on every insert

NumPy and Vectorization

For numerical work, the biggest lever is vectorization — replacing Python loops with NumPy operations that run in C with SIMD instructions.

import numpy as np
import time

n = 1_000_000
data = list(range(n))

# Pure Python
start = time.perf_counter()
result_py = [x ** 2 for x in data]
py_time = time.perf_counter() - start

# NumPy
arr = np.array(data)
start = time.perf_counter()
result_np = arr ** 2
np_time = time.perf_counter() - start

print(f"Python list comprehension: {py_time:.3f}s")
print(f"NumPy vectorized:          {np_time:.4f}s")
print(f"Speedup: {py_time / np_time:.0f}x")
Enter fullscreen mode Exit fullscreen mode

Typical output: 20-50x speedup. No multiprocessing, no C extensions — just letting NumPy use the CPU efficiently.

For Pandas: avoid iterrows(). It iterates row by row in Python, discarding all the performance advantages of the library. Write the transformation as a column operation directly, or use apply() with a vectorized function.

Concurrency: Threads, Processes, and asyncio

Python's GIL means threads don't help CPU-bound code. But they do help I/O-bound work — API calls, database queries, file reads.

The correct tool depends on the bottleneck:

  • I/O-bound (waiting on network, disk): asyncio or ThreadPoolExecutor
  • CPU-bound (computation): ProcessPoolExecutor
from concurrent.futures import ProcessPoolExecutor, as_completed
import os

def heavy_computation(n):
    factors = []
    for i in range(2, int(n ** 0.5) + 1):
        if n % i == 0:
            factors.append(i)
            factors.append(n // i)
    return sorted(set(factors))

numbers = [999983, 999979, 999961, 999931, 999917] * 20

# Sequential
results_seq = [heavy_computation(n) for n in numbers]

# Parallel across CPU cores
with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
    futures = {executor.submit(heavy_computation, n): n for n in numbers}
    results_par = [f.result() for f in as_completed(futures)]
Enter fullscreen mode Exit fullscreen mode

On a 4-core machine, the parallel version runs 3-4x faster. The overhead of spawning processes makes this worthwhile only for tasks that take at least a few milliseconds each — otherwise the inter-process overhead dominates.

For security tooling specifically, parallelizing I/O-bound work such as scanning a list of URLs against a threat intelligence checklist routinely produces 10-50x throughput improvements with asyncio, since you're almost entirely waiting on network responses rather than computing anything.

The Takeaway

Performance optimization in Python follows a strict order: measure, identify the actual bottleneck, apply the cheapest fix first.

  1. Profile with cProfile or py-spy — find where the time actually goes
  2. Fix algorithmic issues first (O(n²) → O(n) beats every other optimization)
  3. Replace Python loops with built-ins and NumPy where possible
  4. Parallelize I/O-bound work with asyncio, CPU-bound work with ProcessPoolExecutor
  5. Only reach for C extensions (Cython, ctypes) or PyPy when nothing else is enough

Skipping step 1 and jumping straight to parallelism is the most common mistake teams make. You'll add complexity without fixing the real problem. Profile first, every time.


I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Top comments (0)