Python is fast enough for most tasks — until it isn't. When a data pipeline that takes 3 seconds starts handling 100x more data, that 3 seconds becomes 300 seconds, and suddenly you have a production incident. The real mistake isn't writing slow Python; it's optimizing blind without first understanding where time is actually going.
Measure Before You Touch Anything
The cardinal rule: never optimize without profiling first. Human intuition about bottlenecks is wrong roughly 80% of the time. The function you think is slow rarely is — the actual culprit is usually somewhere you weren't looking.
Python ships with two useful built-in profilers:
-
cProfile: deterministic, low overhead, good for finding the top time consumers -
line_profiler: line-by-line timing, surgical precision when you know roughly where the problem is
Start with cProfile:
import cProfile
import pstats
import io
from pstats import SortKey
def run_with_profiling(func, *args, **kwargs):
pr = cProfile.Profile()
pr.enable()
result = func(*args, **kwargs)
pr.disable()
stream = io.StringIO()
stats = pstats.Stats(pr, stream=stream)
stats.sort_stats(SortKey.CUMULATIVE)
stats.print_stats(20)
print(stream.getvalue())
return result
The cumulative sort shows functions consuming the most total time, including subcalls — more useful than tottime for initial triage. It tells you which call chain is burning time, not just which individual function.
If you want something with less friction, py-spy is a sampling profiler that attaches to a running process without code changes:
pip install py-spy
py-spy top --pid 12345
py-spy record -o profile.svg --pid 12345
The SVG flamegraph makes hot paths immediately obvious.
The Most Common Python Performance Mistakes
Once you have a profile, you're usually staring at one of these patterns:
Quadratic loops: a nested loop over a list of n items is O(n²). Convert the inner lookup to a dict or set.
Unnecessary object creation: string concatenation with += in a loop creates a new string object on every iteration. Use "".join(list_of_parts) instead.
Calling the same function repeatedly inside a loop: if you call len(my_list) inside a loop condition, Python evaluates it every iteration. Cache it outside.
# Before — O(n³) worst case
def find_duplicates(items):
duplicates = []
for i in range(len(items)):
for j in range(len(items)):
if i != j and items[i] == items[j]:
if items[i] not in duplicates:
duplicates.append(items[i])
return duplicates
# After — O(n)
def find_duplicates_fast(items):
seen = set()
duplicates = set()
for item in items:
if item in seen:
duplicates.add(item)
seen.add(item)
return list(duplicates)
On 10,000 items, the difference is seconds vs microseconds. The algorithmic fix always outperforms any amount of micro-optimization on the slow version.
Leveraging Built-ins and Standard Library Tools
CPython's built-in functions run at C speed. Using them instead of pure Python loops is usually the cheapest performance win available, and it requires no dependencies.
Compare sorting:
import timeit
data = list(range(10000, 0, -1))
def bubble_sort(arr):
n = len(arr)
for i in range(n):
for j in range(0, n - i - 1):
if arr[j] > arr[j + 1]:
arr[j], arr[j + 1] = arr[j + 1], arr[j]
return arr
t1 = timeit.timeit(lambda: bubble_sort(data.copy()), number=10)
t2 = timeit.timeit(lambda: sorted(data), number=10)
print(f"Bubble sort: {t1:.3f}s")
print(f"sorted(): {t2:.5f}s")
The result: sorted() runs 1000x faster because Timsort is implemented in C. No surprises there, but the same principle applies in less obvious places:
-
collections.Counterover manual counting dicts -
itertools.chainover manual list concatenation -
functools.lru_cachefor memoizing expensive pure functions -
bisectfor maintaining sorted lists instead of re-sorting on every insert
NumPy and Vectorization
For numerical work, the biggest lever is vectorization — replacing Python loops with NumPy operations that run in C with SIMD instructions.
import numpy as np
import time
n = 1_000_000
data = list(range(n))
# Pure Python
start = time.perf_counter()
result_py = [x ** 2 for x in data]
py_time = time.perf_counter() - start
# NumPy
arr = np.array(data)
start = time.perf_counter()
result_np = arr ** 2
np_time = time.perf_counter() - start
print(f"Python list comprehension: {py_time:.3f}s")
print(f"NumPy vectorized: {np_time:.4f}s")
print(f"Speedup: {py_time / np_time:.0f}x")
Typical output: 20-50x speedup. No multiprocessing, no C extensions — just letting NumPy use the CPU efficiently.
For Pandas: avoid iterrows(). It iterates row by row in Python, discarding all the performance advantages of the library. Write the transformation as a column operation directly, or use apply() with a vectorized function.
Concurrency: Threads, Processes, and asyncio
Python's GIL means threads don't help CPU-bound code. But they do help I/O-bound work — API calls, database queries, file reads.
The correct tool depends on the bottleneck:
-
I/O-bound (waiting on network, disk):
asyncioorThreadPoolExecutor -
CPU-bound (computation):
ProcessPoolExecutor
from concurrent.futures import ProcessPoolExecutor, as_completed
import os
def heavy_computation(n):
factors = []
for i in range(2, int(n ** 0.5) + 1):
if n % i == 0:
factors.append(i)
factors.append(n // i)
return sorted(set(factors))
numbers = [999983, 999979, 999961, 999931, 999917] * 20
# Sequential
results_seq = [heavy_computation(n) for n in numbers]
# Parallel across CPU cores
with ProcessPoolExecutor(max_workers=os.cpu_count()) as executor:
futures = {executor.submit(heavy_computation, n): n for n in numbers}
results_par = [f.result() for f in as_completed(futures)]
On a 4-core machine, the parallel version runs 3-4x faster. The overhead of spawning processes makes this worthwhile only for tasks that take at least a few milliseconds each — otherwise the inter-process overhead dominates.
For security tooling specifically, parallelizing I/O-bound work such as scanning a list of URLs against a threat intelligence checklist routinely produces 10-50x throughput improvements with asyncio, since you're almost entirely waiting on network responses rather than computing anything.
The Takeaway
Performance optimization in Python follows a strict order: measure, identify the actual bottleneck, apply the cheapest fix first.
- Profile with
cProfileorpy-spy— find where the time actually goes - Fix algorithmic issues first (O(n²) → O(n) beats every other optimization)
- Replace Python loops with built-ins and NumPy where possible
- Parallelize I/O-bound work with
asyncio, CPU-bound work withProcessPoolExecutor - Only reach for C extensions (Cython, ctypes) or PyPy when nothing else is enough
Skipping step 1 and jumping straight to parallelism is the most common mistake teams make. You'll add complexity without fixing the real problem. Profile first, every time.
I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.
Top comments (0)