DEV Community

Cover image for Make Your Python Code 10x Faster with These Tricks
Jesús Becerra
Jesús Becerra

Posted on

Make Your Python Code 10x Faster with These Tricks

#python #development #performance

1. __slots__ 

class Person:
    __slots__ = ['name', 'age']
Enter fullscreen mode Exit fullscreen mode

Why use this trick?:

  • Reduce memory usage: Python usually uses a __dict__ to store object attributes, which consumes more resources and is slow.
  • Faster attribute access: With __slots__, Python use a static array, eliminating the find key overhead.
  • Useful when you create millions of instances Data Processing, ETL, Data Science.

2. Generator Expressions ((x for x in ...)) 

sum(x**2 for x in range(1_000_000))
Enter fullscreen mode Exit fullscreen mode

Why use this trick?:

  • Avoid creating temporary lists: sum([x**2 for x in ...]) first creates a list in memory, then applies the sum function.
  • Reduce memory usage: Useful when you create large datasets.

3. Memoization with functools.lru_cache 

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    return fib(n-1) + fib(n-2) if n > 1 else n
Enter fullscreen mode Exit fullscreen mode

Why use this trick?:

  • Avoid recalculate results: Store in cache the values for the same parameters.
  • Pure functions (same input → same output): Ideal for recursive algorithms like Fibonacci, search in tree.

4. Vectorization with NumPy 

import numpy as np
arr = np.array([1, 2, 3])
result = arr * 2
Enter fullscreen mode Exit fullscreen mode

Why use this trick?:

  • Optimized operations in C: NumPy execute operations in contiguous memory blocks (arrays in C), without loops in Python.
  • Bypassing the GIL: Operations are applied directly to the data without Python loops.

5. Cython compilation 

# example.pyx (Cython)
def sum(int a, int b):
    return a + b
Enter fullscreen mode Exit fullscreen mode

Why use this trick?:

  • Converts Python code into C: Cython compiles code into native C extensions.
  • Static types improve efficiency: Declare Types (int, float), avoid python's dynamic typing.
  • Faster mathematical computations: 100x Faster math operations.

6. collections.defaultdict for Dictionaries 

from collections import defaultdict
count = defaultdict(int) # Initialize values by default
count["word"] += 1
Enter fullscreen mode Exit fullscreen mode

Why use this trick?:

  • Avoid if key in dict checks: In a normal dict, you need check if the key exist first. defaultdict do it for you.
  • Less calls to functions: More efficient that dict.setdefault() or try/except KeyError.

7. Concatenation with str.join() 

"".join(str_list)
Enter fullscreen mode Exit fullscreen mode

Why use this trick?:

  • Strings are immutable: In Python, each += create a new string object (This process is O(n²)).
  • join() pre-assign memory: Join each item in a only operation (This process is O(n)).

Performance Improvements

Trick Improvement Ideal Use Case
__slots__ ~20-30% more faster Handling mulpliple object instances
Generators 2-5x (in-memory) Big data processing
lru_cache 10-100x (velocity) Recursively called functions
NumPy 10-1000x Matrix operations
Cython 10-100x Heavy mathematical loops
defaultdict ~2x Counting
frecuencies
str.join() 10-100x (big N) Mass text concatenations

Compatibility by Python Version

1. __slots__

  • Compatible Versions: Python 2.7 and all Python 3.x versions (3.0+).
  • Details: Works the same across all modern versions.

2. Generator Expressions ((x for x in ...))

  • Compatible Versions: Python 2.4+ (but optimized in Python 3.x).
  • Improvements in Python 3:
    • Generators are more memory-efficient in Python 3.
    • range() in Python 3 is already a generator (in Python 2, it was a list).

3. functools.lru_cache

  • Compatible Versions:
    • Python 3.2+ (included in the standard library).
    • Python 2.7 requires pip install functools32.
  • Changes in Python 3.8+:
    • New user_function parameter for customization.

4. Vectorization with NumPy

  • Compatible Versions: Python 2.7 and Python 3.x (requires NumPy ≥ 1.16 for Python 3.7+).
  • Recommendation: Use NumPy ≥ 1.19 for Python 3.9+ (better support).

5. Cython

  • Compatible Versions:
    • Python 2.7 (supported up to Cython 0.29.x).
    • Python 3.x (requires Cython ≥ 3.0 for Python 3.11+).
  • Important:
    • Python 3.11+ has optimized the interpreter, reducing the performance gap between native Python and Cython in some cases.

6. collections.defaultdict

  • Compatible Versions: Python 2.5+ and all Python 3.x versions.
  • No significant changes across versions.

7. String Concatenation with str.join()

  • Compatible Versions: Python 2.x and 3.x.
  • Optimization in Python 3.9+:
    • Internal improvements in string handling (PEP 584).

Resume

  • Data Structures: __slots__, defaultdict.
  • Memory & CPU Efficiency: Generators, NumPy, Cython.
  • Algorithmic Optimization: lru_cache, str.join().

Top comments (0)