DEV Community: aykhlf yassir

Dropping Docker Build Times from 30s to 2s

aykhlf yassir — Fri, 05 Jun 2026 17:41:42 +0000

If your Docker build takes 30+ seconds every time you change a single line of code, your Dockerfile is working against you. The secret to lightning-fast builds lies in mastering layer caching.

Here are the five core rules to optimize your Dockerfiles for maximum performance and predictability.

1. Order by Volatility (Least to Most)

When Docker builds an image, it caches the instructions as layers and reuses them if their content hash hasn't changed. However, if a single layer changes, all subsequent layers get rebuilt.

Because of this cascading effect, you must sort your instructions from the least volatile (changes rarely) to the most volatile (changes constantly).

2. Separate Dependencies from Source Code

The most common mistake developers make is copying everything at once, which busts the cache on every code commit and forces dependencies to reinstall. Instead, split your COPY steps:

First, copy the manifest files that contain dependency metadata (e.g., pyproject.toml, package.json).
Run your package manager install command (e.g., pip install).
Finally, copy the actual code files.

Since code files change frequently in development, putting them last ensures your heavy dependency installation layer stays cached.

# 1. Copy ONLY dependency manifest
COPY pyproject.toml uv.lock ./

# 2. Install dependencies (Cached!)
RUN pip install --no-cache-dir fastapi uvicorn[standard]

# 3. Copy application code (Volatile)
COPY . .

3. Always Use a `.dockerignore`

Without a .dockerignore file, you risk sending massive or sensitive directories to the Docker build context.

Always exclude directories like .git, .env, .venv, and node_modules. Some of these contain too much irrelevant data that slows down the build, while others (like .env) contain secrets that should never be baked into your image.

4. Chain Your RUN Commands

Every RUN instruction creates its own read-only layer. If you run an update, install a package, and clean up the cache across three different RUN instructions, the removal only deletes the files in that specific layer. The preceding layers remain unnecessarily bloated.

To keep images small, chain your commands and clean up in the same step:

RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -fr /var/lib/apt/lists/*

5. Pin Your Base Images

Never rely on generic tags like python:latest or node:latest.

The latest tag is a moving target. Using it risks silently breaking the reproducibility and predictability of your builds when the underlying OS or language version updates unexpectedly. Always pin to specific, immutable versions (e.g., python:3.12.7-slim-bookworm) to ensure your deployments are bulletproof.

Dockerfile Fundamentals

aykhlf yassir — Fri, 05 Jun 2026 16:58:50 +0000

Building a Docker image from scratch can seem daunting at first, but a Dockerfile is really just a step-by-step recipe. Whether you're containerizing a FastAPI app or an Nginx server, you need to understand exactly what each instruction does.

Here is a concise breakdown of the core Dockerfile concepts and best practices.

1. The Base Image: Avoid `:latest` and Use `slim`

When choosing your base image with the FROM instruction, never use the :latest tag. It is a moving target that defeats the entire point of container reproducibility. Instead, pin a specific version.

Furthermore, prefer slim variants (like python:3.12-slim) over the default images. The standard python:3.12 image is a full Debian operating system, whereas the -slim version discards redundant system files and packages, drastically reducing your image size.

2. Setting Up the Workspace: `WORKDIR` and `COPY`

To get your code into the container, you typically see the combination of WORKDIR and COPY.

WORKDIR /app: This sets /app as the current directory inside the container's filesystem. Think of it as a persistent cd command for the rest of the image build.
COPY . .: This copies everything from your host machine to the container. The first . represents your build context (the current directory on your host), and the second . represents the destination (the container's current working directory, which we just set to /app).

3. Ports: The Truth About `EXPOSE`

It is a common misconception that EXPOSE 8000 publishes a port to your host machine. It does not.

EXPOSE serves strictly as documentation for developers and automated tools, claiming that the container is listening on that port. To actually access the application from the outside, you still need to map the port at runtime using docker run -p 8000:8000.

4. Running the App: `CMD` vs. `ENTRYPOINT`

These two instructions dictate what runs when your container starts, but they behave differently:

CMD: Sets the default command and its arguments. If a user passes arguments to docker run, the CMD instruction gets completely overridden.
ENTRYPOINT: Sets the core executable. Any arguments passed at docker run are appended to this executable rather than overriding it. This is ideal for containers designed to act as single-purpose CLI tools.

5. Always Use the "Exec" Form

When writing your CMD or ENTRYPOINT, always use the JSON array format (e.g., CMD ["uvicorn", "main:app"]), known as the exec form.

If you use the plain text shell form (e.g., CMD uvicorn main:app), Docker wraps your process in /bin/sh -c. This is dangerous because it breaks signal functionality. If your container acts as a single executable tool (like a web server or reverse proxy), the shell wrapper prevents system signals like SIGTERM from reaching your app, making graceful shutdowns impossible.

6. Inspecting Your Work

Once your image is built, you can audit how it was constructed using a single command:

docker history <image-name>

This command breaks down your built image layer by layer, showing you the exact size of each step and the specific Dockerfile instruction that created it. It is your best friend for debugging bloated images.

Everyday Docker CLI

aykhlf yassir — Fri, 05 Jun 2026 15:51:01 +0000

Docker has a massive surface area, but your day-to-day workflow only relies on a handful of commands. here is the essential cheat sheet for running, debugging, and cleaning up.

1. Running Containers (`docker run`)

The docker run command is your workhorse. By combining a few key flags, you can control exactly how your container behaves.

Run in the background with a specific name:

docker run -d --name my-cache redis:7-alpine

-d runs the container in detached mode (background).
--name gives it a recognizable name instead of a random hash.

Expose ports securely (Localhost only):

docker run -p 127.0.0.1:8080:80 nginx:alpine

Maps port 80 inside the container to port 8080 on your host machine, restricting access strictly to your local loopback address.

Inject Environment Variables:

# Single variable
docker run -e ENV=prod python:3.12-slim

# From a file
docker run --env-file .env python:3.12-slim

Auto-cleanup for one-off tasks:

docker run --rm python:3.12-slim python -c "print('Done!')"

--rm ensures the container is automatically deleted from your system the moment it stops running.

2. Debugging and Interacting

Once a container is running, you need visibility into what it's doing.

Stream live logs:

docker logs -f my-cache

The -f (follow) flag streams the logs in real-time. Use Ctrl+C to exit.

Drop into a shell of an already running container:

docker exec -it my-cache sh

Unlike docker run (which creates a new container), exec opens an interactive (-it) shell inside an existing one.

3. Container Lifecycle

Knowing how to stop a container properly prevents data corruption and hanging processes.

Graceful shutdown:

docker stop my-cache

Sends a SIGTERM signal, giving the container time to save state and shut down cleanly.

Force shutdown (when frozen):

docker kill my-cache

Sends an immediate SIGKILL signal. Use this only when stop fails.

4. Visibility and Cleanup

Docker accumulates stopped containers and unused images fast. Keep your system clean.

List all containers (including stopped ones):

docker ps -a

Safe system cleanup:

docker system prune

Removes stopped containers, unused networks, and "dangling" (untagged/unnamed) images.

Aggressive system cleanup:

docker system prune -a

Use with caution: This wipes everything that isn't currently attached to a container, including your standard tagged images (like nginx:alpine). You will have to re-download them from Docker Hub the next time you need them.

What Is a Container?

aykhlf yassir — Fri, 05 Jun 2026 13:46:37 +0000

Fundamentally, a container is not a Virtual Machine (VM); it is a regular Linux process running with three specific kernel abstractions.

Here is the mental model you need to understand how they work, why they took over the software world, and where they fall short.

The Three Kernel Mechanisms

Containers achieve VM-like isolation using these three Linux features:

Namespaces ("What the process can see"): This gives the container its own isolated view of the system resources. It isolates five key things: PIDs, network interfaces, filesystem mounts, hostnames (UTS), and users.
Cgroups ("What the process can use"): Control groups cap the hardware resources a container can consume. This prevents a runaway container from starving the host by limiting CPU, Memory, I/O, and Network bandwidth.
Filesystem Layer ("What the process starts with"): The container gets its own isolated root filesystem. This filesystem comes from a read-only image, and the container adds a thin writable layer on top.

Images vs. Containers

The easiest way to understand the difference between an image and a container is through an Object-Oriented Programming (OOP) analogy.

Image: A read-only filesystem snapshot and template. Think of this as the class in OOP.
Container: A running instance of that image. Think of this as the instance of the class. Many containers can be created and run simultaneously from a single image, each with their own writable layer.

Containers vs. VMs

While containers behave similarly to VMs, the key difference lies in the kernel.

A VM uses a hypervisor and carries its own full operating system and kernel.
A container borrows the host's Linux kernel.

Because of this, VMs offer superior security isolation, but containers are far less costly in memory (MBs vs GBs) and boot in milliseconds rather than minutes.

Why Containers Won (and Where They Fail)

Containers became the default deployment unit for four distinct reasons:

Portability: The image is the deployment artifact, meaning it can ship to and run on any Linux host.
Density: A single host can run hundreds of containers, compared to just tens of VMs on the same hardware.
Speed: They boot quickly and allow for fast iteration (build, push, pull, run in seconds).
Reproducibility: The exact same image behaves the same everywhere, eliminating the "it works on my machine" problem.

However, containers are not magic. Every architectural choice has honest costs:

Security Isolation: Because they share a kernel, a kernel exploit in a container can compromise the host machine.
Image Bloat: Naive image creation can result in massive, unwieldy files.
Linux-Only Limitation: Containers natively run only on Linux. Docker versions for Mac or Windows establish a lightweight Linux VM behind the scenes first, which can impact filesystem performance and memory.
Stateful is Harder: Managing persistent data is more complex in a containerized environment, which is why production databases often rely on managed services instead of containers.

Python Internals: Decorators

aykhlf yassir — Sat, 21 Feb 2026 01:42:45 +0000

Stop treating the @ symbol as magic. Let's tear down the abstraction layer and build decorators from first principles, using heap allocation, closures, and time complexity as our guides.

If you come from a static, compiled background like C++ or Java, Python decorators can feel like black magic. You slap an @login_required above a function, and suddenly it has authentication logic. You add @app.route("/"), and suddenly it's a web endpoint.

It feels like magic because it hides complexity. But as engineers, we know that "magic" is just code we haven't understood yet.

In this post, we are going to demystify Python metaprogramming. We won't just learn how to use decorators; we're going to understand the memory mechanics that make them possible, build a production-ready utility belt, and use them to fundamentally alter the algorithmic complexity of a function.

Part 1: The Mechanics

Before we write a decorator, we need to understand the raw materials. In Python, the mechanisms that enable decorators are First-Class Functions and Closures.

1. Functions are just heap-allocated objects

In C, a function is a block of instructions in the .text segment. In Python, a function is a full-blown object (a PyObject struct) living on the heap. Because it's an object:

You can assign it to a variable.
You can pass it as an argument to another function.
You can return it from another function.

This ability to treat functions as data is the cornerstone of metaprogramming.

2. The Engine: Closures

If you come from C++, you know that local variables die when a function's stack frame is popped. Python is different. If an inner function references a variable from an enclosing scope, Python's compiler notices. It "promotes" that variable from a simple stack item to a Cell Object on the heap.

Even after the outer function returns, the inner function retains a reference to that Cell Object. This "remembered environment" is called a Closure.

A closure is a function that remembers the variables from its enclosing scope even after the outer function has finished executing.

Decorators rely entirely on closures to remember which function they are wrapping.

Part 2: Building the Pattern

A decorator is fundamentally simple: It is a function that takes a function as input and returns a new function as output.

Let's look at the raw pattern before we use the pretty syntax.

# The decorator factory
def my_decorator(target_func):
    # The wrapper closure retains access to 'target_func'
    def wrapper():
        print(">>> Before execution")
        target_func() # Calling the original function
        print("<<< After execution")
    # Returning the new function
    return wrapper

def critical_system_call():
    print("Executing core logic...")

# MANUAL DECORATION: This is all a decorator is.
# We are reassigning the pointer to the new wrapper closure.
critical_system_call = my_decorator(critical_system_call)

critical_system_call()
# Output:
# >>> Before execution
# Executing core logic...
# <<< After execution

The @ symbol is just syntactic sugar for that manual reassignment. The following code is functionally identical to the manual version above:

@my_decorator
def critical_system_call():
    print("Executing core logic...")

The Introspection "Gotcha"

We broke something in the previous example. If we ask Python for the name of our function, it lies to us.

print(critical_system_call.__name__)
# Output: wrapper

Because we reassigned the pointer, the metadata now belongs to the inner wrapper function. This breaks debuggers, profilers, and frameworks that rely on introspection.

The Fix: functools.wraps

The standard library provides a fix. Paradoxically, it's another decorator called @wraps. It copies the metadata (name, docstring, annotations) from the original function to the wrapper.

This is the required boilerplate for nearly every decorator you will ever write:

from functools import wraps

def robust_decorator(func):
    # Apply @wraps to the wrapper closure
    @wraps(func) 
    def wrapper(*args, **kwargs):
        # Note the use of *args and **kwargs to make 
        # the wrapper generic for any function signature.
        print(f"Calling {func.__name__}...")
        return func(*args, **kwargs)
    return wrapper

@robust_decorator
def my_task():
    """Does hard work."""
    pass

print(my_task.__name__) # Output: my_task (Fixed!)
print(my_task.__doc__)  # Output: Does hard work. (Fixed!)

Part 3: The Utility Belt (Practical Examples)

Now that we have the mechanics, let's build tools commonly used in backend development.

1. The @timer (Profiling)

Don't use time.time() for profiling; it's subject to system clock adjustments. Use time.perf_counter() for monotonic timing guarantees.

import time
from functools import wraps

def timer(func):
    """Prints the execution time of the decorated function."""
    @wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.perf_counter()
        result = func(*args, **kwargs)
        end_time = time.perf_counter()
        print(f"[TIMER] {func.__name__} executed in {end_time - start_time:.4f}s")
        return result
    return wrapper

2. The @debug (Introspection Proxy)

This is a lifesaver when debugging complex recursion or deeply nested framework calls. It intercepts arguments and logs exactly what is going into and coming out of a function.

from functools import wraps

def debug(func):
    """Prints function signature and return value on every call."""
    @wraps(func)
    def wrapper(*args, **kwargs):
        args_repr = [repr(a) for a in args]
        kwargs_repr = [f"{k}={repr(v)}" for k, v in kwargs.items()]
        signature = ", ".join(args_repr + kwargs_repr)

        print(f"[DEBUG] Calling {func.__name__}({signature})")
        result = func(*args, **kwargs)
        print(f"[DEBUG] {func.__name__} returned {repr(result)}")
        return result
    return wrapper

Part 4: Boss Level Optimization (@memoize)

We can use decorators to fundamentally change the performance characteristics of an algorithm.

Let's look at the naive recursive Fibonacci implementation. Its time complexity is O(2^n) (exponential) because it re-calculates the same sub-problems endlessly.

# Without optimization, fibonacci(35) takes several seconds.
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

We can fix this with memoization (caching). We need a persistent dictionary to store results. Where does that dictionary live? In the closure's enclosing scope.

The Architectural Challenge

We need to use the function's arguments as keys for our cache dictionary.

Positional arguments (*args) are a tuple. Tuples are immutable and hashable. Good.
Keyword arguments (**kwargs) are a dict. Dicts are mutable and unhashable. Bad.

If we try to use kwargs as part of a dictionary key, Python throws a TypeError. We must convert the kwargs dictionary into an immutable, order-independent representation.

The Implementation

from functools import wraps

def memoize(func):
    # This dictionary lives in the closure's persistent heap state.
    cache = {}

    @wraps(func)
    def wrapper(*args, **kwargs):
        # 1. Convert kwargs to a sorted, immutable tuple of items
        frozen_kwargs = tuple(sorted(kwargs.items()))

        # 2. Create the composite cache key
        cache_key = (args, frozen_kwargs)

        # 3. Check cache (O(1) lookup)
        if cache_key not in cache:
             # Cache miss: compute and store
            cache[cache_key] = func(*args, **kwargs)

        return cache[cache_key]
    return wrapper

The Result

Let's stack our tools. Python applies decorators from the bottom up.

@timer    # Applied last (outer wrapper)
@memoize  # Applied first (inner wrapper around the recursive function)
def fibonacci_optimized(n):
    if n < 2:
        return n
    return fibonacci_optimized(n - 1) + fibonacci_optimized(n - 2)

# The original takes seconds.
# The optimized version executes instantly.
fibonacci_optimized(35) 
# Output: [TIMER] fibonacci_optimized executed in 0.0001s

By using a decorator to act as a caching proxy, we collapsed the time complexity from exponential O(2^n) to linear O(n). We made a classic space-time tradeoff, sacrificing a small amount of heap memory for massive CPU gains, without touching the core algorithm's code.

(Note: In production, prefer Python's built-in functools.lru_cache over rolling your own, as it handles cache bounding to prevent memory leaks).

Summary

Decorators aren't magic. They are an elegant application of First-Class Functions and Closures. By understanding how Python handles scope and object life-cycles on the heap, you gain the power to modify behavior, inject logic, and optimize performance cleanly and imperatively.

Python Internals: `yield from` for Composable Generators

aykhlf yassir — Thu, 19 Feb 2026 18:04:36 +0000

How a single keyword turns nested iterators into elegant, high-performance data pipelines

The Manual Delegation Problem

You've mastered generators. You know yield creates a pause point, that generators are lazy, and that they enable constant-memory processing of infinite streams.

But then you hit this problem:

def flatten(nested_list):
    """Flatten a nested list structure."""
    for item in nested_list:
        if isinstance(item, list):
            # We have a nested list - need to flatten it recursively
            for sub_item in flatten(item):  # Recursive call
                yield sub_item              # Manual forwarding
        else:
            yield item

nested = [1, [2, 3, [4, 5]], 6, [7, [8, 9]]]
print(list(flatten(nested)))  # [1, 2, 3, 4, 5, 6, 7, 8, 9]

It works. But look at that inner loop: for sub_item in flatten(item): yield sub_item. You're manually forwarding every value from the recursive generator to the caller. It's boilerplate—and it gets worse when your generator needs to handle .send() and .throw().

There's a better way:

def flatten(nested_list):
    """Flatten a nested list structure."""
    for item in nested_list:
        if isinstance(item, list):
            yield from flatten(item)  # Delegate completely
        else:
            yield item

Two words replace two lines. But this isn't just syntax sugar—yield from creates a transparent bidirectional channel between the caller and the subgenerator. Every next(), every .send(), every .throw() passes through untouched.

Today we're diving deep into what makes this powerful, when it's essential (not just convenient), and how it became the foundation for Python's async/await.

1. Core Mechanics: What `yield from` Actually Does

1.1. Generator Delegation: The Three Phases

When you write yield from subgen, Python establishes a direct communication channel that has three distinct phases:

def delegator():
    print("[DELEGATOR] Starting")
    result = yield from subgenerator()  # Delegation point
    print(f"[DELEGATOR] Subgen returned: {result}")
    yield "done"

def subgenerator():
    print("[SUBGEN] Starting")
    yield "first"
    yield "second"
    print("[SUBGEN] Exhausted")
    return "FINAL_VALUE"  # This return value is captured!

gen = delegator()
print(next(gen))  # "first"  - comes directly from subgenerator
print(next(gen))  # "second" - comes directly from subgenerator
print(next(gen))  # "done"   - delegator resumes after subgen exhausted

Output:

[DELEGATOR] Starting
[SUBGEN] Starting
first
second
[SUBGEN] Exhausted
[DELEGATOR] Subgen returned: FINAL_VALUE
done

Here's what happened, step by step:

Phase 1: Suspension

delegator() runs until it hits yield from subgenerator()
Control is fully transferred to subgenerator()
The delegating generator (delegator) is suspended—its stack frame is frozen

Phase 2: Transparent Proxying

Every next() call on gen is forwarded to subgenerator()
Values flow directly from subgen to caller—no intermediate handling
The delegator doesn't wake up at all during this phase

Phase 3: Exhaustion and Return Value Capture

When subgenerator() raises StopIteration, Python catches it
The StopIteration.value (the return value) is captured
The delegator resumes, with result bound to "FINAL_VALUE"

1.2. The Bidirectional Channel: Send, Throw, Close

The real power of yield from emerges when your generators are coroutines—not just producing values, but consuming them via .send() and handling errors via .throw().

Without `yield from`: Manual Proxying Hell

def delegator_manual():
    subgen = subgenerator()

    # Manually forward EVERY operation
    value = None
    while True:
        try:
            if value is None:
                result = next(subgen)
            else:
                result = subgen.send(value)
            value = (yield result)  # Yield result, receive next input
        except StopIteration as e:
            return e.value  # Capture final return value

This is 12 lines of intricate control flow just to forward operations. And it's still incomplete—it doesn't handle .throw() or .close() properly.

With `yield from`: Automatic Transparent Proxying

def delegator_clean():
    result = yield from subgenerator()
    return result

Two lines. Functionally identical. All operations—next(), .send(), .throw(), .close()—are automatically forwarded to the subgenerator.

Let's prove it with a coroutine:

def accumulator():
    """A coroutine that sums values sent to it."""
    total = 0
    while True:
        try:
            value = (yield total)  # Send out total, receive next value
            if value is None:
                break
            total += value
        except ValueError:
            print("[ACCUM] Received ValueError, resetting")
            total = 0

def delegator():
    """Delegates all operations to accumulator."""
    print("[DELEG] Starting delegation")
    final = yield from accumulator()
    print(f"[DELEG] Accumulator finished with: {final}")
    return final

# Build the pipeline
gen = delegator()
next(gen)  # Prime the coroutine

# Test send()
print(gen.send(10))   # 10
print(gen.send(20))   # 30

# Test throw()
gen.throw(ValueError)  # Resets accumulator to 0
print(gen.send(5))     # 5

# Test close via sending None
gen.send(None)

Output:

[DELEG] Starting delegation
10
30
[ACCUM] Received ValueError, resetting
0
5
[DELEG] Accumulator finished with: 5

Every .send() and .throw() went directly to accumulator()—the delegator never woke up. This is the transparent channel.

The PEP 380 Specification

yield from is precisely defined in PEP 380. Here's the pseudocode for what Python actually executes:

# yield from EXPR is equivalent to:

_iter = iter(EXPR)
try:
    _y = next(_iter)
except StopIteration as _e:
    _result = _e.value
else:
    while True:
        try:
            _sent = yield _y
        except GeneratorExit as _e:
            try:
                _meth = _iter.close
            except AttributeError:
                pass
            else:
                _meth()
            raise _e
        except BaseException as _e:
            _meth = getattr(_iter, 'throw', None)
            if _meth is None:
                raise
            try:
                _y = _meth(_e)
            except StopIteration as _e:
                _result = _e.value
                break
        else:
            try:
                if _sent is None:
                    _y = next(_iter)
                else:
                    _y = _iter.send(_sent)
            except StopIteration as _e:
                _result = _e.value
                break

This is 35 lines of intricate exception handling and control flow—all handled by two words: yield from.

2. Architectural Patterns: When to Use `yield from`

2.1. Composable Pipelines: Separation of Concerns

The canonical use case is building modular data processing pipelines where each component is a small, testable generator.

def read_logs(path: str):
    """Source: stream lines from a log file."""
    with open(path) as f:
        yield from f  # Delegate to file's iterator

def parse_lines(lines):
    """Transform: parse log format."""
    for line in lines:
        if line.strip():
            timestamp, level, message = line.split("|", 2)
            yield {
                "timestamp": timestamp.strip(),
                "level": level.strip(),
                "message": message.strip(),
            }

def filter_errors(records):
    """Filter: only ERROR-level records."""
    for record in records:
        if record["level"] == "ERROR":
            yield record

def pipeline(path: str):
    """Compose the full pipeline."""
    lines = read_logs(path)
    records = parse_lines(lines)
    errors = filter_errors(records)
    yield from errors  # Delegate to the final stage

# Usage: constant memory, no matter how large the file
for error in pipeline("application.log"):
    alert(error)

Each component:

Does one thing
Is testable in isolation
Composes via yield from
Processes data lazily

2.2. Recursive Generators: Implicit Stack Management

When you need to traverse recursive structures (trees, nested lists, JSON), yield from handles the call stack implicitly.

Example: Deep Flattening with Type Safety

from collections.abc import Iterable

def flatten(obj):
    """
    Recursively flatten any nested iterable structure.

    Handles: lists, tuples, sets, generators, custom iterables.
    Does NOT recurse into strings (they're iterable but shouldn't be flattened).
    """
    # Base case 1: strings are iterable but shouldn't be flattened
    if isinstance(obj, str):
        yield obj
        return

    # Base case 2: not iterable at all
    if not isinstance(obj, Iterable):
        yield obj
        return

    # Recursive case: iterate and recurse
    for item in obj:
        yield from flatten(item)

# Test cases
nested = [1, [2, 3, [4, 5]], 6, [7, [8, 9]]]
print(list(flatten(nested)))
# [1, 2, 3, 4, 5, 6, 7, 8, 9]

mixed = [1, "hello", [2, ["world", [3, 4]]], (5, 6)]
print(list(flatten(mixed)))
# [1, 'hello', 2, 'world', 3, 4, 5, 6]

The String Trap: Without the isinstance(obj, str) check, you'd get infinite recursion:

# BROKEN VERSION
def flatten_broken(obj):
    if not isinstance(obj, Iterable):
        yield obj
    else:
        for item in obj:
            yield from flatten_broken(item)  # Infinite loop on strings!

# This never terminates:
# flatten_broken("hello")
# → "hello" is iterable
# → iterate: 'h', 'e', 'l', 'l', 'o'
# → flatten('h')
# → 'h' is iterable (it's a string!)
# → iterate: 'h'
# → flatten('h')
# → ... infinite recursion

Strings are iterable, but each character is also a string. The check if isinstance(obj, str) breaks the cycle.

3. Data Structure Operations: Tree Traversals

yield from shines when traversing hierarchical structures. The recursive delegation naturally mirrors the tree's structure.

3.1. Binary Tree Traversal

from dataclasses import dataclass
from typing import Optional

@dataclass
class TreeNode:
    value: int
    left: Optional['TreeNode'] = None
    right: Optional['TreeNode'] = None

def inorder(node: Optional[TreeNode]):
    """Inorder traversal: Left → Root → Right"""
    if node is None:
        return  # Base case: empty tree/subtree

    yield from inorder(node.left)   # Recurse left
    yield node.value                # Process root
    yield from inorder(node.right)  # Recurse right

# Build a tree:
#        4
#       / \
#      2   6
#     / \ / \
#    1  3 5  7
root = TreeNode(4,
    left=TreeNode(2, TreeNode(1), TreeNode(3)),
    right=TreeNode(6, TreeNode(5), TreeNode(7))
)

print(list(inorder(root)))
# [1, 2, 3, 4, 5, 6, 7]  - sorted order!

The elegance: each recursive call handles its own subtree. The yield from stitches them together into a single stream. No explicit stack. No manual queue management.

3.2. File System Traversal (N-ary Tree)

from pathlib import Path

def walk_tree(path: Path):
    """
    Recursively traverse a directory tree, yielding all file paths.

    This is a generator-based reimplementation of os.walk().
    """
    if not path.exists():
        return

    if path.is_file():
        yield path  # Base case: leaf node (file)
    elif path.is_dir():
        for child in path.iterdir():
            yield from walk_tree(child)  # Recurse into subdirectory

# Usage
for filepath in walk_tree(Path(".")):
    if filepath.suffix == ".py":
        print(f"Found Python file: {filepath}")

Each directory is a node with N children. The recursion naturally handles arbitrary depth. Memory usage: O(depth), not O(total files).

Compare to the eager version:

# EAGER: Builds entire list before returning
def walk_tree_eager(path: Path) -> list[Path]:
    if path.is_file():
        return [path]
    elif path.is_dir():
        results = []
        for child in path.iterdir():
            results.extend(walk_tree_eager(child))  # Accumulates in memory
        return results

For a directory with 1 million files, the eager version allocates a list with 1 million Path objects before you can process the first one. The lazy version yields them one at a time.

4. Performance & Profiling: Measuring the Impact

Let's quantify the difference between lazy and eager evaluation.

4.1. Memory Comparison: `sys.getsizeof` vs. `tracemalloc`

import sys
import tracemalloc

def numbers_eager(n: int) -> list[int]:
    """Eager: build entire list."""
    return list(range(n))

def numbers_lazy(n: int):
    """Lazy: yield from range."""
    yield from range(n)

# Test with 1 million integers
n = 1_000_000

# Measure eager version
tracemalloc.start()
eager = numbers_eager(n)
eager_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

# Measure lazy version
tracemalloc.start()
lazy = numbers_lazy(n)
lazy_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

print(f"Eager list size:    {sys.getsizeof(eager):>12,} bytes")
print(f"Lazy generator:     {sys.getsizeof(lazy):>12,} bytes")
print(f"Eager traced mem:   {eager_mem:>12,} bytes")
print(f"Lazy traced mem:    {lazy_mem:>12,} bytes")
print(f"Memory savings:     {eager_mem / lazy_mem:.1f}x")

Output (typical):

Eager list size:        8,448,728 bytes
Lazy generator:               112 bytes
Eager traced mem:       8,448,824 bytes
Lazy traced mem:                496 bytes
Memory savings:         17034.7x

The lazy version uses 17,000x less memory. For a million integers, it's 8MB vs. 112 bytes.

4.2. Deep Memory Profiling: Tree Flattening

Let's compare eager vs. lazy flattening of a deeply nested structure:

import tracemalloc
import sys

def build_nested_list(depth: int, width: int):
    """Build a tree-like nested list structure."""
    if depth == 0:
        return list(range(width))
    return [build_nested_list(depth - 1, width) for _ in range(width)]

def flatten_eager(nested):
    """Eager flattening: accumulate in a list."""
    result = []
    if isinstance(nested, str):
        return [nested]
    if not isinstance(nested, list):
        return [nested]
    for item in nested:
        result.extend(flatten_eager(item))
    return result

def flatten_lazy(nested):
    """Lazy flattening: yield from."""
    if isinstance(nested, str):
        yield nested
        return
    if not isinstance(nested, list):
        yield nested
        return
    for item in nested:
        yield from flatten_lazy(item)

# Build test data: 5 levels deep, 3 children per level
# Total leaf nodes: 3^5 = 243
nested = build_nested_list(depth=5, width=3)

# Measure eager
tracemalloc.start()
eager_result = flatten_eager(nested)
eager_current, eager_peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

# Measure lazy (without consuming)
tracemalloc.start()
lazy_result = flatten_lazy(nested)
lazy_current, lazy_peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

# Measure lazy (with consumption)
tracemalloc.start()
lazy_consumed = list(flatten_lazy(nested))
lazy_consumed_current, lazy_consumed_peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

print(f"Nested structure has {len(eager_result)} leaf values")
print(f"\nEager flattening:")
print(f"  Peak memory: {eager_peak:>10,} bytes")
print(f"\nLazy generator (not consumed):")
print(f"  Peak memory: {lazy_peak:>10,} bytes")
print(f"\nLazy consumed into list:")
print(f"  Peak memory: {lazy_consumed_peak:>10,} bytes")
print(f"\nSavings (lazy vs eager): {eager_peak / lazy_peak:.1f}x")

Typical output:

Nested structure has 243 leaf values

Eager flattening:
  Peak memory:     24,832 bytes

Lazy generator (not consumed):
  Peak memory:        448 bytes

Lazy consumed into list:
  Peak memory:     12,160 bytes

Savings (lazy vs eager): 55.4x

Key insights:

Eager: Builds multiple intermediate lists during recursion → high memory
Lazy (not consumed): Just the generator object → minimal memory
Lazy consumed: Eventually needs storage, but ~50% less than eager due to no intermediate lists

4.3. Execution Time: Does Laziness Cost Performance?

import timeit

nested = build_nested_list(depth=6, width=3)  # 729 items

def benchmark_eager():
    result = flatten_eager(nested)
    return len(result)

def benchmark_lazy():
    result = list(flatten_lazy(nested))
    return len(result)

eager_time = timeit.timeit(benchmark_eager, number=10_000)
lazy_time = timeit.timeit(benchmark_lazy, number=10_000)

print(f"Eager: {eager_time:.4f} seconds")
print(f"Lazy:  {lazy_time:.4f} seconds")
print(f"Difference: {abs(lazy_time - eager_time) / eager_time * 100:.1f}%")

Typical output:

Eager: 0.8234 seconds
Lazy:  0.7891 seconds
Difference: 4.2% faster (lazy)

The lazy version is typically equal or slightly faster because:

No intermediate list allocations
No repeated extend() calls (which involve copying)
Generators have optimized C implementations

The overhead of generator frames is negligible compared to list operations.

5. Historical Context: The Road to `async/await`

yield from (introduced in Python 3.3 via PEP 380) wasn't just about nested generators—it was the necessary foundation for coroutine-based concurrency.

The Evolution

Python 3.3 (2012): yield from enables transparent delegation

def task():
    yield from subtask()  # Transparent channel

Python 3.4 (2014): asyncio uses generators as coroutines

@asyncio.coroutine
def fetch(url):
    response = yield from aiohttp.get(url)  # Async I/O
    return response

Python 3.5 (2015): async/await syntax replaces generator-based coroutines

async def fetch(url):
    response = await aiohttp.get(url)  # Same semantics, clearer syntax
    return response

The semantics are identical: await is yield from with a type check. The runtime behavior—transparent delegation, bidirectional channels, exception routing—is the same.

Why the New Syntax?

Generator-based coroutines (yield from) were powerful but confusing:

# This is a generator (produces values)
def numbers():
    yield from range(10)

# This is ALSO a generator, but used as a coroutine (consumes control flow)
@asyncio.coroutine
def fetch():
    yield from aiohttp.get(url)

Same syntax, completely different purposes. The async/await keywords made the distinction explicit:

# Clearly a generator
def numbers():
    yield from range(10)

# Clearly a coroutine
async def fetch():
    await aiohttp.get(url)

But under the hood, await still does everything yield from does—it's yield from with runtime type validation.

Conclusion: When to Reach for `yield from`

Use yield from when you need to:

1. Delegate to Another Generator Completely

If you're writing for x in gen: yield x, replace it with yield from gen. It's:

More readable
More efficient
Handles .send() and .throw() correctly

2. Compose Data Processing Pipelines

Break your pipeline into small, testable generators and connect them with yield from:

def pipeline():
    raw = read_data()
    parsed = parse(raw)
    filtered = filter_valid(parsed)
    yield from transform(filtered)

3. Traverse Recursive Structures

Trees, nested lists, file systems—any recursive data structure benefits from yield from's implicit stack management:

def traverse(node):
    if node.is_leaf():
        yield node
    else:
        for child in node.children:
            yield from traverse(child)

4. Build Coroutine-Based State Machines

If you're building complex coroutines that delegate to subcomponents (pre-asyncio or for synchronous use cases):

def coordinator():
    result1 = yield from worker1()
    result2 = yield from worker2(result1)
    return result2

Python Internals: Generators & Coroutines

aykhlf yassir — Mon, 16 Feb 2026 21:24:31 +0000

There's a function that exists in almost every Python codebase. It looks harmless:

def get_trades(symbol: str) -> list[dict]:
    results = []
    for record in enormous_database_cursor:
        if record["symbol"] == symbol:
            results.append(record)
    return results

trades = get_trades("AAPL")  # 💀 Waits. Waits. Waits. Then crashes.

The problems stack up fast:

High latency: You get zero items until the entire database is scanned
Massive RAM: Every matching record is held in memory simultaneously
Fragility: One spike in result size kills the process

This is the "eager" pattern—do all the work, collect all the results, then hand them over. For small datasets, you'll never notice. For anything real-world, it's a time bomb.

The fix is a single keyword. But to use it correctly, you need to understand what it actually does to your function.

1. The Basics: What `yield` Does to a Function

Every Python function you've written follows the same lifecycle:

Call → Execute → return value → Stack frame is destroyed → Done

Local variables evaporate. State is gone. The function has no memory that it ever ran.

yield breaks this contract entirely.

The Normal Function: A Sprint

def countdown_list(n: int) -> list[int]:
    result = []
    while n > 0:
        result.append(n)
        n -= 1
    return result  # Hands you everything at once, then dies

One call. One massive result. The function's stack frame is created, used, and destroyed.

The Generator: A Pause Button

def countdown(n: int):
    while n > 0:
        yield n    # Pause here, hand back n, wait to be resumed
        n -= 1

The moment Python sees yield in a function body, the rules change. Calling countdown(5) no longer executes a single line of code. Instead, Python hands you back a generator object—a suspended, ready-to-run machine.

gen = countdown(5)
print(gen)          # <generator object countdown at 0x7f...>
print(next(gen))    # 5  → Runs until yield, pauses, returns 5
print(next(gen))    # 4  → Resumes, runs until yield, pauses, returns 4
print(next(gen))    # 3  → Same

What makes this possible? When a generator pauses at yield, its entire stack frame—local variables, current line number, the value of n—is moved from the stack to the heap. It doesn't disappear. It waits, frozen in time, until next() is called again.

Normal function:    [Stack Frame] → return → [Destroyed]

Generator:          [Stack Frame] → yield → [Moved to Heap, frozen]
                                          ↓
                    next() called  → [Thawed, execution resumes]
                                          ↓
                                   yield → [Frozen again]

Old Way vs. New Way: Side by Side

Before generators, you had to implement the iterator protocol manually—a verbose class with __iter__ and __next__:

# THE OLD WAY: Class-based iterator (20 lines of boilerplate)
class Countdown:
    def __init__(self, start: int) -> None:
        self.current = start

    def __iter__(self):
        return self

    def __next__(self) -> int:
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

# THE NEW WAY: Generator function (4 lines, zero boilerplate)
def countdown(n: int):
    while n > 0:
        yield n
        n -= 1

Same behavior. Same memory efficiency. Same protocol compatibility—countdown(5) works anywhere Countdown(5) does.

The yield keyword is a class-based iterator, fully implemented, in one line.

2. Infinite Data Pipelines: The "Pull" Model

Here's where generators move from "interesting" to "indispensable."

Consider the difference:

import sys

# Eager: allocate the entire sequence in RAM
big_list = [x ** 2 for x in range(1_000_000)]
print(f"List size:      {sys.getsizeof(big_list):>12,} bytes")  
# List size:       8,448,728 bytes (~8 MB)

# Lazy: a tiny object that knows *how* to produce values
big_gen = (x ** 2 for x in range(1_000_000))
print(f"Generator size: {sys.getsizeof(big_gen):>12,} bytes")
# Generator size:         104 bytes (~104 bytes)

Eight megabytes vs. 104 bytes. The generator doesn't store the squares—it stores the recipe for producing the next one. Scale this to 10GB of log files or a live market feed, and this difference is what separates a working system from a crashed one.

Generator Expressions: Lazy List Comprehensions

The (x for x in ...) syntax is a generator expression—the lazy sibling of list comprehensions.

# List comprehension: eager, executes immediately
squares_list = [x ** 2 for x in range(10)]    # All 10 computed NOW

# Generator expression: lazy, executes on demand
squares_gen  = (x ** 2 for x in range(10))    # NONE computed yet

Square brackets → list (eager). Parentheses → generator (lazy).

The Pipeline Architecture

Generators compose naturally into pipelines—a chain of lazy transformations where data flows through only when pulled from the end:

  Source          Filter               Processor
    │                │                     │
market_ticker() ──► (t for t              ──► trading logic
[infinite stream]    if t == 'AAPL')          (processes one
                    [lazy filter]              tick at a time)

def market_ticker():
    """Simulates an infinite stream of market data."""
    import random, itertools
    symbols = ['AAPL', 'GOOG', 'MSFT', 'AMZN']
    for i in itertools.count():
        yield {
            'symbol': random.choice(symbols),
            'price': round(random.uniform(100, 300), 2),
            'volume': random.randint(100, 10000),
        }

# Build the pipeline — nothing executes yet
ticker    = market_ticker()                             # Source: infinite generator
aapl_only = (t for t in ticker if t['symbol'] == 'AAPL') # Filter: lazy expression

# Data flows ONLY when we pull from the end
tick = next(aapl_only)  # NOW it runs: pulls from ticker until it finds AAPL
print(tick)  # {'symbol': 'AAPL', 'price': 172.34, 'volume': 4821}

Nothing ran when we built the pipeline. No data was fetched, no filtering occurred. The entire chain is dormant until we call next(). This is lazy evaluation—the pipeline pulls data through only as fast as you consume it.

This is how you process a terabyte log file with 104 bytes of working memory.

3. The Deep Dive: Generators as Coroutines

Everything above treats generators as producers: you pull data out of them via next().

But generators can also be consumers: you push data into them via .send(). This transforms a generator from a simple stream into a stateful processing unit—what computer scientists call a coroutine.

`yield` as an Expression

Normally, yield value is a statement—it sends a value out. But it can also be an expression that receives a value:

def accumulator():
    total = 0
    while True:
        value = (yield total)  # Pause: send out total, wait to receive a value
        if value is not None:
            total += value

acc = accumulator()
next(acc)        # Prime the coroutine (advance to first yield)
acc.send(10)     # Push 10 in → total becomes 10
acc.send(20)     # Push 20 in → total becomes 30
result = acc.send(5)
print(result)    # 35

The priming step (next(acc)) is required. A fresh generator is frozen at the start of the function, before any yield has been reached. You must advance it to the first yield before you can send anything to it.

The Three Generator Controls

Operation	Syntax	What it does
Pull	`next(gen)`	Resume, run until next `yield`, return yielded value
Push	`gen.send(val)`	Resume with `val` as the result of `yield`, run until next `yield`
Throw	`gen.throw(ExcType)`	Resume by raising an exception at the yield point
Close	`gen.close()`	Throw `GeneratorExit`, shut down cleanly

The .throw() method is particularly powerful. Instead of crashing your pipeline when bad data appears, you can inject the error directly at the coroutine's pause point and let it handle recovery internally.

Building a Finite State Machine with `yield`

A coroutine's "current line number" is its state. No state variables. No if state == "WATCHING" branching at the top. The control flow itself encodes the state.

from enum import Enum, auto

class BotState(Enum):
    WATCHING = auto()
    ACTIVE   = auto()

def trading_bot(entry_threshold: float = 150.0,
                exit_threshold:  float = 200.0) -> None:
    """
    A coroutine FSM with two states:
      WATCHING → waiting for a low price to enter a position
      ACTIVE   → holding a position, waiting to exit at profit
    """
    print("[BOT] Initialized. State: WATCHING")
    entry_price: float = 0.0

    while True:
        try:
            # ── STATE: WATCHING ──────────────────────────────
            while True:
                price: float = (yield)               # Wait for next tick
                print(f"[WATCHING] AAPL @ ${price:.2f}")
                if price <= entry_threshold:
                    entry_price = price
                    print(f"[SIGNAL] Entry at ${entry_price:.2f} → switching to ACTIVE")
                    break                            # Transition to ACTIVE

            # ── STATE: ACTIVE ────────────────────────────────
            while True:
                price = (yield)                      # Wait for next tick
                pnl   = price - entry_price
                print(f"[ACTIVE]   AAPL @ ${price:.2f} | PnL: ${pnl:+.2f}")
                if price >= exit_threshold:
                    print(f"[SIGNAL] Exit at ${price:.2f} | Profit: ${pnl:.2f} → switching to WATCHING")
                    break                            # Transition back to WATCHING

        except ValueError as e:
            # Bad tick injected via .throw() — reset to WATCHING without crashing
            print(f"[ERROR] Bad data received: {e}. Resetting to WATCHING.")
            entry_price = 0.0
            # Loop continues: back to WATCHING state

4. The Grand Finale: The High-Frequency Trading Bot

Let's wire everything together. Four components. One elegant pipeline.

The Architecture

┌─────────────────────────────────────────────────────────┐
│                    PIPELINE OVERVIEW                     │
│                                                          │
│  [Source]          [Filter]           [Sink]             │
│  market_ticker()──►aapl_stream ──────►trading_bot()      │
│  (Generator)       (Gen. Expression)  (Coroutine FSM)    │
│       │                  │                  │            │
│   Produces all       Passes only        Consumes AAPL   │
│   symbols lazily     AAPL ticks         ticks, manages  │
│                                         state internally │
│                                                          │
│                   [Bridge]                               │
│                   for loop with                          │
│                   .send() / .throw()                     │
└─────────────────────────────────────────────────────────┘

The Complete System

import random
import itertools
import sys

# ── COMPONENT 1: THE SOURCE ──────────────────────────────────────
def market_ticker():
    """Infinite stream of market ticks. Never terminates."""
    symbols = ['AAPL', 'GOOG', 'MSFT', 'AMZN']
    for _ in itertools.count():
        yield {
            'symbol': random.choice(symbols),
            'price':  round(random.uniform(100, 300), 2),
        }

# ── COMPONENT 2: THE FILTER ──────────────────────────────────────
def build_pipeline():
    ticker    = market_ticker()
    # Occasionally inject a bad tick to test error handling
    aapl_stream = (
        t for t in ticker
        if t['symbol'] == 'AAPL'
    )
    return aapl_stream

# ── COMPONENT 3: THE CONSUMER (FSM Coroutine) ────────────────────
# (trading_bot as defined in Section 3 above)

# ── COMPONENT 4: THE BRIDGE ──────────────────────────────────────
def run(tick_limit: int = 30) -> None:
    stream = build_pipeline()
    bot    = trading_bot(entry_threshold=150.0, exit_threshold=200.0)

    # Prime the coroutine — advance it to the first yield
    next(bot)

    ticks_processed = 0
    for tick in stream:
        if ticks_processed >= tick_limit:
            break

        price = tick['price']

        # Simulate occasional bad data (1-in-10 chance)
        if random.random() < 0.1:
            bad_price = -abs(price)  # Corrupt tick: negative price
            try:
                bot.throw(ValueError(f"Negative price: {bad_price}"))
                next(bot)            # Re-prime after error recovery
            except StopIteration:
                print("[BRIDGE] Bot shut down during error recovery.")
                break
        else:
            try:
                bot.send(price)      # Normal operation: push price to bot
            except StopIteration:
                print("[BRIDGE] Bot has shut down.")
                break

        ticks_processed += 1

    bot.close()  # Send GeneratorExit — clean shutdown
    print(f"\n[BRIDGE] Pipeline complete. Processed {ticks_processed} AAPL ticks.")

if __name__ == "__main__":
    run(tick_limit=30)

Why `.throw()` is the Pro Pattern

Most tutorials show .send() and call it a day. But .throw() is what makes a coroutine-based FSM production-grade.

The alternative to .throw() is sentinel values:

# Naive approach: use magic values to signal errors
bot.send(-1)  # Hope the bot understands -1 means "bad data"

This is fragile. It poisons your data channel with control signals. What if -1 is a legitimate (if unusual) price? What if you need to distinguish between different error types?

.throw() keeps the error channel and the data channel separate:

# The Bridge: clear separation of concerns
for tick in stream:
    if tick['price'] < 0:
        bot.throw(ValueError(f"Corrupt tick: {tick}"))  # Error channel
    else:
        bot.send(tick['price'])                          # Data channel

The coroutine catches it in a try/except at its yield point—exactly like a normal function. The state machine resets cleanly. The pipeline keeps running. Zero sentinel values. Zero ambiguity.

Conclusion: Why This Matters

Let's close with three concrete reasons this mental model changes how you write code.

1. Memory Efficiency: O(1) Space for Infinite Streams

# This processes a 10GB log file in constant memory
def find_errors(path: str):
    with open(path) as f:
        yield from (line for line in f if "ERROR" in line)

for error_line in find_errors("application.log"):
    alert(error_line)

No list. No .readlines(). A single line lives in memory at a time.

2. State Management: Your Line Number Is Your State

The trading_bot() coroutine has zero explicit state variables for its FSM transitions. The while True loop it's currently executing in is the state. Python's own call stack manages it.

Compare that to the class-based alternative:

# The non-generator version: manual state management
class TradingBot:
    def __init__(self):
        self.state = "WATCHING"    # Explicit state
        self.entry_price = 0.0

    def process(self, price: float) -> None:
        if self.state == "WATCHING":   # State checks everywhere
            ...
        elif self.state == "ACTIVE":
            ...

More code. More surface area for bugs. More branching to read and maintain.

3. Composability: UNIX Pipes for Your Data

Each component in our pipeline does exactly one thing: the source generates ticks, the filter screens symbols, the bot manages trades. They're connected by convention—the iterator protocol—not by inheritance or tight coupling.

You can swap any component without touching the others:

# Swap source: real broker API instead of random data
ticker = broker_api.stream()           # Same interface, different source

# Swap filter: multiple symbols
stream = (t for t in ticker if t['symbol'] in {'AAPL', 'MSFT'})

# Swap sink: logging bot instead of trading bot
bot = audit_logger(output="trades.log")  # Same .send() interface

Small tools. Single responsibilities. Glued by protocol.

Python Internals: The Iterator Protocol

aykhlf yassir — Sun, 15 Feb 2026 17:34:27 +0000

Stop Thinking in Lists. Start Thinking in Streams.

Here's a question that separates junior Python developers from senior ones:

What happens when you run for item in my_list?

If your answer was "Python loops over the list," you're not wrong—but you're describing the what, not the how. And the how is where the real engineering lives.

The for loop is syntactic sugar. Under the hood, it's a precise protocol—a contract between your code and Python's runtime. Once you understand that contract, you'll never load a 10GB file into memory again, your pipelines will run in constant memory, and you'll finally understand why generators feel like magic.

By the end of this post, you'll know:

The difference between an iterable and an iterator
What Python actually executes when you write a for loop
Why the same iterator can't be looped twice
How to build your own memory-efficient data pipelines from scratch

Let's pull back the curtain.

1. The Definitions:

The words "iterable" and "iterator" are used interchangeably in most tutorials. This can lead to confusion. Let's define them precisely.

The Analogy

Think of a book and a bookmark.

The book (Iterable) contains all the data. It can be read from the beginning any number of times. You can hand it to anyone, and they can start reading from page one. A list, a tuple, a str—these are all books.

The bookmark (Iterator) tracks where you are in a specific reading session. It has state. It knows you're on page 47. It can tell you the next word, then advance one position. Crucially: there is only one reading session. When you reach the last page, the bookmark is spent. Generators, file objects, map() results—these are all bookmarks.

The Interface

Python makes this concrete through two dunder methods:

Concept	Must Implement	Behavior
Iterable	`__iter__`	Returns a new Iterator
Iterator	`__iter__` AND `__next__`	`__next__` returns the next value, raises `StopIteration` when exhausted

# A list is an ITERABLE: it implements __iter__
my_list = [1, 2, 3]
print(hasattr(my_list, '__iter__'))  # True
print(hasattr(my_list, '__next__'))  # False - not an iterator!

# Calling __iter__ on a list creates an ITERATOR
my_iterator = iter(my_list)
print(hasattr(my_iterator, '__iter__'))  # True
print(hasattr(my_iterator, '__next__'))  # True - now we have both

The iter() built-in calls __iter__. The next() built-in calls __next__. Everything else in Python is built on top of these two primitives.

2. The Mechanics: Deconstructing the `for` Loop

Here is the lie Python tells you every day:

# What you write (The Sugar)
for item in [1, 2, 3]:
    print(item)

Here is what Python actually executes:

# What Python runs (The Reality)
_iter = iter([1, 2, 3])   # Step 1: Call __iter__, get an iterator

while True:                # Step 2: Loop forever
    try:
        item = next(_iter) # Step 3: Call __next__, get the next item
        print(item)        # Step 4: Execute the loop body
    except StopIteration:  # Step 5: Iterator is exhausted
        break              # Step 6: Exit the loop

That's it. That's the entire for loop. No magic. No special knowledge of lists or tuples or strings. Just a mechanical protocol: call __iter__ once, call __next__ repeatedly until StopIteration is raised.

The Critical Insight: The Loop is Blind

The for loop does not care about your data structure. It cannot see whether you gave it a list, a database cursor, a network socket, or a custom class you wrote this morning.

It only asks one question: "Does this object speak the iterator protocol?"

If __iter__ and __next__ exist, the loop works. This is Python's duck typing philosophy applied at the language level. No inheritance required. No registration required. Just implement two methods.

# COMPLETELY custom iteration - no list, no built-in involved
class Countdown:
    def __init__(self, start: int) -> None:
        self.current = start

    def __iter__(self):
        return self  # More on this shortly

    def __next__(self) -> int:
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

# The for loop has NO IDEA this is a custom class
for n in Countdown(5):
    print(n)  # 5, 4, 3, 2, 1

3. Why Does `iter` Return `self`?

If you look carefully at the Countdown class above, you'll notice something odd: __iter__ returns self. An iterator returning itself as an iterator.

If an iterable is a factory that creates iterators, and an iterator is a consumer that gets exhausted... why does an iterator pretend to be an iterable?

The Answer: Polymorphism

Consider this function:

def print_first_three(iterable):
    it = iter(iterable)  # Always call iter() first
    print(next(it))
    print(next(it))
    print(next(it))

If we follow the "always call iter() first" convention, this function works with both iterables and iterators:

# Works with an iterable (list)
print_first_three([10, 20, 30, 40, 50])

# Works with an iterator (generator, file, network stream)
my_stream = iter([10, 20, 30, 40, 50])
print_first_three(my_stream)

The second call works because when you call iter() on an iterator, it returns self. The protocol stays consistent. The caller doesn't need to know—or care—whether it received a fresh list or a half-consumed stream.

This is the rule, codified:

Iterable:  __iter__() → returns a NEW Iterator (fresh reading session)
Iterator:  __iter__() → returns SELF     (I am already the session)

Object	`__iter__`	`__next__`
`list`, `tuple`, `str`	Returns a new list_iterator	✗ Not present
`list_iterator`	Returns `self`	✓ Advances position
Generator	Returns `self`	✓ Runs until `yield`
File object	Returns `self`	✓ Reads next line

In short: Every iterator is also iterable. Not every iterable is an iterator.

4. The "One-Shot" Trap:

Here is a bug that has shipped to production more times than anyone will admit:

squares = map(lambda x: x**2, [1, 2, 3])

print(list(squares))  # [1, 4, 9] ✓
print(list(squares))  # []        ✗ Silent failure

The second call returns an empty list. No error. No warning. Just silently wrong data.

Why? Because map() returns an iterator—a bookmark, not a book. The first list() call consumes every item, advancing the bookmark to the end. The second call finds the iterator exhausted. StopIteration is raised immediately. list() catches it and returns [].

This trap catches developers with:

# All of these are ONE-SHOT iterators:
squares = map(lambda x: x**2, range(5))
evens = filter(lambda x: x % 2 == 0, range(10))
pairs = zip([1, 2, 3], ['a', 'b', 'c'])

lines = open("data.txt")  # File objects too!

The Fix: Know What You Have

The decision tree is simple:

Do I need to iterate this data multiple times?
├── YES → Store it: data = list(my_iterator)
└── NO  → Stream it: consume it once, discard it

# If you need multiple passes: materialize to list
squares = list(map(lambda x: x**2, [1, 2, 3]))
print(squares)  # [1, 4, 9]
print(squares)  # [1, 4, 9] - Works! It's a list now.

# If one pass is enough: stay lazy, save memory
for square in map(lambda x: x**2, range(10_000_000)):
    process(square)  # Never loads 10M items into RAM

The Generator Version of the One-Shot Trap

Generator functions are iterators, so they have the same behavior:

def squares(n: int):
    for i in range(n):
        yield i ** 2

gen = squares(5)

print(sum(gen))   # 30 ✓
print(sum(gen))   # 0  ✗ - exhausted!

Every time you call squares(5), you get a new generator object—a fresh bookmark. But if you assign it to a variable and reuse that variable, you're sharing one bookmark between two consumers.

5. The Toolkit: `itertools` — Don't Reinvent the Wheel

Once you think in streams, you'll want to transform, combine, and slice them. Python's itertools module is the standard library for doing exactly this—all lazily, all in O(1) memory.
itertools are

The itertools module is high performance C code exposed to Python. Using it is almost always faster than writing your own loop logic.

Here are the three tools you'll reach for every week:

`itertools.count` — Infinite Streams

import itertools

# Infinite counter: 10, 11, 12, 13, ...
counter = itertools.count(start=10, step=1)

# Always pair with islice or a break condition
for n in counter:
    if n > 15:
        break
    print(n)  # 10, 11, 12, 13, 14, 15

Use this anywhere you'd use while True with a manual counter.

`itertools.chain` — Linking Streams

import itertools

# Process multiple files as one continuous stream
# NO intermediate list created
log_lines = itertools.chain(
    open("january.log"),
    open("february.log"),
    open("march.log"),
)

for line in log_lines:
    process(line)  # Streams through all three files sequentially

The alternative—open("jan") + open("feb") — doesn't work on file objects. chain is the correct tool for combining any iterables without materializing them, good stuff.

`itertools.islice` — Safe Consumption

import itertools

# Take the first 5 from any iterator, safely
def squares():
    n = 0
    while True:
        yield n ** 2
        n += 1

# Without islice, this would run forever
first_five = list(itertools.islice(squares(), 5))
print(first_five)  # [0, 1, 4, 9, 16]

islice is the lazy equivalent of my_list[:5]. It doesn't generate all items first—it stops consuming after n items.

The Combinators at a Glance

Tool	What it does	Memory
`count(n)`	Infinite incrementing stream	O(1)
`cycle(it)`	Repeat iterable forever	O(n)
`repeat(x, n)`	Yield x exactly n times	O(1)
`chain(*its)`	Concatenate iterables	O(1)
`islice(it, n)`	Take first n items lazily	O(1)
`takewhile(pred, it)`	Take while condition holds	O(1)
`dropwhile(pred, it)`	Skip while condition holds	O(1)
`batched(it, n)`	Group into n-sized chunks	O(n)

Conclusion:

Let's close with the mental model shift.

Junior mindset: A for loop iterates over a collection.

Senior mindset: A for loop drives a protocol. Any object that speaks the protocol—regardless of what it is or where the data lives—can be iterated. The data could be in a list in RAM, a file on disk, a socket on the network, a database cursor, or computed on the fly from a mathematical formula. The loop doesn't care.

This is the power:

# These all work identically with the for loop:
for item in [1, 2, 3]:                          # In memory
for line in open("data.csv"):                   # On disk
for record in database.execute("SELECT ..."):   # In database
for packet in socket.recv_packets():            # On network
for value in itertools.count():                 # Computed infinitely

Same protocol. Same syntax. Radically different resource profiles.

And the payoff in memory:

# Junior code: loads entire file
def count_errors(path: str) -> int:
    lines = open(path).readlines()  # 10GB in RAM 💀
    return sum(1 for line in lines if "ERROR" in line)

# Senior code: streams the file
def count_errors(path: str) -> int:
    with open(path) as f:           # ~constant memory ✓
        return sum(1 for line in f if "ERROR" in line)

Both produce the same number. One will kill your server at 3AM.

Your Action Item

Go find that one function in your codebase that does this:

data = list(some_query_or_file_or_api_call())
for item in data:
    process(item)

If you never use data again after the loop, that list() call is pure waste. Delete it. Stream directly. Your memory graph will thank you.

The for loop doesn't need the list. It never did.

This is part of an ongoing series on Python internals. If this post changed how you think about loops, follow along.

Python Internals: Type Hints

aykhlf yassir — Sat, 07 Feb 2026 19:46:45 +0000

From dynamic chaos to static certainty: mastering MyPy and strict mode

The "I Forgot What This Returns" Problem

You're six months into a project. You need to call a function someone else wrote:

def process(data):
    # ... 50 lines of code ...
    return result

What does data need to be? A list? A dict? A pandas DataFrame? A custom class?

What does result actually contain? A boolean? A tuple? None?

You have three options:

Read all 50 lines of implementation code
Find example usage somewhere else in the codebase
Run it and hope for the best

All three are terrible.

The Solution

def process(data: pd.DataFrame) -> tuple[bool, str]:
    # ... 50 lines of code ...
    return result

Now you know instantly:

Input: pandas DataFrame
Output: tuple of (bool, str)

No runtime cost. No reading code.

The Misconception

Many developers think type hints are for Python—that they change how the code runs. They don't.

def add(a: int, b: int) -> int:
    return a + b

print(add("hello", "world"))  # "helloworld" - Python doesn't care!

Python completely ignores type hints at runtime. They're not constraints; they're documentation with a superpower: they can be automatically verified.

Type hints are for:

Your IDE - Autocomplete and error detection
MyPy - Static type checker that finds bugs before runtime
Other developers - Understanding your code without reading it
Future you - Remembering what you meant six months ago

The Modern Syntax (Python 3.10+)

If you learned Python type hints before 2021, forget what you know. The syntax changed dramatically in Python 3.9 and 3.10.

The Old Way (Verbose and Ugly)

from typing import List, Dict, Union, Optional, Tuple

def get_user(user_id: Union[int, None]) -> Dict[str, Union[str, int]]:
    pass

def process_items(items: Optional[List[str]]) -> Tuple[bool, str]:
    pass

This required importing half of the typing module and using capital letters for built-in types (List instead of list).

The New Way (Clean and Native)

def get_user(user_id: int | None) -> dict[str, str | int]:
    pass

def process_items(items: list[str] | None) -> tuple[bool, str]:
    pass

What changed:

Union syntax: Union[int, None] → int | None
Optional is dead: Optional[str] → str | None
Lowercase collections: List[int] → list[int], Dict[str, int] → dict[str, int]
Tuple syntax: Tuple[bool, str] → tuple[bool, str]

The Compatibility Timeline

Python 3.9: Can use list[int], dict[str, int] but not | for unions
Python 3.10+: Can use | for unions, making Union and Optional obsolete
Python 3.12+: New generic syntax for classes (we'll cover this)

If you're on Python 3.10+, never import Union or Optional again.

Generic Collections: The Cheat Sheet

# Lists - homogeneous collection
numbers: list[int] = [1, 2, 3]
names: list[str] = ["Alice", "Bob"]
mixed: list[int | str] = [1, "two", 3]

# Dictionaries - key and value types
scores: dict[str, int] = {"Alice": 100, "Bob": 95}
config: dict[str, str | int | bool] = {"debug": True, "port": 8080}

# Tuples - fixed length and types
point: tuple[float, float] = (1.0, 2.0)
user: tuple[int, str, bool] = (1, "Alice", True)

# Tuples - variable length, same type
coordinates: tuple[float, ...] = (1.0, 2.0, 3.0, 4.0)  # Any number of floats

# Sets
unique_ids: set[int] = {1, 2, 3}

The ellipsis (...) in tuple types means "zero or more elements of this type":

tuple[int, int] - exactly 2 integers
tuple[int, ...] - any number of integers

The "Chicken and Egg" Problem: Forward References

Try to write a method that returns the same type as the class:

class Money:
    def __init__(self, amount: float, currency: str):
        self.amount = amount
        self.currency = currency

    def __add__(self, other: Money) -> Money:  # NameError!
        return Money(self.amount + other.amount, self.currency)

Boom. NameError: name 'Money' is not defined.

Why? Because when Python is executing the class body, the class doesn't exist yet. It's being defined. So other: Money tries to reference a name that doesn't exist.

The Old Fix (Quotes)

The traditional workaround was to use strings:

class Money:
    def __add__(self, other: "Money") -> "Money":  # Works!
        return Money(self.amount + other.amount, self.currency)

Strings aren't evaluated at import time they're just stored. MyPy evaluates them later when the class exists.

But this is ugly and error-prone (typos don't cause errors until type checking).

The New Fix (Future Annotations)

Add one line at the top of your file:

from __future__ import annotations

class Money:
    def __init__(self, amount: float, currency: str):
        self.amount = amount
        self.currency = currency

    def __add__(self, other: Money) -> Money:  # Works!
        return Money(self.amount + other.amount, self.currency)

This directive tells Python: "Treat all type annotations as strings automatically."

Under the hood, Python stores Money as the string "Money", but you write it without quotes. MyPy evaluates it later when type checking.

Why This Matters

Without this, you'd need quotes everywhere:

class TreeNode:
    def __init__(self, value: int):
        self.value = value
        self.left: "TreeNode | None" = None
        self.right: "TreeNode | None" = None

    def add_child(self, node: "TreeNode") -> None:
        pass

With from __future__ import annotations:

from __future__ import annotations

class TreeNode:
    def __init__(self, value: int):
        self.value = value
        self.left: TreeNode | None = None
        self.right: TreeNode | None = None

    def add_child(self, node: TreeNode) -> None:
        pass

Always use this import in every file with type hints. It will become the default behavior in Python 4.0.

The `Any` vs `object` Distinction

When you want to accept "anything," you have two options with very different meanings.

`object` - The Safe Universal Type

def print_anything(value: object) -> None:
    print(value)

print_anything(42)        # OK
print_anything("hello")   # OK
print_anything([1, 2, 3]) # OK

When you type something as object, you're saying:

"I accept any type"
"But I can only do things that work on ANY object"

What can you do with an object?

def process(value: object) -> None:
    print(value)           # OK - all objects can be printed
    str(value)            # OK - all objects can be converted to string
    value == 5            # OK - all objects support equality

    # These are errors:
    value + 5             # Error! Not all objects support addition
    value.some_method()   # Error! Not all objects have this method
    value[0]              # Error! Not all objects are indexable

MyPy knows that object only supports the universal protocol: __str__, __repr__, __eq__, etc.

`Any` - The "Turn Off Type Checking" Escape Hatch

from typing import Any

def process(value: Any) -> None:
    print(value)           # OK
    value + 5             # OK - MyPy assumes you know what you're doing
    value.some_method()   # OK - MyPy trusts you
    value[0]              # OK - MyPy allows everything

Any tells MyPy: "Don't check this variable. I know what I'm doing."

It's dangerous because it disables type safety:

def broken(x: Any) -> int:
    return x + 5  # MyPy says this is fine

broken("hello")  # Runtime error: can't add string and int

When to Use Each

Use object when:

You truly accept any type
You only use universal operations
Example: __eq__ methods that compare with anything

def __eq__(self, other: object) -> bool:
    if not isinstance(other, Money):
        return NotImplemented
    return self.amount == other.amount

Use Any when:

You're working with truly dynamic data (JSON from API)
You're gradually typing legacy code
You're interfacing with untyped libraries
You've exhausted other options

import json

def load_config(path: str) -> Any:  # JSON can be anything
    with open(path) as f:
        return json.load(f)

The Gradual Typing Strategy

When retrofitting types to legacy code:

# Stage 1: Everything is Any
def process_data(data: Any) -> Any:
    result = transform(data)
    return result

# Stage 2: Narrow the outputs
def process_data(data: Any) -> dict[str, int]:
    result = transform(data)
    return result

# Stage 3: Narrow the inputs
def process_data(data: list[str]) -> dict[str, int]:
    result = transform(data)
    return result

# Stage 4: No more Any!

Start with Any everywhere, then gradually replace it with specific types as you understand the code.

MyPy Configuration

Type hints are useless unless you enforce them. That's where MyPy comes in, a static type checker that analyzes your code without running it.

Installing MyPy

pip install mypy

Basic Usage

# Check a single file
mypy my_script.py

# Check a directory
mypy src/

# Check and show error codes
mypy --show-error-codes .

The Configuration File

Create a pyproject.toml file (or mypy.ini) in your project root:

[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true

# Strict mode - the final boss
strict = true

Let's break down what these do:

disallow_untyped_defs = true - Every function MUST have type hints:

# Error: Function is missing a type annotation
def process(data):
    return data.upper()

# OK
def process(data: str) -> str:
    return data.upper()

warn_return_any = true - Functions can't return Any:

import json

# Error: Returning Any from function declared to return int
def load_number() -> int:
    return json.load(open("config.json"))["number"]  # json.load returns Any

# OK
def load_number() -> int:
    value: Any = json.load(open("config.json"))["number"]
    return int(value)  # Explicit conversion

strict = true - Enables ALL strict checks:

# Error: Need to handle None case
def get_first(items: list[str]) -> str:
    return items[0]  # What if items is empty?

# OK
def get_first(items: list[str]) -> str | None:
    return items[0] if items else None

The `NotImplemented` Problem

Remember that arithmetic operations should return NotImplemented, not raise exceptions? MyPy doesn't like this:

def __add__(self, other: Money) -> Money:
    if isinstance(other, Money):
        return Money(self.amount + other.amount, self.currency)
    return NotImplemented  # Error: Incompatible return type!

MyPy says: "You declared this returns Money, but you're returning NotImplemented!"

The fix is to be explicit about both return types:

from typing import NotImplementedType

def __add__(self, other: object) -> Money | NotImplementedType:
    if isinstance(other, Money):
        return Money(self.amount + other.amount, self.currency)
    return NotImplemented

Note we also changed other: Money to other: object—we accept any type and check it at runtime.

Handling Third-Party Libraries

Many packages don't have type stubs. MyPy will complain:

error: Skipping analyzing "requests": module is installed, but missing library stubs

Solutions:

[tool.mypy]
# Option 1: Ignore missing imports globally
ignore_missing_imports = true

# Option 2: Configure per-package
[[tool.mypy.overrides]]
module = "requests.*"
ignore_missing_imports = true

# Option 3: Install type stubs
# pip install types-requests

Python Internals: Reference Cycles and Garbage Collection

aykhlf yassir — Thu, 05 Feb 2026 16:59:44 +0000

Understanding reference counting, garbage collection, and why your objects won't die

The Box vs. The Label

Quick: what does this code do?

a = [1, 2, 3]
b = a
b.append(4)
print(a)  # What prints?

If you're coming from C++ or Java, you might think "I assigned a to b, so they're separate copies. a should still be [1, 2, 3]."

But run this code and you'll see [1, 2, 3, 4].

Why? Because your mental model is wrong.

Variables Are Not Boxes

In C++, a variable is like a box with a name written on it. When you write int x = 5;, you create a box labeled "x" and put the value 5 inside it. When you assign int y = x;, you create a new box labeled "y" and copy the value into it.

Python doesn't work this way.

In Python, objects live in the heap. Variables are not boxes that contain objects; they're labels attached to objects.

a = [1, 2, 3]

What actually happens:

Python creates a list object [1, 2, 3] somewhere in memory
Python creates a label (reference) named a
Python sticks that label onto the object

When you write:

b = a

You're not copying the object. You're creating a second label b and sticking it onto the same object that a is pointing to.

    [1, 2, 3, 4] ← Object in memory
      ↑     ↑
      a     b   ← Two labels on one object

This is why modifying through b affects what you see through a, they're both pointing at the same object.

Proving It With `id()`

Every object in Python has a unique identifier its memory address:

a = [1, 2, 3]
b = a

print(id(a))  # 140234567890
print(id(b))  # 140234567890 - Same address!
print(a is b) # True - Same object

c = [1, 2, 3]
print(id(c))  # 140234567999 - Different address!
print(a is c) # False - Different objects
print(a == c) # True - Same contents

The is operator checks if two labels point to the same object, not whether the objects have the same value.

The Immutability Exception

This model explains why immutable types seem to behave differently:

x = 5
y = x
y = 10
print(x)  # Still 5!

Did the label model break? No! When you write y = 10, you're not modifying the object, you're unsticking the label y from the object 5 and re-sticking it to a different object 10. The original object 5 is unchanged, and x still points to it.

Before y = 10:
    5 ← Object
    ↑
   x,y

After y = 10:
    5 ← Object    10 ← New object
    ↑              ↑
    x              y

Immutable objects can't be modified, so every operation that looks like modification is actually creating a new object and moving the label.

Mutable Default Arguments

Now that you understand the label model, let's look at Python's most infamous gotcha:

def add_passenger(passenger, manifest=[]):
    manifest.append(passenger)
    return manifest

flight1 = add_passenger("Alice")
print(flight1)  # ['Alice']

flight2 = add_passenger("Bob")
print(flight2)  # ['Alice', 'Bob'] - WHAT?!

Why is Alice on Bob's flight?

The Hidden Function Object

When Python executes a def statement, it doesn't just "define a function and forget it." It creates a function object that lives in memory. Default argument values are created once when the function is defined and stored inside that function object.

Let's prove it:

def add_passenger(passenger, manifest=[]):
    manifest.append(passenger)
    return manifest

# The function object has a __defaults__ attribute
print(add_passenger.__defaults__)  # ([],)

# Call it once
add_passenger("Alice")
print(add_passenger.__defaults__)  # (['Alice'],)

# Call it again
add_passenger("Bob")
print(add_passenger.__defaults__)  # (['Alice', 'Bob'],)

The default list [] is created once, when def executes. Every call to the function that doesn't provide a manifest argument reuses that same list object.

The Execution Timeline

# DEFINITION TIME (happens once)
def add_passenger(passenger, manifest=[]):  # Create empty list, store in __defaults__
    manifest.append(passenger)
    return manifest

# CALL TIME 1
add_passenger("Alice")  # No manifest provided, use default (the list in __defaults__)
# The list is now ['Alice']

# CALL TIME 2  
add_passenger("Bob")    # No manifest provided, use default (SAME list!)
# The list is now ['Alice', 'Bob']

The Idiomatic Fix

Never use mutable objects as default arguments. Use None as a sentinel:

def add_passenger(passenger, manifest=None):
    if manifest is None:
        manifest = []  # Create a NEW list for each call
    manifest.append(passenger)
    return manifest

flight1 = add_passenger("Alice")
print(flight1)  # ['Alice']

flight2 = add_passenger("Bob")
print(flight2)  # ['Bob'] - Separate list!

Now each call that doesn't provide manifest creates a fresh, independent list.

When to Use Mutable Defaults (Intentionally)

There are rare cases where you want to preserve state across calls:

def cache_result(key, cache={}):
    if key not in cache:
        print(f"Computing {key}...")
        cache[key] = expensive_computation(key)
    return cache[key]

cache_result(5)  # Computing 5...
cache_result(5)  # (no output - cached!)

But this is almost always better expressed with a class or explicit cache variable.

The Life and Death of an Object: Reference Counting

So we know objects are created in memory and variables are labels pointing to them. But when does an object die?

The Reference Counter

Every Python object has a hidden field called ob_refcnt, a counter tracking how many labels are currently stuck to it.

typedef struct {
    Py_ssize_t ob_refcnt;   // Reference count
    PyTypeObject *ob_type;  // Type
    // ... actual data
} PyObject;

The rules are simple:

Create a label → Count +1
Remove a label → Count -1
Count reaches 0 → Immediate death

Let's trace the life cycle:

import sys

x = object()  # Create object, refcount = 1
print(sys.getrefcount(x))  # 2 (why? read on...)

y = x  # New label, refcount = 2 (+ 1 from getrefcount call = 3)
print(sys.getrefcount(x))  # 3

del y  # Remove label, refcount = 2
print(sys.getrefcount(x))  # 2

del x  # Refcount = 1 (from getrefcount), then 0 when function returns
# Object is immediately destroyed

The `sys.getrefcount()` Trap

Notice the reference count is always higher than expected? This is because calling sys.getrefcount(x) creates a temporary reference when passing x as an argument!

x = []
# Actual refs: x
print(sys.getrefcount(x))  # 2
# Why 2? Because during the call, refs are: x, argument to getrefcount()

The real count is always getrefcount(x) - 1.

What Creates References?

Understanding what increases the reference count is critical:

import sys

obj = object()
print(sys.getrefcount(obj))  # 2 (variable + function arg)

# Assignment creates a reference
another = obj
print(sys.getrefcount(obj))  # 3

# Lists/tuples/dicts create references
lst = [obj]
print(sys.getrefcount(obj))  # 4

# Function arguments create temporary references
def foo(x):
    print(sys.getrefcount(x))  # +1 from function call

foo(obj)  # Will print 5

# Deleting removes references
del another
del lst
print(sys.getrefcount(obj))  # Back to 2

Immediate Destruction

The beauty of reference counting is deterministic cleanup. The moment the last reference disappears, the object dies:

class Mortal:
    def __init__(self, name):
        self.name = name
        print(f"{name} is born")

    def __del__(self):
        # Called when refcount hits 0
        print(f"{name} dies")

print("Creating object...")
x = Mortal("Socrates")  # Socrates is born

print("Deleting reference...")
del x  # Socrates dies - immediately!

print("After deletion")

This is why Python doesn't need explicit free() or delete calls like C/C++. Objects clean up automatically when they're no longer needed.

The Reference Cycles Problem

Reference counting sounds perfect and intuitive. But it has a fatal weakness:

class Node:
    def __init__(self, name):
        self.name = name
        self.partner = None

    def __del__(self):
        print(f"Deleting {self.name}")

# Create two nodes
alice = Node("Alice")
bob = Node("Bob")

# Create a circular reference
alice.partner = bob
bob.partner = alice

# Delete our references
del alice
del bob

# ... crickets ...
# __del__ is never called!

What happened? Let's trace the references:

Before del:
    Node("Alice") ← alice variable
          ↕ partner
    Node("Bob") ← bob variable

After del alice, del bob:
    Node("Alice") 
          ↕ partner
    Node("Bob")

Even after we delete the variables alice and bob, the Node objects still hold references to each other. Their reference counts are still 1 (from the partner attribute), so they never get destroyed.

These objects are now unreachable from our code (we have no variables pointing to them), but they're still in memory, causing a memory leak.

The Real-World Impact

This isn't just theoretical. Reference cycles are common:

# Parent-child relationships
class Parent:
    def __init__(self):
        self.children = []

class Child:
    def __init__(self, parent):
        self.parent = parent
        parent.children.append(self)

# Event listeners
class Button:
    def __init__(self):
        self.click_handler = None

    def on_click(self, handler):
        self.click_handler = handler

# Handler holds button, button holds handler
button = Button()
button.on_click(lambda: print(f"Clicked {button}"))

# Data structures
class TreeNode:
    def __init__(self, value):
        self.value = value
        self.parent = None
        self.left = None
        self.right = None

root = TreeNode(1)
root.left = TreeNode(2)
root.left.parent = root  # Cycle!

The Garbage Collector

Python's solution is the Generational Garbage Collector, which exists specifically to find and destroy reference cycles.

How It Works

The GC periodically:

Scans all objects looking for those that reference each other
Builds a graph of references
Finds cycles (strongly connected components)
Determines if the cycle is reachable from any external reference
If not reachable, destroys the entire cycle

You can trigger it manually:

import gc

class Node:
    def __init__(self, name):
        self.name = name
        self.partner = None

    def __del__(self):
        print(f"Deleting {self.name}")

alice = Node("Alice")
bob = Node("Bob")
alice.partner = bob
bob.partner = alice

del alice, bob
print("After del...")

# Force garbage collection
gc.collect()  # Deleting Alice
              # Deleting Bob
print("After gc.collect()")

Generational Hypothesis

The GC is "generational" because it's based on an empirical observation: most objects die young.

Python divides objects into three generations:

Generation 0: New objects (checked frequently)
Generation 1: Objects that survived one GC pass (checked less often)
Generation 2: Long-lived objects (checked rarely)

import gc

# See the current thresholds
print(gc.get_threshold())  # (700, 10, 10)
# Meaning: Run gen0 after 700 allocations,
#          Run gen1 after 10 gen0 collections,
#          Run gen2 after 10 gen1 collections

# See collection stats
print(gc.get_count())  # (453, 5, 2)
# Current counts in each generation

The Cost of Cycles

The garbage collector isn't free. It has to:

Pause your program to scan objects
Build reference graphs
Traverse cycles

This is why avoiding cycles improves performance:

# Slower: Creates cycles
class Child:
    def __init__(self, parent):
        self.parent = parent
        parent.children.append(self)

# Faster: No cycles
class Child:
    def __init__(self, parent_name):
        self.parent_name = parent_name  # Store name, not reference

Weak References

Imagine building a cache:

cache = {}

def get_user(user_id):
    if user_id not in cache:
        cache[user_id] = load_from_database(user_id)
    return cache[user_id]

Problem: objects in cache never die. Even if nobody else needs them, the cache keeps them alive. This is a memory leak.

Strong vs. Weak References

Think of references like this:

Strong Reference (normal):

Like holding a dog's leash
The dog cannot leave while you hold the leash
The dog exists because you're holding it

Weak Reference:

Like pointing at a dog
You don't prevent the dog from leaving
If the owner (strong reference) leaves, the dog disappears
You're now pointing at nothing (None)

The `weakref` Module

import weakref

class HeavyObject:
    def __init__(self, data):
        self.data = data

    def __del__(self):
        print(f"Deleting {self.data}")

# Strong reference
obj = HeavyObject("important")
print(obj.data)  # "important"

# Weak reference
weak = weakref.ref(obj)
print(weak())  # <__main__.HeavyObject object> - obj still alive

# Delete strong reference
del obj  # Deleting important

# Weak reference now points to nothing
print(weak())  # None

The weak reference doesn't count toward ob_refcnt. When the last strong reference dies, the object is destroyed, even if weak references still exist.

The Optimized Cache: `WeakValueDictionary`

import weakref

cache = weakref.WeakValueDictionary()

class User:
    def __init__(self, user_id, name):
        self.user_id = user_id
        self.name = name

    def __del__(self):
        print(f"User {self.name} deleted from memory")

# Add to cache
user = User(1, "Alice")
cache[1] = user

print(cache[1].name)  # "Alice" - cache works

# Delete the strong reference
del user  # User Alice deleted from memory

# Cache automatically removed the entry!
print(1 in cache)  # False

When user was deleted, its reference count hit 0 (the cache only held a weak reference). The object was destroyed, and WeakValueDictionary automatically removed the entry.

This is how you build caches that don't leak memory.

Use Cases for Weak References

Caches: Store objects without preventing their cleanup
Observer patterns: Listeners shouldn't keep subjects alive
Circular references: Parent doesn't prevent child GC

# Observer pattern with weak refs
class Subject:
    def __init__(self):
        self._observers = []

    def attach(self, observer):
        # Store weak reference, not strong
        self._observers.append(weakref.ref(observer))

    def notify(self):
        # Filter out dead references
        self._observers = [obs for obs in self._observers if obs() is not None]
        for obs_ref in self._observers:
            obs = obs_ref()
            if obs is not None:
                obs.update()

When NOT to Use Weak References

Weak references have limitations:

# Can't create weak refs to basic types
try:
    weak = weakref.ref(42)
except TypeError:
    print("Can't weakref int")

try:
    weak = weakref.ref("hello")
except TypeError:
    print("Can't weakref str")

# Weak refs add overhead
# Don't use them everywhere - only where you need them

Recommended Resource

I highly recommend Nina Zakharenko's "Memory Management in Python" talk from PyCon 2016.

Watch on YouTube

Summary:

We've learned how Python manages object lifecycles:

Mental Models

Variables are labels, not boxes → b = a creates two labels on one object
Mutable defaults are shared → Use None sentinel pattern
Reference counting is immediate → Objects die when count hits 0

The Reference Counting System

Every object has ob_refcnt tracking references
Assignment increments count, deletion decrements
Count = 0 triggers immediate destruction via __del__

The Garbage Collector

Purpose: Find and destroy reference cycles
Strategy: Generational collection (young objects die fast)
Trigger: Automatically or via gc.collect()
Cost: Pause program to scan objects

Weak References

Strong reference: Keeps object alive (normal)
Weak reference: Points to object without preventing cleanup
WeakValueDictionary: Auto-cleaning cache
Use for: Caches, observers, avoiding cycles

The Professional Checklist

When designing classes:

# Avoid cycles in data structures
class TreeNode:
    def __init__(self, value):
        self.value = value
        self.children = []
        # DON'T store self.parent - creates cycle
        # Store parent_id or use weakref instead

# Use weak refs for caches
import weakref
class ResourceManager:
    def __init__(self):
        self._cache = weakref.WeakValueDictionary()

# Clean up resources explicitly
class FileHandler:
    def __enter__(self):
        return self

    def __exit__(self, *args):
        self.cleanup()  # Don't rely on __del__

When writing functions:

# NEVER use mutable defaults
def process_items(items, buffer=[]):  # BAD
    pass

def process_items(items, buffer=None):  # GOOD
    if buffer is None:
        buffer = []

Python Internals: Operator Protocols

aykhlf yassir — Wed, 04 Feb 2026 19:29:05 +0000

How to make your custom objects behave exactly like native types through protocol adherence

The "Native Citizen" Test

Here's how you spot a junior Python developer:

# Junior code
m1 = Money(10, "USD")
m2 = Money(5, "USD")
result = m1.add(m2)  # Java flashbacks intensify

And here's a senior:

# Senior code
m1 = Money(10, "USD")
m2 = Money(5, "USD")
result = m1 + m2  # Feels like native Python

The difference isn't just aesthetics it's understanding Python's protocol-oriented design philosophy.

Python doesn't care if your class inherits from int or float. It only cares if your object behaves like a number. This is duck typing at the language level: if it implements the arithmetic protocol, it's a number. If it implements the sequence protocol, it's a sequence.

Today, we're going to make our custom classes first-class citizens of the Python ecosystem by implementing the magic methods that let them integrate seamlessly with operators, loops, and built-in functions.

The Arithmetic Negotiation: `add` vs `radd`

Let's build a Money class that supports addition:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __add__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot add different currencies")
            return Money(self.amount + other.amount, self.currency)
        return NotImplemented

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

m1 = Money(10, "USD")
m2 = Money(5, "USD")
print(m1 + m2)  # Money(15, 'USD')

Great! But what about this?

m = Money(10, "USD")
result = m + 5  # What should this do?

We could support it:

def __add__(self, other):
    if isinstance(other, Money):
        if self.currency != other.currency:
            raise ValueError("Cannot add different currencies")
        return Money(self.amount + other.amount, self.currency)
    if isinstance(other, (int, float)):
        return Money(self.amount + other, self.currency)
    return NotImplemented

Now m + 5 works. But what about 5 + m?

m = Money(10, "USD")
result = 5 + m  # TypeError: unsupported operand type(s) for +: 'int' and 'Money'

The Dispatch Sequence

Here's what actually happens when you write a + b:

Python calls a.__add__(b)
If that returns NotImplemented, Python calls b.__radd__(a)
If that also returns NotImplemented, Python raises TypeError

So when you do 5 + m:

Python calls (5).__add__(m) → int.__add__ doesn't know about Money, returns NotImplemented
Python calls m.__radd__(5) → We haven't implemented this yet!

The solution is reflected operations:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __add__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot add different currencies")
            return Money(self.amount + other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount + other, self.currency)
        return NotImplemented

    def __radd__(self, other):
        # Reflected addition: other + self
        # Just delegate to __add__ since addition is commutative
        return self.__add__(other)

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

m = Money(10, "USD")
print(m + 5)   # Money(15, 'USD')
print(5 + m)   # Money(15, 'USD') - now works!

The Critical Distinction: `NotImplemented` vs `NotImplementedError`

This is where juniors crash and seniors cooperate:

# WRONG - Don't do this!
def __add__(self, other):
    if isinstance(other, Money):
        return Money(self.amount + other.amount, self.currency)
    raise NotImplementedError("Cannot add Money and " + type(other).__name__)

# RIGHT - Be a good citizen
def __add__(self, other):
    if isinstance(other, Money):
        return Money(self.amount + other.amount, self.currency)
    return NotImplemented

What's the difference?

NotImplementedError: An exception. Your program crashes. You're saying "I can't do this and nobody else can either."
NotImplemented: A sentinel value. You're saying "I don't know how to handle this, but maybe the other object does."

When you return NotImplemented, you're participating in Python's cooperative operator dispatch. You're giving the other object a chance to handle the operation.

class Discount:
    def __init__(self, percent):
        self.percent = percent

    def __radd__(self, other):
        if isinstance(other, Money):
            discount_amount = other.amount * (self.percent / 100)
            return Money(other.amount - discount_amount, other.currency)
        return NotImplemented

    def __repr__(self):
        return f"Discount({self.percent}%)"

m = Money(100, "USD")
d = Discount(10)

# This works because Money.__add__ returns NotImplemented,
# so Python tries Discount.__radd__
result = m + d  # Money(90.0, 'USD')

If Money.__add__ had raised NotImplementedError, this would never work.

The Full Arithmetic Protocol

For a complete numeric type, you need:

__add__, __radd__ - Addition (a + b)
__sub__, __rsub__ - Subtraction (a - b)
__mul__, __rmul__ - Multiplication (a * b)
__truediv__, __rtruediv__ - Division (a / b)
__floordiv__, __rfloordiv__ - Floor division (a // b)
__mod__, __rmod__ - Modulo (a % b)
__pow__, __rpow__ - Exponentiation (a ** b)

And the in-place variants:

__iadd__ - In-place addition (a += b)
__isub__, __imul__, etc.

class Money:
    # ... __init__, __repr__ ...

    def __add__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot add different currencies")
            return Money(self.amount + other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount + other, self.currency)
        return NotImplemented

    def __radd__(self, other):
        return self.__add__(other)

    def __sub__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot subtract different currencies")
            return Money(self.amount - other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount - other, self.currency)
        return NotImplemented

    def __rsub__(self, other):
        # Subtraction is NOT commutative: other - self
        if isinstance(other, (int, float)):
            return Money(other - self.amount, self.currency)
        return NotImplemented

    def __mul__(self, other):
        if isinstance(other, (int, float)):
            return Money(self.amount * other, self.currency)
        return NotImplemented

    def __rmul__(self, other):
        return self.__mul__(other)

The Comparison Shortcut: `@total_ordering`

Now we need comparison operators. All six of them:

__eq__ (equality: ==)
__ne__ (inequality: !=)
__lt__ (less than: <)
__le__ (less than or equal: <=)
__gt__ (greater than: >)
__ge__ (greater than or equal: >=)

That's a lot of boilerplate. Fortunately, Python provides a shortcut:

from functools import total_ordering

@total_ordering
class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __eq__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        return self.amount == other.amount and self.currency == other.currency

    def __lt__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        if self.currency != other.currency:
            raise ValueError("Cannot compare different currencies")
        return self.amount < other.amount

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

# Now ALL of these work:
m1 = Money(10, "USD")
m2 = Money(20, "USD")

print(m1 < m2)   # True
print(m1 <= m2)  # True
print(m1 > m2)   # False
print(m1 >= m2)  # False
print(m1 == m2)  # False
print(m1 != m2)  # True

The @total_ordering decorator automatically generates __le__, __gt__, __ge__, and __ne__ based on your __eq__ and __lt__ implementations.

The Mathematical Inference

How does it work? Mathematics:

a <= b is equivalent to a < b or a == b
a > b is equivalent to not (a <= b)
a >= b is equivalent to not (a < b)
a != b is equivalent to not (a == b)

The decorator generates these derived methods automatically.

The Hashability Warning

If your object is hashable (implements __hash__), you must be careful with equality.

Never allow this if your object is hashable:

m = Money(10, "USD")
print(m == 10)  # Should return False or NotImplemented, NEVER True

Why? Because if Money(10, "USD") == 10 returns True, but hash(Money(10, "USD")) != hash(10), you violate the hashability contract:

If a == b, then hash(a) must equal hash(b)

This is why our __eq__ returns NotImplemented for non-Money objects:

def __eq__(self, other):
    if not isinstance(other, Money):
        return NotImplemented  # Let Python figure it out
    return self.amount == other.amount and self.currency == other.currency

The Sequence Protocol: Free Features Through Convention

Here's something magical. Implement just two methods, and Python gives you iteration, indexing, slicing, and membership testing for free.

class Vector:
    def __init__(self, *components):
        self._components = list(components)

    def __len__(self):
        return len(self._components)

    def __getitem__(self, index):
        return self._components[index]

    def __repr__(self):
        return f"Vector{tuple(self._components)}"

v = Vector(1, 2, 3, 4, 5)

# Indexing - we implemented this explicitly
print(v[0])     # 1
print(v[-1])    # 5

# Slicing - we get this for FREE
print(v[1:3])   # [2, 3]

# Iteration - we get this for FREE
for component in v:
    print(component)  # 1, 2, 3, 4, 5

# Membership - we get this for FREE
print(3 in v)   # True
print(10 in v)  # False

# Length - we implemented this explicitly
print(len(v))   # 5

How is this possible? When you write for x in v:, Python first checks if v has an __iter__ method. If not, it falls back to the sequence protocol:

Call v[0], assign to x, execute loop body
Call v[1], assign to x, execute loop body
Keep incrementing until v[n] raises IndexError
Catch the IndexError and stop iteration

Similarly, when you write 3 in v, Python:

Checks if v has a __contains__ method
If not, iterates through v checking item == 3 for each item

Making It More Pythonic

We can add more magic methods to make our Vector even more powerful:

class Vector:
    def __init__(self, *components):
        self._components = list(components)

    def __len__(self):
        return len(self._components)

    def __getitem__(self, index):
        return self._components[index]

    def __setitem__(self, index, value):
        self._components[index] = value

    def __add__(self, other):
        if isinstance(other, Vector):
            if len(self) != len(other):
                raise ValueError("Vectors must have same length")
            return Vector(*(a + b for a, b in zip(self, other)))
        return NotImplemented

    def __mul__(self, scalar):
        if isinstance(scalar, (int, float)):
            return Vector(*(x * scalar for x in self))
        return NotImplemented

    def __rmul__(self, scalar):
        return self.__mul__(scalar)

    def __repr__(self):
        return f"Vector{tuple(self._components)}"

v1 = Vector(1, 2, 3)
v2 = Vector(4, 5, 6)

print(v1 + v2)    # Vector(5, 7, 9)
print(v1 * 2)     # Vector(2, 4, 6)
print(3 * v1)     # Vector(3, 6, 9)

# We can modify vectors
v1[0] = 10
print(v1)         # Vector(10, 2, 3)

The Iterator Protocol

If you want more control over iteration, you can implement __iter__:

class Vector:
    def __init__(self, *components):
        self._components = list(components)

    def __len__(self):
        return len(self._components)

    def __getitem__(self, index):
        return self._components[index]

    def __iter__(self):
        # Return an iterator object
        return iter(self._components)

    def __repr__(self):
        return f"Vector{tuple(self._components)}"

Now iteration uses __iter__ instead of repeated __getitem__ calls, which can be more efficient for large sequences.

The Missing Link: How `defaultdict` Actually Works

You've probably used collections.defaultdict:

from collections import defaultdict

counts = defaultdict(int)
counts['a'] += 1  # No KeyError!
counts['b'] += 1
print(counts)  # defaultdict(<class 'int'>, {'a': 1, 'b': 1})

But how does it work? Through a special hook called __missing__.

The `missing` Protocol

The __missing__ method is exclusive to dict subclasses. It's called automatically when a key lookup fails:

class DefaultDict(dict):
    def __init__(self, default_factory):
        super().__init__()
        self.default_factory = default_factory

    def __missing__(self, key):
        # This is called when self[key] raises KeyError
        if self.default_factory is None:
            raise KeyError(key)

        # Create the default value
        value = self.default_factory()

        # Store it in the dict
        self[key] = value

        # Return it
        return value

# Now we've built our own defaultdict!
counts = DefaultDict(int)
counts['a'] += 1  # __missing__ creates 0, then +=1 makes it 1
counts['b'] += 1
print(counts)  # {'a': 1, 'b': 1}

The Execution Flow

When you do counts['a'] += 1 and 'a' doesn't exist:

Python calls counts.__getitem__('a')
The inherited dict.__getitem__ looks for 'a', doesn't find it
dict.__getitem__ calls counts.__missing__('a')
__missing__ creates 0, stores it at counts['a'], returns 0
Python increments the returned 0 to get 1
Python calls counts.__setitem__('a', 1)

Critical detail: __missing__ is only called by __getitem__. It's not called by .get():

counts = DefaultDict(int)
print(counts['x'])      # 0 - __missing__ called
print(counts.get('y'))  # None - __missing__ NOT called

A Practical Use Case

Let's build a case-insensitive dictionary:

class CaseInsensitiveDict(dict):
    def __setitem__(self, key, value):
        # Always store with lowercase keys
        super().__setitem__(key.lower(), value)

    def __getitem__(self, key):
        # Always lookup with lowercase keys
        return super().__getitem__(key.lower())

    def __missing__(self, key):
        # Provide a helpful error message
        raise KeyError(f"Key '{key}' not found (case-insensitive)")

config = CaseInsensitiveDict()
config['Server'] = 'localhost'
print(config['SERVER'])  # 'localhost'
print(config['server'])  # 'localhost'
print(config['SeRvEr'])  # 'localhost'

Or a tree structure that auto-creates nested dictionaries:

class TreeDict(dict):
    def __missing__(self, key):
        # Create a new TreeDict for missing keys
        value = TreeDict()
        self[key] = value
        return value

tree = TreeDict()
tree['a']['b']['c'] = 42  # No KeyErrors!
print(tree)  # {'a': {'b': {'c': 42}}}

The Complete Protocol Checklist

Here's your reference for making objects feel native:

Arithmetic Protocol

__add__, __radd__ → +
__sub__, __rsub__ → -
__mul__, __rmul__ → *
__truediv__, __rtruediv__ → /
__floordiv__, __rfloordiv__ → //
__mod__, __rmod__ → %
__pow__, __rpow__ → **

Always return NotImplemented for unsupported types, never raise NotImplementedError.

Comparison Protocol

__eq__ → ==
__ne__ → !=
__lt__ → <
__le__ → <=
__gt__ → >
__ge__ → >=

Use @total_ordering to generate most of these from __eq__ and __lt__.

Sequence Protocol

__len__ → len(obj)
__getitem__ → obj[key]
__setitem__ → obj[key] = value
__delitem__ → del obj[key]
__contains__ → x in obj (optional if __getitem__ + __iter__ exist)
__iter__ → for x in obj (optional, falls back to __getitem__)

Dictionary Protocol

Inherit from dict
__missing__ → handles missing key lookups

Summary:

Today we've learned how to make custom objects integrate seamlessly with Python's syntax:

Key Principles

Return NotImplemented, not NotImplementedError → Enables cooperative dispatch
Implement reflected operations (__radd__, etc.) → Supports 5 + obj in addition to obj + 5
Use @total_ordering → Generate six comparison methods from two
Implement __len__ + __getitem__ → Get iteration, slicing, and membership for free
Use __missing__ in dict subclasses → Intercept missing key lookups

The Professional Template

from functools import total_ordering

@total_ordering
class Money:
    __slots__ = ['_amount', '_currency']

    def __init__(self, amount, currency):
        object.__setattr__(self, '_amount', amount)
        object.__setattr__(self, '_currency', currency)

    # Representation
    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

    # Properties
    @property
    def amount(self):
        return self._amount

    @property
    def currency(self):
        return self._currency

    # Arithmetic
    def __add__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot add different currencies")
            return Money(self.amount + other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount + other, self.currency)
        return NotImplemented

    def __radd__(self, other):
        return self.__add__(other)

    def __sub__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot subtract different currencies")
            return Money(self.amount - other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount - other, self.currency)
        return NotImplemented

    def __mul__(self, other):
        if isinstance(other, (int, float)):
            return Money(self.amount * other, self.currency)
        return NotImplemented

    def __rmul__(self, other):
        return self.__mul__(other)

    # Comparison (total_ordering generates <=, >, >=, !=)
    def __eq__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        return self.amount == other.amount and self.currency == other.currency

    def __lt__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        if self.currency != other.currency:
            raise ValueError("Cannot compare different currencies")
        return self.amount < other.amount

    # Hashability
    def __hash__(self):
        return hash((self.amount, self.currency))

Python Internals: init is Not the Constructor

aykhlf yassir — Tue, 03 Feb 2026 19:23:58 +0000

Mastering __new__, __repr__, and __hash__

The Constructor Myth

Pop quiz: What method creates a Python object?

If you answered __init__, you're in good company, and you're wrong!

class Point:
    def __init__(self, x, y):
        print(f"__init__ called with {self}")
        self.x = x
        self.y = y

p = Point(3, 4)
# Output: __init__ called with <__main__.Point object at 0x7f8b4c>

Notice that inside __init__, we already have self. The object already exists. So what actually created it?

The answer is __new__, a method so fundamental that Python calls it automatically, and most developers never even know it exists.

The Truth About Object Creation

Here's what actually happens when you call Point(3, 4):

class Point:
    def __new__(cls, x, y):
        print(f"__new__ called with class {cls}")
        instance = super().__new__(cls)
        print(f"__new__ created {instance}")
        return instance

    def __init__(self, x, y):
        print(f"__init__ called with {self}")
        self.x = x
        self.y = y

p = Point(3, 4)
# Output:
# __new__ called with class <class '__main__.Point'>
# __new__ created <__main__.Point object at 0x7f8b4c>
# __init__ called with <__main__.Point object at 0x7f8b4c>

The execution flow is:

__new__(cls, ...) - The Architect
- Allocates memory for a new instance
- Returns the newly created object
- Receives the class as first parameter, not an instance
__init__(self, ...) - The Interior Decorator
- Receives the instance created by __new__
- Populates it with data
- Returns None (always!)

The Mental Model:

Point(3, 4)
    ↓
__new__(Point, 3, 4) → creates empty object → instance
    ↓
__init__(instance, 3, 4) → populates instance.x, instance.y
    ↓
return instance

99% of the time, you don't need to touch __new__. Python's default implementation (inherited from object) handles memory allocation perfectly. But there's one critical use case where __new__ is not just useful, it's essential.

Deep Dive: The Singleton Pattern

Imagine you're building a database connection pool, a configuration manager, or a logger. You want exactly one instance of the class to exist, no matter how many times someone calls the constructor.

# What we want:
db1 = Database()
db2 = Database()
print(db1 is db2)  # Should be True!

Can we do this with __init__? Let's try:

class Database:
    _instance = None

    def __init__(self):
        if Database._instance is not None:
            # Too late! Memory is already allocated
            # We can't "un-create" this object
            pass
        Database._instance = self

db1 = Database()
db2 = Database()
print(db1 is db2)  # False - we created two objects!

The problem: by the time __init__ runs, __new__ has already allocated memory for a new object. We can't prevent the creation—only configure what's already been created.

The Solution: Intercept at `new`

class Database:
    _instance = None

    def __new__(cls):
        if cls._instance is None:
            print("Creating the one true Database instance...")
            cls._instance = super().__new__(cls)
        else:
            print("Returning existing instance...")
        return cls._instance

    def __init__(self):
        print(f"__init__ called on {id(self)}")

db1 = Database()
# Output:
# Creating the one true Database instance...
# __init__ called on 140234567890

db2 = Database()
# Output:
# Returning existing instance...
# __init__ called on 140234567890

print(db1 is db2)  # True!
print(id(db1), id(db2))  # Same memory address

What's happening:

First call: _instance is None, so we call super().__new__(cls) to actually allocate memory
We cache this instance in _instance
Second call: _instance exists, so we return the cached object
__init__ still runs every time (be careful with this!)

The Critical Detail: `super().new(cls)`

This line is calling object.__new__(cls), the base implementation that actually talks to Python's memory allocator. You're delegating the "real" work of memory allocation to Python's core object class.

Do NOT do this:

def __new__(cls):
    if cls._instance is None:
        cls._instance = cls()  # RECURSION ERROR!
    return cls._instance

Calling cls() inside __new__ calls __new__ again, which calls __new__ again... infinite recursion.

Singleton Best Practice

If __init__ shouldn't run multiple times, use a flag:

class Database:
    _instance = None
    _initialized = False

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

    def __init__(self, connection_string="localhost"):
        if not Database._initialized:
            self.connection_string = connection_string
            Database._initialized = True
            print(f"Connected to {connection_string}")

db1 = Database("prod-server")  # Connected to prod-server
db2 = Database("dev-server")   # (no output - already initialized)
print(db1.connection_string)   # prod-server

The Representation Layer: `str` vs `repr`

You've built a beautiful class. Now it looks like this in the debugger:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

m = Money(10, "USD")
print(m)  # <__main__.Money object at 0x7f8b4c>

Useless. Let's fix it.

The Two Faces of Representation

Python has two methods for converting objects to strings:

__str__ - The User-Friendly Version

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __str__(self):
        return f"${self.amount} {self.currency}"

m = Money(10, "USD")
print(m)  # $10 USD
print(str(m))  # $10 USD

__repr__ - The Developer Version

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"Money({self.amount}, {self.currency})"

m = Money(10, "USD")
print(repr(m))  # Money(10, USD)
print([m])  # [Money(10, USD)] - repr is used in containers!

The Golden Rule of `repr`

The output should be valid Python code that recreates the object.

This is often stated as: eval(repr(obj)) == obj

m = Money(10, "USD")
code = repr(m)  # "Money(10, USD)"
m2 = eval(code)  # Recreate the object!
print(m2.amount)  # 10

Wait... did that actually work? Let's test it:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"Money({self.amount}, {self.currency})"

m = Money(10, "USD")
print(repr(m))  # Money(10, USD)
eval(repr(m))  # NameError: name 'USD' is not defined

The problem: USD without quotes isn't a string—it's treated as a variable name!

The `!r` Trick

Python's f-strings have a special formatter that automatically calls repr() on values:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

m = Money(10, "USD")
print(repr(m))  # Money(10, 'USD') - notice the quotes!
m2 = eval(repr(m))  # Works perfectly!

The !r format specifier calls repr() on each value, which for strings adds the quotes. This ensures the output is valid Python syntax.

Pro comparison:

amount = 10
currency = "USD"

# Without !r
print(f"Money({amount}, {currency})")  # Money(10, USD)

# With !r
print(f"Money({amount!r}, {currency!r})")  # Money(10, 'USD')

When to Use Which

__repr__: Always implement this. It's used by debuggers, logs, and the interactive interpreter. Make it unambiguous.
__str__: Optional. Only implement if you need a user-friendly format. If not defined, Python falls back to __repr__.

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

    def __str__(self):
        symbols = {"USD": "$", "EUR": "€", "GBP": "£"}
        symbol = symbols.get(self.currency, self.currency)
        return f"{symbol}{self.amount}"

m = Money(10, "USD")
print(str(m))   # $10 (user-friendly)
print(repr(m))  # Money(10, 'USD') (code-like)
print(m)        # $10 (print uses str)
print([m])      # [Money(10, 'USD')] (containers use repr)

The Hashability Contract: Making Objects Dictionary Keys

You've probably used strings and tuples as dictionary keys:

cache = {}
cache["user:123"] = {"name": "Alice"}  # String key - works
cache[(1, 2)] = "point"  # Tuple key - works

But try this:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

p = Point(1, 2)
cache = {}
cache[p] = "point"  # TypeError: unhashable type: 'Point'

Why can't we use our custom object as a key? Because it's not hashable.

What Does Hashable Mean?

To be used as a dictionary key or stored in a set, an object must:

Have a __hash__ method that returns an integer
Have an __eq__ method to check equality
Follow the hashability contract

The Hashability Contract

Rule 1: Equal objects must have equal hashes

If a == b, then hash(a) MUST equal hash(b)

Rule 2: The hash must never change

Once created, an object's hash must remain constant for its entire lifetime. This is why lists aren't hashable—you can modify them!

# This is why lists fail:
lst = [1, 2, 3]
hash(lst)  # TypeError: unhashable type: 'list'

# But tuples work:
tpl = (1, 2, 3)
hash(tpl)  # 529344067295497451

Implementing Hashability

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __eq__(self, other):
        if not isinstance(other, Point):
            return NotImplemented
        return self.x == other.x and self.y == other.y

    def __hash__(self):
        return hash((self.x, self.y))

p1 = Point(1, 2)
p2 = Point(1, 2)
p3 = Point(3, 4)

print(p1 == p2)  # True
print(hash(p1) == hash(p2))  # True - contract satisfied!

cache = {p1: "origin"}
print(cache[p2])  # "origin" - found it using p2!

Why Delegate to a Tuple?

The line return hash((self.x, self.y)) is the idiomatic way to hash objects. Here's why:

Tuples are immutable - Their hash is guaranteed stable
Python's tuple hash is well-designed - It combines element hashes efficiently
It's simple - You don't have to write your own hash combining logic

Under the hood, Python's tuple hash does something like:

# Simplified version of what Python does
def hash_tuple(items):
    result = 0x345678
    for item in items:
        result = (1000003 * result) ^ hash(item)
    return result

But you don't need to know that—just pack your state into a tuple and let Python handle it.

The `NotImplemented` Pattern

Notice this line in __eq__:

if not isinstance(other, Point):
    return NotImplemented

Don't return False here! Returning NotImplemented tells Python "I don't know how to compare with this type—ask the other object."

class Point:
    def __eq__(self, other):
        if not isinstance(other, Point):
            return NotImplemented
        return self.x == other.x and self.y == other.y

p = Point(1, 2)
print(p == 5)  # False (Python tries both p.__eq__(5) and (5).__eq__(p))

If you returned False instead, you'd be claiming "a Point is definitely not equal to an integer," which might not be true if someone subclasses Point and adds custom comparison logic.

The Immutability Trap

Remember: hashable objects should be immutable. If you allow modification, weird things happen:

class MutablePoint:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __eq__(self, other):
        if not isinstance(other, MutablePoint):
            return NotImplemented
        return self.x == other.x and self.y == other.y

    def __hash__(self):
        return hash((self.x, self.y))

p = MutablePoint(1, 2)
cache = {p: "original"}

print(cache[p])  # "original" - works

# Mutate the object
p.x = 99

# Now the hash changed!
print(cache[p])  # KeyError: MutablePoint object not found

The object is now "lost" in the dictionary because its hash changed. The dictionary is looking in the wrong bucket!

Best practice: If you implement __hash__, make your object immutable using __slots__ and properties:

class ImmutablePoint:
    __slots__ = ['_x', '_y']

    def __init__(self, x, y):
        object.__setattr__(self, '_x', x)
        object.__setattr__(self, '_y', y)

    @property
    def x(self):
        return self._x

    @property
    def y(self):
        return self._y

    def __setattr__(self, name, value):
        raise AttributeError("ImmutablePoint is immutable")

    def __eq__(self, other):
        if not isinstance(other, ImmutablePoint):
            return NotImplemented
        return self.x == other.x and self.y == other.y

    def __hash__(self):
        return hash((self.x, self.y))

    def __repr__(self):
        return f"ImmutablePoint({self.x!r}, {self.y!r})"

Summary: The Professional Object Checklist

Today we've learned the lifecycle methods that make Python objects behave like first-class types:

Creation & Representation

__new__(cls, ...) creates the object; __init__(self, ...) configures it
Use __new__ for Singletons and other creation-control patterns
__repr__ is for developers (make it code-like with !r)
__str__ is for users (optional, human-friendly)