DEV Community: aykhlf yassir

URL to Pixel: How you got Rickrolled

aykhlf yassir — Fri, 10 Apr 2026 10:44:11 +0000

A complete sequential trace of every system layer traversed when a user types http://www.youtube.com/watch?v=dQw4w9WgXcQ into Chrome and presses Enter — from the first kernel interrupt to the final GPU composited frame.

Layer ① — Client

Phase 1 — Input Processing

KEY_UP → URI parse

The browser's UI thread intercepts the KEY_UP event on Enter. The raw string http://www.youtube.com/watch?v=dQw4w9WgXcQ is passed to Chrome's embedded GURL parser.

1.1 Event dispatch (UI thread)

The OS delivers a WM_KEYUP (Windows) or KeyRelease (X11) event to the browser process's message pump. The Compositor thread forwards it to the Browser's Main (UI) thread via an IPC message.

1.2 URI structural parse (GURL)

GURL splits the string into:

scheme: http
host: www.youtube.com
path: /watch
query: ?v=dQw4w9WgXcQ

A strict URI regex confirms it is not an Omnibox search query — routing to the search engine is bypassed.

1.3 Search vs. navigate decision (Omnibox)

If the parsed scheme is http, https, ftp, file, or a registered custom scheme, the input is treated as a navigation. Any ambiguous string (no TLD, embedded spaces) would instead be forwarded to the default search engine.

Phase 2 — HSTS Enforcement & Scheme Upgrade

307 internal redirect

Before any network I/O, Chrome consults its preloaded HSTS list (sourced from the HSTS Preload List, baked into Chromium's binary and also cached locally). youtube.com is on this list.

2.1 Preload list lookup (HSTS)

Chrome walks a prefix tree over the eTLD+1 of the host (youtube.com). The entry signals includeSubDomains, so www.youtube.com is also covered. Stored policy fields: max-age, includeSubDomains, preload.

2.2 Scheme rewrite (307 Internal)

Chrome synthesises a 307 Internal Redirect entirely in-process — no bytes leave the machine. The URL is rewritten to https://www.youtube.com/watch?v=dQw4w9WgXcQ. This prevents any HTTP probe packet that could be intercepted by a MITM before a 301 redirect arrives from the server.

2.3 MITM downgrade prevention

Without HSTS, an attacker on the local network could intercept the initial http:// SYN, impersonate the server, serve a spoof page, and later redirect to HTTPS — the user would never notice. HSTS eliminates the plaintext window entirely.

Phase 3 — Inter-Process Communication (IPC)

UI thread → Network thread

The UI thread cannot perform blocking network I/O directly. It packages the navigation intent into an IPC message and posts it asynchronously to the Network Service process (isolated since Chrome 66).

3.1 Navigation request serialisation (Mojo IPC)

Chrome uses the Mojo IPC framework. The UI thread creates a network::ResourceRequest struct, serialises it over a Mojo pipe to the Network Service. This pipe is a named shared-memory channel; passing a file descriptor for the socket avoids a data copy.

3.2 Network Service sandbox

The Network Service runs in a restricted sandbox (no filesystem access, limited syscalls via seccomp-BPF). It owns all sockets, DNS resolution, and TLS. Compromise of the Renderer or UI process does not expose raw socket access — only the Network Service can read/write encrypted bytes.

Phase 4 — DNS Resolution — Cache Traversal

Browser → OS → hosts file

Chrome's Network Service walks a three-tier cache hierarchy before issuing any external DNS packet. Each tier is a TTL-governed LRU cache.

4.1 Browser DNS cache (Tier 1)

Chrome maintains its own in-process DNS cache, independent of the OS. Keyed by hostname + record type. chrome://net-internals/#dns exposes its contents. On a cache hit, the IP is returned immediately; the A/AAAA record TTL governs expiry.

4.2 OS stub resolver — `getaddrinfo` (Tier 2)

On a miss, Chrome calls getaddrinfo() (POSIX) or DnsQuery() (Win32). The OS checks its own DNS cache (nscd on Linux, DNSCache service on Windows). It also reads /etc/hosts (POSIX) or C:\Windows\System32\drivers\etc\hosts for static overrides.

4.3 Hosts file override (Tier 3)

127.0.0.1 localhost is the canonical example. Entries here short-circuit all network lookups. Malware often poisons this file to redirect domains — it takes effect before any DNS packet is emitted.

Phase 5 — DNS Resolution — Recursive Hierarchy

Stub → Root → TLD → Authoritative

On a full cache miss, the OS stub resolver emits a UDP datagram to the configured recursive resolver (e.g., 8.8.8.8). If that resolver also lacks the record, it traverses the global DNS tree.

5.1 DNS query datagram (UDP)

The stub resolver constructs a 12-byte DNS header plus the QNAME wire format: \x03www\x07youtube\x03com\x00, QTYPE A (0x0001), QCLASS IN (0x0001). Sent as a single UDP datagram to port 53 of the configured resolver.

5.2 Root → TLD → Authoritative traversal (Recursive)

If the recursive resolver (e.g. Google Public DNS at 8.8.8.8) lacks a cached answer:

It queries one of the 13 root server clusters (A–M.ROOT-SERVERS.NET) for a referral to the .com TLD servers.
It queries a Verisign TLD server for the NS records of youtube.com.
It queries Google's authoritative nameservers (ns1–ns4.google.com) for the A/AAAA record of www.youtube.com.

5.3 DNSSEC validation (optional)

Google's public resolvers support DNSSEC. The resolver validates the chain of trust from the root KSK → .com ZSK → youtube.com ZSK → the RRSet signature. An invalid signature causes the resolver to return SERVFAIL rather than a spoofed IP.

5.4 A/AAAA record returned

Google's authoritative servers return an Anycast VIP (Virtual IP) belonging to Google's global load-balancing fabric — a Google Front End (GFE). The TTL is typically 300 seconds. The recursive resolver caches the answer and returns it to the OS, which caches it and returns it to Chrome.

Layer ② — Network Transit

Phase 6 — Socket Allocation & Transport Handshake

Socket alloc, QUIC/TCP handshake

Chrome prefers HTTP/3 over QUIC (UDP-based) when connecting to Google services. It remembers QUIC availability via an Alt-Svc header from previous sessions. If QUIC is blocked, it falls back to TCP/TLS.

6.1 QUIC 0-RTT or 1-RTT handshake

QUIC multiplexes TLS 1.3 and transport into a single UDP-based protocol. On a known server (cached session ticket), Chrome attempts a 0-RTT handshake: it sends the ClientHello + HTTP request data in the very first UDP datagram. The server can respond immediately, saving one full round-trip versus TCP+TLS.

6.2 TCP three-way handshake (fallback)

If QUIC is unavailable (UDP blocked, no cached session): the OS allocates a socket FD, binds an ephemeral source port (e.g., 54321), and sends a TCP SYN to the GFE's Anycast IP on port 443. The server responds with SYN-ACK; the client sends ACK. The connection is established after one RTT.

6.3 TLS 1.3 handshake (TCP path)

Immediately after the TCP ACK, the client sends a ClientHello containing:

Supported cipher suites: TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256
An ECDH key share (X25519 curve)
SNI extension: www.youtube.com
ALPN extension: h2

The server responds with ServerHello + Certificate + CertificateVerify + Finished. The client verifies the cert chain against Mozilla's Root CA store (bundled in Chrome). A symmetric session key is derived via ECDHE (Elliptic Curve Diffie-Hellman Ephemeral). Total handshake overhead: 1 RTT.

6.4 Cipher negotiation & symmetric key (AES-NI)

The agreed cipher is typically AES-128-GCM. On modern x86 CPUs, the AES-NI instruction set executes each AES round as a single opcode, achieving ~40 Gbps throughput per core. GCM mode provides both encryption and a Message Authentication Code (MAC) in one pass.

Phase 7 — HTTP Request Construction & Transmission

GET /watch HEADERS frame

The Network Service formats an HTTP/2 HEADERS frame (or HTTP/3 HEADERS frame over QUIC) containing the full request, then encrypts and transmits it.

7.1 Header compression — HPACK/QPACK

HTTP/2 uses HPACK to compress headers. Common headers like :method: GET, :scheme: https are represented as 1-byte indices into a static table. Custom headers like Cookie are Huffman-encoded and stored in a dynamic table for future requests on the same connection. This reduces header overhead from ~800 bytes to ~100 bytes.

7.2 HEADERS frame structure

The HEADERS frame contains:

:method GET
:path /watch?v=dQw4w9WgXcQ
:authority www.youtube.com
:scheme https
user-agent (Chrome version string)
accept-encoding: br, gzip
cookie: SID=...; HSID=...; SSID=... (YouTube session tokens)

Stream ID is 1 (first client-initiated stream).

7.3 TLS record layer encryption

The HEADERS frame bytes are passed to BoringSSL's SSL_write(). They are chunked into TLS records (max 16 KB each), each prepended with a 5-byte header (content type, version, length) and appended with a 16-byte GCM authentication tag. The ciphertext is written to the kernel's TCP send buffer via write() / sendmsg().

Phase 8 — Network Transit & BGP Routing

BGP routing, PoP ingress

Encrypted IP packets traverse the public internet, routed by BGP (Border Gateway Protocol), before arriving at a Google Point of Presence (PoP).

8.1 Autonomous system routing (BGP)

Each internet router makes next-hop decisions by looking up the destination IP in its BGP routing table. The table is built by exchanging AS PATH advertisements between Autonomous Systems. Packets may traverse 10–20 router hops, each performing a longest-prefix match in nanoseconds via hardware TCAM (Ternary Content Addressable Memory).

8.2 Google Anycast ingress

Google's GFE VIPs are announced from hundreds of PoPs simultaneously via BGP Anycast. The internet's routing infrastructure naturally directs the packet to the topologically nearest Google PoP. A user in a given region (e.g. North Africa) reaches a Google edge in a nearby city (e.g. Paris or Frankfurt) rather than a US datacenter.

8.3 Maglev load balancer

At the PoP, Maglev — Google's software load balancer — receives the packet. It computes a consistent hash over the 5-tuple (src IP, src port, dst IP, dst port, protocol) to deterministically map the connection to a specific GFE server. The same 5-tuple always maps to the same GFE, preserving TCP state across server pool changes via consistent hashing.

Layer ③ — Server

Phase 9 — NIC Ingress & Kernel DMA

DMA → sk_buff → NAPI

The TCP SYN (or QUIC UDP datagram) physically arrives at the GFE server's Network Interface Controller (NIC).

9.1 Optical → electrical conversion (PHY)

The NIC's PHY transceiver converts arriving optical pulses (1310 nm or 1550 nm photons on the fibre) into differential electrical signals, then into digital logic. The MAC layer reads the Ethernet frame and verifies the Frame Check Sequence (FCS) — a CRC-32 over the entire frame. A bad FCS causes the frame to be silently dropped.

9.2 PCIe DMA → sk_buff

On a valid FCS, the NIC's PCIe DMA engine copies the packet payload directly into a pre-allocated kernel ring buffer (the RX descriptor ring) without CPU involvement. Linux represents each packet as an sk_buff (socket buffer) struct — a 256-byte metadata header pointing to the payload in kernel memory.

9.3 IRQ → NET_RX_SOFTIRQ → NAPI poll

The NIC asserts a hardware IRQ. The CPU services it, but immediately masks further NIC interrupts and schedules a NET_RX_SOFTIRQ. The kernel shifts to NAPI polling mode: it spins draining the RX ring in a tight loop, processing up to budget=64 packets per poll cycle. This prevents interrupt storms at high packet rates (10–100 Gbps).

Phase 10 — Kernel TCP State Machine & Accept Queue

SYN queue → Accept queue → epoll

The packet ascends the Linux network stack. The TCP layer manages the SYN queue and Accept queue, then notifies user space via epoll.

10.1 IP layer demultiplexing

The IPv4/IPv6 layer validates the header checksum and the destination IP against the server's interface addresses. It looks up the transport protocol (6 = TCP, 17 = UDP) in the protocol table and passes the payload upward.

10.2 SYN queue → Transmission Control Block (TCB)

The TCP layer hashes the 4-tuple to locate or create a Transmission Control Block (TCB). For a new SYN, it allocates a half-open TCB in the SYN queue and sends SYN-ACK. The SYN queue is protected against SYN flood attacks by SYN cookies — the initial sequence number encodes a hash of the 4-tuple so no per-connection state is needed until the final ACK arrives.

10.3 ACK → Accept queue → `accept4()`

When the client's final ACK arrives, the kernel moves the TCB to the socket's Accept queue. The GFE process is blocked on epoll_wait(). The kernel signals an EPOLLIN event on the listening socket FD. A GFE worker thread wakes, calls:

accept4(listen_fd, (struct sockaddr *)&client_addr, &addr_len, SOCK_NONBLOCK)

This dequeues the connection and returns a new non-blocking socket FD plus the client's sockaddr_in struct (IP + ephemeral port, big-endian network byte order):

struct sockaddr_in {
    sa_family_t    sin_family; /* AF_INET */
    in_port_t      sin_port;   /* client ephemeral port, big-endian */
    struct in_addr sin_addr;   /* client public IP address */
    char           sin_zero[8];/* padding */
};

Phase 11 — User-Space TLS Termination

BoringSSL, AES-NI, ECDHE

The GFE registers the new client FD with its epoll instance and performs TLS termination using BoringSSL, Google's internal fork of OpenSSL.

11.1 BoringSSL handshake

BoringSSL processes the ClientHello, selects a cipher suite, signs with its private key (ECDSA P-256 or RSA-2048), and derives symmetric session keys via ECDHE. The server's private key is stored in a hardware security module (HSM) or in Google's Titan security chips.

11.2 `SSL_read()` + AES-NI decryption

After the handshake, incoming TLS records trigger EPOLLIN. The GFE calls SSL_read(), which:

Pulls ciphertext from the kernel socket buffer
Invokes the EVP_AEAD_CTX_open() primitive
Calls the AES-NI-accelerated aesni_gcm_decrypt routine

Each 16-byte AES block is processed in a single VAESENC SIMD instruction. Simultaneously, the GCM authentication tag is verified — a forged or bit-flipped byte causes immediate rejection and connection termination.

11.3 Private key security (FIDO/Titan)

The TLS private key never exists in cleartext RAM on production GFE servers. Signing operations are delegated to a Titan coprocessor or an Asylo TEE (Trusted Execution Environment). This limits the blast radius of a memory-disclosure vulnerability such as Heartbleed.

Phase 12 — HTTP/2 Parsing, Header Decompression & RPC Dispatch

HPACK → gRPC dispatch

Decrypted bytes enter the GFE's HTTP/2 state machine. Headers are decompressed, the request is parsed, and an internal gRPC call is formed.

12.1 HPACK decompression & header extraction

The GFE's HTTP/2 parser consumes HEADERS frames. It runs the HPACK decoder — maintaining a dynamic table of past headers — to reconstruct the full header list: :method GET, :path /watch?v=dQw4w9WgXcQ, cookie: SID=.... The SID token is extracted for Gaia authentication.

12.2 Virtual host & path routing

The GFE evaluates the Host header (www.youtube.com) and path (/watch) against its routing table — a radix tree keyed on (host, path prefix). It resolves to the "Watch" backend cluster. A weighted round-robin or least-loaded selection picks a specific backend server.

12.3 Protocol translation → protobuf / gRPC

The GFE strips the external TLS envelope, constructs an internal Protocol Buffer message encoding the request (method, path, headers, client IP from sockaddr_in.sin_addr), and transmits it as a gRPC call over Google's internal encrypted datacenter mesh (ALTS — Application Layer Transport Security).

Phase 13 — Backend Microservice Orchestration — Fan-out RPCs

Gaia, Spanner, TPU, Ads

The YouTube Watch service (C++ / Go) receives the gRPC call and immediately fans out parallel RPCs to specialised datastores and ML serving systems.

13.1 Identity / auth — Gaia service

An RPC to Gaia (Google's identity service) validates the SID session cookie. Gaia decrypts the token, checks its HMAC signature, looks up the user record in a distributed Bigtable, and returns the internal User ID (GAIA ID) and OAuth scopes within ~1 ms.

13.2 Video metadata — Cloud Spanner / Vitess

An RPC queries the video metadata shard keyed on dQw4w9WgXcQ. Cloud Spanner provides globally consistent reads via TrueTime timestamps (GPS + atomic clocks). The result includes title, description, channel ID, DRM flags, and publish date — retrieved in typically ~5 ms from a local replica.

13.3 Recommendations — TensorFlow / TPU

An async RPC to a TPU serving cluster running a TensorFlow ranking model. Input: user embedding vector + video embedding vector. The model scores hundreds of candidate videos via dot-product attention. Results (the "Up Next" list) arrive asynchronously while the rest of the page is assembled.

13.4 Monetisation — ad pre-roll check

An RPC to the ads engine checks the user's subscription status (YouTube Premium → no ads) and the video's monetisation flags. If an ad is to be served, an ad auction runs: DSPs submit bids in a real-time bidding (RTB) second-price auction within a ~100 ms deadline.

13.5 DASH manifest assembly

The video CDN registry is queried for all available encodings of dQw4w9WgXcQ: VP9 4K/1080p/720p/360p, AV1 1080p, H.264 1080p (for Safari/iOS). A DASH MPD manifest (XML) is generated listing segment URIs for each representation, enabling the client to perform adaptive bitrate streaming.

Phase 14 — Response Serialisation & Egress

Brotli, HTTP/2 DATA frames

The Watch service aggregates the fan-out responses and serialises the final HTML+JSON payload. The GFE compresses, frames, and encrypts it for transmission.

14.1 Brotli compression

The ~120 KB HTML+JSON payload is compressed with the Brotli algorithm (RFC 7932), which achieves ~20% better compression than gzip for text. Compression runs on a pre-built static Brotli dictionary tuned for web content. The compressed payload is typically ~25 KB.

14.2 HTTP/2 DATA frame chunking

The compressed bytes are split into HTTP/2 DATA frames (max frame size typically 16 KB). Each DATA frame carries a 9-byte header (length, type 0x0, flags, stream ID 1). Flow-control window management ensures the client's receive buffer is not overflowed.

14.3 `SSL_write()` → `sendmsg()` → NIC TX ring

Each DATA frame is passed to SSL_write() for TLS record encryption. The GFE calls sendmsg(), which copies ciphertext from user-space buffers into the kernel's TCP TX socket buffer. The TCP stack segments the data, appends IP/TCP headers (including TCP sequence numbers and ACK fields), and enqueues segments in the NIC's DMA TX ring. The NIC's DMA engine reads from the ring and drives the PHY to emit optical pulses back toward the client.

Layer ④ — Rendering & Media

Phase 15 — Renderer Process Allocation & Stream Handoff

MIME sniff, Site Isolation

As the first bytes of the HTTP response arrive at Chrome's Network Service, the browser decides which renderer process will own this navigation.

15.1 MIME sniff & response validation

The Network Service reads the Content-Type: text/html; charset=utf-8 header. It also performs MIME sniffing (reading the first 512 bytes) as a fallback. It confirms the response is navigable HTML, not a download or an opaque resource.

15.2 Site Isolation & renderer selection

Chrome's Site Isolation policy (enabled since Chrome 67, post-Spectre) requires that youtube.com content runs in a renderer process dedicated exclusively to the youtube.com origin. The Browser process either spawns a new sandboxed Renderer Process or reuses an existing one already dedicated to youtube.com. Each renderer is a separate OS process with a seccomp-BPF sandbox — a compromised renderer cannot directly access the filesystem, GPU, or network.

15.3 Byte stream handoff to renderer (Mojo data pipe)

The Network Service streams response bytes to the Renderer Process via a Mojo data pipe — a shared-memory ring buffer. The renderer reads from its end while the network thread writes, enabling true streaming parsing: the DOM can begin construction before the last byte of HTML arrives.

Phase 16 — DOM Construction — Tokenisation & Tree Building

Tokeniser → tree builder

The renderer's HTML parser (Blink's HTMLParser) processes the byte stream into the Document Object Model.

16.1 Byte stream → UTF-8 character stream

The TextResourceDecoder reads the charset=utf-8 from the Content-Type header and decodes the byte stream to Unicode code points. If charset is absent, the parser sniffs the BOM or uses the HTML5 encoding detection heuristics.

16.2 Tokenisation → StartTag / EndTag / Text

Blink's HTML tokeniser implements the HTML5 parsing algorithm — a state machine with ~80 states. It emits tokens: StartTag, EndTag, Character, Comment, DOCTYPE. The parser is speculative: it scans ahead for resource-loading tags (<img>, <script>, <link>) and issues preload requests before fully constructing the tree.

16.3 Token stream → DOM tree (tree builder)

The tree builder consumes tokens and constructs DOM nodes (HTMLElement, TextNode, etc.) using a stack-based insertion mode automaton (Initial → BeforeHtml → BeforeHead → InHead → InBody → etc.). The result is the document tree rooted at the Document node.

Phase 17 — CSSOM Construction

StyleSheets → CSSOM

Concurrently with DOM construction, the parser encounters <link rel="stylesheet"> tags and fetches CSS resources.

17.1 CSS parsing → StyleSheetContents

Each CSS file is parsed by Blink's CSSParser into a StyleSheetContents object — a sorted set of rules. Selectors are parsed into SelectorLists (trees of compound selectors). Property values are tokenised and validated against their grammar.

17.2 Rule matching & specificity → ComputedStyle

Blink builds the CSS Object Model (CSSOM): a mapping from DOM nodes to computed style objects. Rule matching traverses the style sheet tree; specificity (inline > ID > class > element) determines which conflicting declarations win. The result is stored in a ComputedStyle struct per element, containing the final resolved value for every CSS property.

Phase 18 — JavaScript Compilation & Execution (V8)

AST → Ignition → TurboFan

When the parser encounters a <script> tag (without async/defer), it pauses HTML parsing and hands the script source to the V8 engine.

18.1 Source → AST (Parser)

V8's Parser tokenises and parses the JS source into an Abstract Syntax Tree (AST). A pre-parser (scanner-only) performs a fast first pass to identify function boundaries, enabling the full parser to skip inner function bodies until they are called.

18.2 AST → bytecode (Ignition interpreter)

The Ignition interpreter walks the AST and generates register-machine bytecode (compact 1–3 bytes per instruction). Bytecode is immediately executed. Ignition profiles execution frequency and type feedback — recording, for example, that a particular addition always operates on SMI (small integer) operands.

18.3 Hot path → native code (TurboFan JIT)

Functions that exceed a call or loop-iteration threshold are submitted to the TurboFan optimising compiler. TurboFan builds a Sea of Nodes IR, applies type-specialised optimisations (function inlining, escape analysis, SIMD auto-vectorisation), and emits native x86-64 / ARM64 machine code. If a type assumption is violated at runtime, a deoptimisation trap drops execution back to Ignition bytecode.

18.4 YouTube SPA bootstrap

YouTube's JS initialises a Web Components / Polymer-based Single Page Application. It registers custom elements (<ytd-app>, <ytd-video-primary-info-renderer>), sets up a client-side router listening to history.pushState, and issues Fetch API calls for the video metadata JSON (loaded separately from the initial HTML skeleton).

Phase 19 — Render Tree Formation & Layout (Reflow)

Layout tree, reflow

The DOM and CSSOM are merged into the Layout Tree. The rendering engine then computes the exact geometry of every visible element.

19.1 DOM + CSSOM → Layout tree

Blink traverses the DOM and, for each node with a ComputedStyle that has display ≠ none, creates a LayoutObject. Pseudo-elements (::before, ::after) become anonymous LayoutObjects. The Layout Tree is distinct from the DOM tree — some DOM nodes have no LayoutObject (e.g., <head>); others (anonymous block boxes) have no corresponding DOM node.

19.2 Block formatting context & reflow

Blink's layout engine runs the CSS block/inline formatting algorithm. For each LayoutObject, it computes (x, y, width, height) relative to the containing block. Flexbox and Grid elements run their respective constraint-solving algorithms. YouTube's responsive layout relies heavily on CSS Grid and CSS Custom Properties. Reflow is triggered top-down; dirty bits propagate to ensure only subtrees with changed input geometry are re-laid out (incremental reflow).

Phase 20 — Paint, Layer Compositing & GPU Rasterisation

Paint records, layers, GPU

Blink generates paint operations, divides the page into compositor layers, rasterises them (hardware-accelerated on the GPU), and hands the final frame to the OS.

20.1 Display list generation (PaintController)

The PaintController traverses the Layout Tree in paint order (stacking contexts, z-index) and records a display list of drawing commands: DrawRect, DrawTextBlob, DrawImage. This is a serialisable record, not a bitmap — it can be replayed at any resolution or scale factor.

20.2 Compositor layer promotion

Elements are promoted to their own compositor layer when they have:

CSS will-change: transform (or transform/opacity animations)
A <video> or <canvas> element
Heuristic prediction of animation

YouTube's video player is always a separate layer. Layer promotion trades VRAM (each layer is a GPU texture) for scroll/animation smoothness — the compositor can reposition a layer without re-running JavaScript or layout.

20.3 GPU rasterisation (OOP-R, Skia-Ganesh)

Blink sends display lists to the GPU process (out-of-process rasterisation, OOP-R). The GPU process submits them to the Skia-Ganesh rendering library, which issues OpenGL / Vulkan / Metal draw calls. The GPU rasterises each layer into a GPU texture stored in VRAM.

20.4 Compositor thread → final frame → vsync

The Compositor Thread (separate from the Main Thread) receives the list of layers and their transforms. It composites them into a final frame using the GPU's texture blending units — no CPU involvement. It submits a swap command to the OS graphics API (DirectX 12 / Metal / Vulkan). The display controller reads the framebuffer and drives the monitor's scan-out circuit at the refresh rate (60 or 120 Hz). The first pixels appear on screen.

Phase 21 — Media Source Extensions & DASH Adaptive Streaming

DASH manifest, SourceBuffer, hardware decode

The JS application initialises the <video> element using the Media Source Extensions (MSE) API to implement Dynamic Adaptive Streaming over HTTP (DASH).

21.1 MediaSource & SourceBuffer creation

JavaScript calls video.src = URL.createObjectURL(new MediaSource()). When the MediaSource sourceopen event fires, JS calls:

addSourceBuffer('video/webm; codecs="vp9"')
addSourceBuffer('audio/webm; codecs="opus"')

Creating separate demuxed buffers for video and audio tracks.

21.2 MPD manifest fetch & ABR selection

JS fetches the DASH MPD manifest (XML) from YouTube's CDN. It parses available Representations (1080p VP9 @ 4 Mbps, 720p VP9 @ 2 Mbps, etc.). An ABR (Adaptive Bitrate) algorithm estimates available bandwidth using throughput history and buffer occupancy, then selects the highest quality that avoids rebuffering.

21.3 Segment fetch → `SourceBuffer.appendBuffer()`

JS issues fetch() calls for ~2–10 second video/audio segments. Received ArrayBuffers are passed to sourceBuffer.appendBuffer(data). The browser's media pipeline demultiplexes the WebM container (EBML format), extracts compressed frames, and maintains a coded frame buffer. As the video element's currentTime advances, the media pipeline dequeues the next coded frame.

21.4 Hardware media decoder (VP9 / AV1 / H.264)

Compressed video frames are submitted to the OS's hardware media decoder:

VAAPI (Linux / Intel/AMD GPU)
VideoToolbox (macOS / iOS)
DXVA2 / D3D11VA (Windows)
MediaCodec (Android)

The dedicated video decode ASIC on the GPU die processes frames at full resolution without CPU involvement. Output is planar YUV 4:2:0 video frames in VRAM.

21.5 A/V sync via presentation timestamps (PTS)

Each video frame and audio packet carries a Presentation Timestamp (PTS) in the container's timebase. The media pipeline maintains a media clock locked to the audio renderer's playback position (audio is the master clock). Video frames are held in a decoded frame queue and released to the compositor exactly when currentTime ≥ frame.PTS.

21.6 YUV texture upload, GPU composite & audio DAC output

Decoded YUV frames are uploaded as GPU textures and composited into the video compositor layer by the Compositor Thread, with a YUV→RGB conversion shader applied on the GPU.

Audio PCM samples are written into the OS audio daemon's ring buffer:

CoreAudio (macOS / iOS)
PulseAudio / PipeWire (Linux)
WASAPI (Windows)

The DAC converts the digital PCM samples to analogue waveforms, drives the speaker amplifier — and Rick Astley begins to sing.

Summary of Layer Boundaries

Layer	Entry point	Exit point
① Client	`KEY_UP` event on Enter	TLS-encrypted bytes leave the NIC
② Network transit	First router hop after the client NIC	Packet arrives at Google PoP NIC
③ Server	GFE NIC PHY receives optical pulse	Encrypted HTTP/2 DATA frames leave GFE NIC
④ Rendering & media	Chrome Network Service receives first response bytes	GPU scans out final composited frame; DAC emits audio

Total elapsed wall-clock time (typical broadband, nearby PoP): ~200–400 ms to first meaningful paint; ~1–2 s to first video frame playback.

Python Internals: Decorators

aykhlf yassir — Sat, 21 Feb 2026 01:42:45 +0000

Stop treating the @ symbol as magic. Let's tear down the abstraction layer and build decorators from first principles, using heap allocation, closures, and time complexity as our guides.

If you come from a static, compiled background like C++ or Java, Python decorators can feel like black magic. You slap an @login_required above a function, and suddenly it has authentication logic. You add @app.route("/"), and suddenly it's a web endpoint.

It feels like magic because it hides complexity. But as engineers, we know that "magic" is just code we haven't understood yet.

In this post, we are going to demystify Python metaprogramming. We won't just learn how to use decorators; we're going to understand the memory mechanics that make them possible, build a production-ready utility belt, and use them to fundamentally alter the algorithmic complexity of a function.

Part 1: The Mechanics

Before we write a decorator, we need to understand the raw materials. In Python, the mechanisms that enable decorators are First-Class Functions and Closures.

1. Functions are just heap-allocated objects

In C, a function is a block of instructions in the .text segment. In Python, a function is a full-blown object (a PyObject struct) living on the heap. Because it's an object:

You can assign it to a variable.
You can pass it as an argument to another function.
You can return it from another function.

This ability to treat functions as data is the cornerstone of metaprogramming.

2. The Engine: Closures

If you come from C++, you know that local variables die when a function's stack frame is popped. Python is different. If an inner function references a variable from an enclosing scope, Python's compiler notices. It "promotes" that variable from a simple stack item to a Cell Object on the heap.

Even after the outer function returns, the inner function retains a reference to that Cell Object. This "remembered environment" is called a Closure.

A closure is a function that remembers the variables from its enclosing scope even after the outer function has finished executing.

Decorators rely entirely on closures to remember which function they are wrapping.

Part 2: Building the Pattern

A decorator is fundamentally simple: It is a function that takes a function as input and returns a new function as output.

Let's look at the raw pattern before we use the pretty syntax.

# The decorator factory
def my_decorator(target_func):
    # The wrapper closure retains access to 'target_func'
    def wrapper():
        print(">>> Before execution")
        target_func() # Calling the original function
        print("<<< After execution")
    # Returning the new function
    return wrapper

def critical_system_call():
    print("Executing core logic...")

# MANUAL DECORATION: This is all a decorator is.
# We are reassigning the pointer to the new wrapper closure.
critical_system_call = my_decorator(critical_system_call)

critical_system_call()
# Output:
# >>> Before execution
# Executing core logic...
# <<< After execution

The @ symbol is just syntactic sugar for that manual reassignment. The following code is functionally identical to the manual version above:

@my_decorator
def critical_system_call():
    print("Executing core logic...")

The Introspection "Gotcha"

We broke something in the previous example. If we ask Python for the name of our function, it lies to us.

print(critical_system_call.__name__)
# Output: wrapper

Because we reassigned the pointer, the metadata now belongs to the inner wrapper function. This breaks debuggers, profilers, and frameworks that rely on introspection.

The Fix: functools.wraps

The standard library provides a fix. Paradoxically, it's another decorator called @wraps. It copies the metadata (name, docstring, annotations) from the original function to the wrapper.

This is the required boilerplate for nearly every decorator you will ever write:

from functools import wraps

def robust_decorator(func):
    # Apply @wraps to the wrapper closure
    @wraps(func) 
    def wrapper(*args, **kwargs):
        # Note the use of *args and **kwargs to make 
        # the wrapper generic for any function signature.
        print(f"Calling {func.__name__}...")
        return func(*args, **kwargs)
    return wrapper

@robust_decorator
def my_task():
    """Does hard work."""
    pass

print(my_task.__name__) # Output: my_task (Fixed!)
print(my_task.__doc__)  # Output: Does hard work. (Fixed!)

Part 3: The Utility Belt (Practical Examples)

Now that we have the mechanics, let's build tools commonly used in backend development.

1. The @timer (Profiling)

Don't use time.time() for profiling; it's subject to system clock adjustments. Use time.perf_counter() for monotonic timing guarantees.

import time
from functools import wraps

def timer(func):
    """Prints the execution time of the decorated function."""
    @wraps(func)
    def wrapper(*args, **kwargs):
        start_time = time.perf_counter()
        result = func(*args, **kwargs)
        end_time = time.perf_counter()
        print(f"[TIMER] {func.__name__} executed in {end_time - start_time:.4f}s")
        return result
    return wrapper

2. The @debug (Introspection Proxy)

This is a lifesaver when debugging complex recursion or deeply nested framework calls. It intercepts arguments and logs exactly what is going into and coming out of a function.

from functools import wraps

def debug(func):
    """Prints function signature and return value on every call."""
    @wraps(func)
    def wrapper(*args, **kwargs):
        args_repr = [repr(a) for a in args]
        kwargs_repr = [f"{k}={repr(v)}" for k, v in kwargs.items()]
        signature = ", ".join(args_repr + kwargs_repr)

        print(f"[DEBUG] Calling {func.__name__}({signature})")
        result = func(*args, **kwargs)
        print(f"[DEBUG] {func.__name__} returned {repr(result)}")
        return result
    return wrapper

Part 4: Boss Level Optimization (@memoize)

We can use decorators to fundamentally change the performance characteristics of an algorithm.

Let's look at the naive recursive Fibonacci implementation. Its time complexity is O(2^n) (exponential) because it re-calculates the same sub-problems endlessly.

# Without optimization, fibonacci(35) takes several seconds.
def fibonacci(n):
    if n < 2:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

We can fix this with memoization (caching). We need a persistent dictionary to store results. Where does that dictionary live? In the closure's enclosing scope.

The Architectural Challenge

We need to use the function's arguments as keys for our cache dictionary.

Positional arguments (*args) are a tuple. Tuples are immutable and hashable. Good.
Keyword arguments (**kwargs) are a dict. Dicts are mutable and unhashable. Bad.

If we try to use kwargs as part of a dictionary key, Python throws a TypeError. We must convert the kwargs dictionary into an immutable, order-independent representation.

The Implementation

from functools import wraps

def memoize(func):
    # This dictionary lives in the closure's persistent heap state.
    cache = {}

    @wraps(func)
    def wrapper(*args, **kwargs):
        # 1. Convert kwargs to a sorted, immutable tuple of items
        frozen_kwargs = tuple(sorted(kwargs.items()))

        # 2. Create the composite cache key
        cache_key = (args, frozen_kwargs)

        # 3. Check cache (O(1) lookup)
        if cache_key not in cache:
             # Cache miss: compute and store
            cache[cache_key] = func(*args, **kwargs)

        return cache[cache_key]
    return wrapper

The Result

Let's stack our tools. Python applies decorators from the bottom up.

@timer    # Applied last (outer wrapper)
@memoize  # Applied first (inner wrapper around the recursive function)
def fibonacci_optimized(n):
    if n < 2:
        return n
    return fibonacci_optimized(n - 1) + fibonacci_optimized(n - 2)

# The original takes seconds.
# The optimized version executes instantly.
fibonacci_optimized(35) 
# Output: [TIMER] fibonacci_optimized executed in 0.0001s

By using a decorator to act as a caching proxy, we collapsed the time complexity from exponential O(2^n) to linear O(n). We made a classic space-time tradeoff, sacrificing a small amount of heap memory for massive CPU gains, without touching the core algorithm's code.

(Note: In production, prefer Python's built-in functools.lru_cache over rolling your own, as it handles cache bounding to prevent memory leaks).

Summary

Decorators aren't magic. They are an elegant application of First-Class Functions and Closures. By understanding how Python handles scope and object life-cycles on the heap, you gain the power to modify behavior, inject logic, and optimize performance cleanly and imperatively.

Python Internals: `yield from` for Composable Generators

aykhlf yassir — Thu, 19 Feb 2026 18:04:36 +0000

How a single keyword turns nested iterators into elegant, high-performance data pipelines

The Manual Delegation Problem

You've mastered generators. You know yield creates a pause point, that generators are lazy, and that they enable constant-memory processing of infinite streams.

But then you hit this problem:

def flatten(nested_list):
    """Flatten a nested list structure."""
    for item in nested_list:
        if isinstance(item, list):
            # We have a nested list - need to flatten it recursively
            for sub_item in flatten(item):  # Recursive call
                yield sub_item              # Manual forwarding
        else:
            yield item

nested = [1, [2, 3, [4, 5]], 6, [7, [8, 9]]]
print(list(flatten(nested)))  # [1, 2, 3, 4, 5, 6, 7, 8, 9]

It works. But look at that inner loop: for sub_item in flatten(item): yield sub_item. You're manually forwarding every value from the recursive generator to the caller. It's boilerplate—and it gets worse when your generator needs to handle .send() and .throw().

There's a better way:

def flatten(nested_list):
    """Flatten a nested list structure."""
    for item in nested_list:
        if isinstance(item, list):
            yield from flatten(item)  # Delegate completely
        else:
            yield item

Two words replace two lines. But this isn't just syntax sugar—yield from creates a transparent bidirectional channel between the caller and the subgenerator. Every next(), every .send(), every .throw() passes through untouched.

Today we're diving deep into what makes this powerful, when it's essential (not just convenient), and how it became the foundation for Python's async/await.

1. Core Mechanics: What `yield from` Actually Does

1.1. Generator Delegation: The Three Phases

When you write yield from subgen, Python establishes a direct communication channel that has three distinct phases:

def delegator():
    print("[DELEGATOR] Starting")
    result = yield from subgenerator()  # Delegation point
    print(f"[DELEGATOR] Subgen returned: {result}")
    yield "done"

def subgenerator():
    print("[SUBGEN] Starting")
    yield "first"
    yield "second"
    print("[SUBGEN] Exhausted")
    return "FINAL_VALUE"  # This return value is captured!

gen = delegator()
print(next(gen))  # "first"  - comes directly from subgenerator
print(next(gen))  # "second" - comes directly from subgenerator
print(next(gen))  # "done"   - delegator resumes after subgen exhausted

Output:

[DELEGATOR] Starting
[SUBGEN] Starting
first
second
[SUBGEN] Exhausted
[DELEGATOR] Subgen returned: FINAL_VALUE
done

Here's what happened, step by step:

Phase 1: Suspension

delegator() runs until it hits yield from subgenerator()
Control is fully transferred to subgenerator()
The delegating generator (delegator) is suspended—its stack frame is frozen

Phase 2: Transparent Proxying

Every next() call on gen is forwarded to subgenerator()
Values flow directly from subgen to caller—no intermediate handling
The delegator doesn't wake up at all during this phase

Phase 3: Exhaustion and Return Value Capture

When subgenerator() raises StopIteration, Python catches it
The StopIteration.value (the return value) is captured
The delegator resumes, with result bound to "FINAL_VALUE"

1.2. The Bidirectional Channel: Send, Throw, Close

The real power of yield from emerges when your generators are coroutines—not just producing values, but consuming them via .send() and handling errors via .throw().

Without `yield from`: Manual Proxying Hell

def delegator_manual():
    subgen = subgenerator()

    # Manually forward EVERY operation
    value = None
    while True:
        try:
            if value is None:
                result = next(subgen)
            else:
                result = subgen.send(value)
            value = (yield result)  # Yield result, receive next input
        except StopIteration as e:
            return e.value  # Capture final return value

This is 12 lines of intricate control flow just to forward operations. And it's still incomplete—it doesn't handle .throw() or .close() properly.

With `yield from`: Automatic Transparent Proxying

def delegator_clean():
    result = yield from subgenerator()
    return result

Two lines. Functionally identical. All operations—next(), .send(), .throw(), .close()—are automatically forwarded to the subgenerator.

Let's prove it with a coroutine:

def accumulator():
    """A coroutine that sums values sent to it."""
    total = 0
    while True:
        try:
            value = (yield total)  # Send out total, receive next value
            if value is None:
                break
            total += value
        except ValueError:
            print("[ACCUM] Received ValueError, resetting")
            total = 0

def delegator():
    """Delegates all operations to accumulator."""
    print("[DELEG] Starting delegation")
    final = yield from accumulator()
    print(f"[DELEG] Accumulator finished with: {final}")
    return final

# Build the pipeline
gen = delegator()
next(gen)  # Prime the coroutine

# Test send()
print(gen.send(10))   # 10
print(gen.send(20))   # 30

# Test throw()
gen.throw(ValueError)  # Resets accumulator to 0
print(gen.send(5))     # 5

# Test close via sending None
gen.send(None)

Output:

[DELEG] Starting delegation
10
30
[ACCUM] Received ValueError, resetting
0
5
[DELEG] Accumulator finished with: 5

Every .send() and .throw() went directly to accumulator()—the delegator never woke up. This is the transparent channel.

The PEP 380 Specification

yield from is precisely defined in PEP 380. Here's the pseudocode for what Python actually executes:

# yield from EXPR is equivalent to:

_iter = iter(EXPR)
try:
    _y = next(_iter)
except StopIteration as _e:
    _result = _e.value
else:
    while True:
        try:
            _sent = yield _y
        except GeneratorExit as _e:
            try:
                _meth = _iter.close
            except AttributeError:
                pass
            else:
                _meth()
            raise _e
        except BaseException as _e:
            _meth = getattr(_iter, 'throw', None)
            if _meth is None:
                raise
            try:
                _y = _meth(_e)
            except StopIteration as _e:
                _result = _e.value
                break
        else:
            try:
                if _sent is None:
                    _y = next(_iter)
                else:
                    _y = _iter.send(_sent)
            except StopIteration as _e:
                _result = _e.value
                break

This is 35 lines of intricate exception handling and control flow—all handled by two words: yield from.

2. Architectural Patterns: When to Use `yield from`

2.1. Composable Pipelines: Separation of Concerns

The canonical use case is building modular data processing pipelines where each component is a small, testable generator.

def read_logs(path: str):
    """Source: stream lines from a log file."""
    with open(path) as f:
        yield from f  # Delegate to file's iterator

def parse_lines(lines):
    """Transform: parse log format."""
    for line in lines:
        if line.strip():
            timestamp, level, message = line.split("|", 2)
            yield {
                "timestamp": timestamp.strip(),
                "level": level.strip(),
                "message": message.strip(),
            }

def filter_errors(records):
    """Filter: only ERROR-level records."""
    for record in records:
        if record["level"] == "ERROR":
            yield record

def pipeline(path: str):
    """Compose the full pipeline."""
    lines = read_logs(path)
    records = parse_lines(lines)
    errors = filter_errors(records)
    yield from errors  # Delegate to the final stage

# Usage: constant memory, no matter how large the file
for error in pipeline("application.log"):
    alert(error)

Each component:

Does one thing
Is testable in isolation
Composes via yield from
Processes data lazily

2.2. Recursive Generators: Implicit Stack Management

When you need to traverse recursive structures (trees, nested lists, JSON), yield from handles the call stack implicitly.

Example: Deep Flattening with Type Safety

from collections.abc import Iterable

def flatten(obj):
    """
    Recursively flatten any nested iterable structure.

    Handles: lists, tuples, sets, generators, custom iterables.
    Does NOT recurse into strings (they're iterable but shouldn't be flattened).
    """
    # Base case 1: strings are iterable but shouldn't be flattened
    if isinstance(obj, str):
        yield obj
        return

    # Base case 2: not iterable at all
    if not isinstance(obj, Iterable):
        yield obj
        return

    # Recursive case: iterate and recurse
    for item in obj:
        yield from flatten(item)

# Test cases
nested = [1, [2, 3, [4, 5]], 6, [7, [8, 9]]]
print(list(flatten(nested)))
# [1, 2, 3, 4, 5, 6, 7, 8, 9]

mixed = [1, "hello", [2, ["world", [3, 4]]], (5, 6)]
print(list(flatten(mixed)))
# [1, 'hello', 2, 'world', 3, 4, 5, 6]

The String Trap: Without the isinstance(obj, str) check, you'd get infinite recursion:

# BROKEN VERSION
def flatten_broken(obj):
    if not isinstance(obj, Iterable):
        yield obj
    else:
        for item in obj:
            yield from flatten_broken(item)  # Infinite loop on strings!

# This never terminates:
# flatten_broken("hello")
# → "hello" is iterable
# → iterate: 'h', 'e', 'l', 'l', 'o'
# → flatten('h')
# → 'h' is iterable (it's a string!)
# → iterate: 'h'
# → flatten('h')
# → ... infinite recursion

Strings are iterable, but each character is also a string. The check if isinstance(obj, str) breaks the cycle.

3. Data Structure Operations: Tree Traversals

yield from shines when traversing hierarchical structures. The recursive delegation naturally mirrors the tree's structure.

3.1. Binary Tree Traversal

from dataclasses import dataclass
from typing import Optional

@dataclass
class TreeNode:
    value: int
    left: Optional['TreeNode'] = None
    right: Optional['TreeNode'] = None

def inorder(node: Optional[TreeNode]):
    """Inorder traversal: Left → Root → Right"""
    if node is None:
        return  # Base case: empty tree/subtree

    yield from inorder(node.left)   # Recurse left
    yield node.value                # Process root
    yield from inorder(node.right)  # Recurse right

# Build a tree:
#        4
#       / \
#      2   6
#     / \ / \
#    1  3 5  7
root = TreeNode(4,
    left=TreeNode(2, TreeNode(1), TreeNode(3)),
    right=TreeNode(6, TreeNode(5), TreeNode(7))
)

print(list(inorder(root)))
# [1, 2, 3, 4, 5, 6, 7]  - sorted order!

The elegance: each recursive call handles its own subtree. The yield from stitches them together into a single stream. No explicit stack. No manual queue management.

3.2. File System Traversal (N-ary Tree)

from pathlib import Path

def walk_tree(path: Path):
    """
    Recursively traverse a directory tree, yielding all file paths.

    This is a generator-based reimplementation of os.walk().
    """
    if not path.exists():
        return

    if path.is_file():
        yield path  # Base case: leaf node (file)
    elif path.is_dir():
        for child in path.iterdir():
            yield from walk_tree(child)  # Recurse into subdirectory

# Usage
for filepath in walk_tree(Path(".")):
    if filepath.suffix == ".py":
        print(f"Found Python file: {filepath}")

Each directory is a node with N children. The recursion naturally handles arbitrary depth. Memory usage: O(depth), not O(total files).

Compare to the eager version:

# EAGER: Builds entire list before returning
def walk_tree_eager(path: Path) -> list[Path]:
    if path.is_file():
        return [path]
    elif path.is_dir():
        results = []
        for child in path.iterdir():
            results.extend(walk_tree_eager(child))  # Accumulates in memory
        return results

For a directory with 1 million files, the eager version allocates a list with 1 million Path objects before you can process the first one. The lazy version yields them one at a time.

4. Performance & Profiling: Measuring the Impact

Let's quantify the difference between lazy and eager evaluation.

4.1. Memory Comparison: `sys.getsizeof` vs. `tracemalloc`

import sys
import tracemalloc

def numbers_eager(n: int) -> list[int]:
    """Eager: build entire list."""
    return list(range(n))

def numbers_lazy(n: int):
    """Lazy: yield from range."""
    yield from range(n)

# Test with 1 million integers
n = 1_000_000

# Measure eager version
tracemalloc.start()
eager = numbers_eager(n)
eager_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

# Measure lazy version
tracemalloc.start()
lazy = numbers_lazy(n)
lazy_mem = tracemalloc.get_traced_memory()[0]
tracemalloc.stop()

print(f"Eager list size:    {sys.getsizeof(eager):>12,} bytes")
print(f"Lazy generator:     {sys.getsizeof(lazy):>12,} bytes")
print(f"Eager traced mem:   {eager_mem:>12,} bytes")
print(f"Lazy traced mem:    {lazy_mem:>12,} bytes")
print(f"Memory savings:     {eager_mem / lazy_mem:.1f}x")

Output (typical):

Eager list size:        8,448,728 bytes
Lazy generator:               112 bytes
Eager traced mem:       8,448,824 bytes
Lazy traced mem:                496 bytes
Memory savings:         17034.7x

The lazy version uses 17,000x less memory. For a million integers, it's 8MB vs. 112 bytes.

4.2. Deep Memory Profiling: Tree Flattening

Let's compare eager vs. lazy flattening of a deeply nested structure:

import tracemalloc
import sys

def build_nested_list(depth: int, width: int):
    """Build a tree-like nested list structure."""
    if depth == 0:
        return list(range(width))
    return [build_nested_list(depth - 1, width) for _ in range(width)]

def flatten_eager(nested):
    """Eager flattening: accumulate in a list."""
    result = []
    if isinstance(nested, str):
        return [nested]
    if not isinstance(nested, list):
        return [nested]
    for item in nested:
        result.extend(flatten_eager(item))
    return result

def flatten_lazy(nested):
    """Lazy flattening: yield from."""
    if isinstance(nested, str):
        yield nested
        return
    if not isinstance(nested, list):
        yield nested
        return
    for item in nested:
        yield from flatten_lazy(item)

# Build test data: 5 levels deep, 3 children per level
# Total leaf nodes: 3^5 = 243
nested = build_nested_list(depth=5, width=3)

# Measure eager
tracemalloc.start()
eager_result = flatten_eager(nested)
eager_current, eager_peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

# Measure lazy (without consuming)
tracemalloc.start()
lazy_result = flatten_lazy(nested)
lazy_current, lazy_peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

# Measure lazy (with consumption)
tracemalloc.start()
lazy_consumed = list(flatten_lazy(nested))
lazy_consumed_current, lazy_consumed_peak = tracemalloc.get_traced_memory()
tracemalloc.stop()

print(f"Nested structure has {len(eager_result)} leaf values")
print(f"\nEager flattening:")
print(f"  Peak memory: {eager_peak:>10,} bytes")
print(f"\nLazy generator (not consumed):")
print(f"  Peak memory: {lazy_peak:>10,} bytes")
print(f"\nLazy consumed into list:")
print(f"  Peak memory: {lazy_consumed_peak:>10,} bytes")
print(f"\nSavings (lazy vs eager): {eager_peak / lazy_peak:.1f}x")

Typical output:

Nested structure has 243 leaf values

Eager flattening:
  Peak memory:     24,832 bytes

Lazy generator (not consumed):
  Peak memory:        448 bytes

Lazy consumed into list:
  Peak memory:     12,160 bytes

Savings (lazy vs eager): 55.4x

Key insights:

Eager: Builds multiple intermediate lists during recursion → high memory
Lazy (not consumed): Just the generator object → minimal memory
Lazy consumed: Eventually needs storage, but ~50% less than eager due to no intermediate lists

4.3. Execution Time: Does Laziness Cost Performance?

import timeit

nested = build_nested_list(depth=6, width=3)  # 729 items

def benchmark_eager():
    result = flatten_eager(nested)
    return len(result)

def benchmark_lazy():
    result = list(flatten_lazy(nested))
    return len(result)

eager_time = timeit.timeit(benchmark_eager, number=10_000)
lazy_time = timeit.timeit(benchmark_lazy, number=10_000)

print(f"Eager: {eager_time:.4f} seconds")
print(f"Lazy:  {lazy_time:.4f} seconds")
print(f"Difference: {abs(lazy_time - eager_time) / eager_time * 100:.1f}%")

Typical output:

Eager: 0.8234 seconds
Lazy:  0.7891 seconds
Difference: 4.2% faster (lazy)

The lazy version is typically equal or slightly faster because:

No intermediate list allocations
No repeated extend() calls (which involve copying)
Generators have optimized C implementations

The overhead of generator frames is negligible compared to list operations.

5. Historical Context: The Road to `async/await`

yield from (introduced in Python 3.3 via PEP 380) wasn't just about nested generators—it was the necessary foundation for coroutine-based concurrency.

The Evolution

Python 3.3 (2012): yield from enables transparent delegation

def task():
    yield from subtask()  # Transparent channel

Python 3.4 (2014): asyncio uses generators as coroutines

@asyncio.coroutine
def fetch(url):
    response = yield from aiohttp.get(url)  # Async I/O
    return response

Python 3.5 (2015): async/await syntax replaces generator-based coroutines

async def fetch(url):
    response = await aiohttp.get(url)  # Same semantics, clearer syntax
    return response

The semantics are identical: await is yield from with a type check. The runtime behavior—transparent delegation, bidirectional channels, exception routing—is the same.

Why the New Syntax?

Generator-based coroutines (yield from) were powerful but confusing:

# This is a generator (produces values)
def numbers():
    yield from range(10)

# This is ALSO a generator, but used as a coroutine (consumes control flow)
@asyncio.coroutine
def fetch():
    yield from aiohttp.get(url)

Same syntax, completely different purposes. The async/await keywords made the distinction explicit:

# Clearly a generator
def numbers():
    yield from range(10)

# Clearly a coroutine
async def fetch():
    await aiohttp.get(url)

But under the hood, await still does everything yield from does—it's yield from with runtime type validation.

Conclusion: When to Reach for `yield from`

Use yield from when you need to:

1. Delegate to Another Generator Completely

If you're writing for x in gen: yield x, replace it with yield from gen. It's:

More readable
More efficient
Handles .send() and .throw() correctly

2. Compose Data Processing Pipelines

Break your pipeline into small, testable generators and connect them with yield from:

def pipeline():
    raw = read_data()
    parsed = parse(raw)
    filtered = filter_valid(parsed)
    yield from transform(filtered)

3. Traverse Recursive Structures

Trees, nested lists, file systems—any recursive data structure benefits from yield from's implicit stack management:

def traverse(node):
    if node.is_leaf():
        yield node
    else:
        for child in node.children:
            yield from traverse(child)

4. Build Coroutine-Based State Machines

If you're building complex coroutines that delegate to subcomponents (pre-asyncio or for synchronous use cases):

def coordinator():
    result1 = yield from worker1()
    result2 = yield from worker2(result1)
    return result2

Python Internals: Generators & Coroutines

aykhlf yassir — Mon, 16 Feb 2026 21:24:31 +0000

There's a function that exists in almost every Python codebase. It looks harmless:

def get_trades(symbol: str) -> list[dict]:
    results = []
    for record in enormous_database_cursor:
        if record["symbol"] == symbol:
            results.append(record)
    return results

trades = get_trades("AAPL")  # 💀 Waits. Waits. Waits. Then crashes.

The problems stack up fast:

High latency: You get zero items until the entire database is scanned
Massive RAM: Every matching record is held in memory simultaneously
Fragility: One spike in result size kills the process

This is the "eager" pattern—do all the work, collect all the results, then hand them over. For small datasets, you'll never notice. For anything real-world, it's a time bomb.

The fix is a single keyword. But to use it correctly, you need to understand what it actually does to your function.

1. The Basics: What `yield` Does to a Function

Every Python function you've written follows the same lifecycle:

Call → Execute → return value → Stack frame is destroyed → Done

Local variables evaporate. State is gone. The function has no memory that it ever ran.

yield breaks this contract entirely.

The Normal Function: A Sprint

def countdown_list(n: int) -> list[int]:
    result = []
    while n > 0:
        result.append(n)
        n -= 1
    return result  # Hands you everything at once, then dies

One call. One massive result. The function's stack frame is created, used, and destroyed.

The Generator: A Pause Button

def countdown(n: int):
    while n > 0:
        yield n    # Pause here, hand back n, wait to be resumed
        n -= 1

The moment Python sees yield in a function body, the rules change. Calling countdown(5) no longer executes a single line of code. Instead, Python hands you back a generator object—a suspended, ready-to-run machine.

gen = countdown(5)
print(gen)          # <generator object countdown at 0x7f...>
print(next(gen))    # 5  → Runs until yield, pauses, returns 5
print(next(gen))    # 4  → Resumes, runs until yield, pauses, returns 4
print(next(gen))    # 3  → Same

What makes this possible? When a generator pauses at yield, its entire stack frame—local variables, current line number, the value of n—is moved from the stack to the heap. It doesn't disappear. It waits, frozen in time, until next() is called again.

Normal function:    [Stack Frame] → return → [Destroyed]

Generator:          [Stack Frame] → yield → [Moved to Heap, frozen]
                                          ↓
                    next() called  → [Thawed, execution resumes]
                                          ↓
                                   yield → [Frozen again]

Old Way vs. New Way: Side by Side

Before generators, you had to implement the iterator protocol manually—a verbose class with __iter__ and __next__:

# THE OLD WAY: Class-based iterator (20 lines of boilerplate)
class Countdown:
    def __init__(self, start: int) -> None:
        self.current = start

    def __iter__(self):
        return self

    def __next__(self) -> int:
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

# THE NEW WAY: Generator function (4 lines, zero boilerplate)
def countdown(n: int):
    while n > 0:
        yield n
        n -= 1

Same behavior. Same memory efficiency. Same protocol compatibility—countdown(5) works anywhere Countdown(5) does.

The yield keyword is a class-based iterator, fully implemented, in one line.

2. Infinite Data Pipelines: The "Pull" Model

Here's where generators move from "interesting" to "indispensable."

Consider the difference:

import sys

# Eager: allocate the entire sequence in RAM
big_list = [x ** 2 for x in range(1_000_000)]
print(f"List size:      {sys.getsizeof(big_list):>12,} bytes")  
# List size:       8,448,728 bytes (~8 MB)

# Lazy: a tiny object that knows *how* to produce values
big_gen = (x ** 2 for x in range(1_000_000))
print(f"Generator size: {sys.getsizeof(big_gen):>12,} bytes")
# Generator size:         104 bytes (~104 bytes)

Eight megabytes vs. 104 bytes. The generator doesn't store the squares—it stores the recipe for producing the next one. Scale this to 10GB of log files or a live market feed, and this difference is what separates a working system from a crashed one.

Generator Expressions: Lazy List Comprehensions

The (x for x in ...) syntax is a generator expression—the lazy sibling of list comprehensions.

# List comprehension: eager, executes immediately
squares_list = [x ** 2 for x in range(10)]    # All 10 computed NOW

# Generator expression: lazy, executes on demand
squares_gen  = (x ** 2 for x in range(10))    # NONE computed yet

Square brackets → list (eager). Parentheses → generator (lazy).

The Pipeline Architecture

Generators compose naturally into pipelines—a chain of lazy transformations where data flows through only when pulled from the end:

  Source          Filter               Processor
    │                │                     │
market_ticker() ──► (t for t              ──► trading logic
[infinite stream]    if t == 'AAPL')          (processes one
                    [lazy filter]              tick at a time)

def market_ticker():
    """Simulates an infinite stream of market data."""
    import random, itertools
    symbols = ['AAPL', 'GOOG', 'MSFT', 'AMZN']
    for i in itertools.count():
        yield {
            'symbol': random.choice(symbols),
            'price': round(random.uniform(100, 300), 2),
            'volume': random.randint(100, 10000),
        }

# Build the pipeline — nothing executes yet
ticker    = market_ticker()                             # Source: infinite generator
aapl_only = (t for t in ticker if t['symbol'] == 'AAPL') # Filter: lazy expression

# Data flows ONLY when we pull from the end
tick = next(aapl_only)  # NOW it runs: pulls from ticker until it finds AAPL
print(tick)  # {'symbol': 'AAPL', 'price': 172.34, 'volume': 4821}

Nothing ran when we built the pipeline. No data was fetched, no filtering occurred. The entire chain is dormant until we call next(). This is lazy evaluation—the pipeline pulls data through only as fast as you consume it.

This is how you process a terabyte log file with 104 bytes of working memory.

3. The Deep Dive: Generators as Coroutines

Everything above treats generators as producers: you pull data out of them via next().

But generators can also be consumers: you push data into them via .send(). This transforms a generator from a simple stream into a stateful processing unit—what computer scientists call a coroutine.

`yield` as an Expression

Normally, yield value is a statement—it sends a value out. But it can also be an expression that receives a value:

def accumulator():
    total = 0
    while True:
        value = (yield total)  # Pause: send out total, wait to receive a value
        if value is not None:
            total += value

acc = accumulator()
next(acc)        # Prime the coroutine (advance to first yield)
acc.send(10)     # Push 10 in → total becomes 10
acc.send(20)     # Push 20 in → total becomes 30
result = acc.send(5)
print(result)    # 35

The priming step (next(acc)) is required. A fresh generator is frozen at the start of the function, before any yield has been reached. You must advance it to the first yield before you can send anything to it.

The Three Generator Controls

Operation	Syntax	What it does
Pull	`next(gen)`	Resume, run until next `yield`, return yielded value
Push	`gen.send(val)`	Resume with `val` as the result of `yield`, run until next `yield`
Throw	`gen.throw(ExcType)`	Resume by raising an exception at the yield point
Close	`gen.close()`	Throw `GeneratorExit`, shut down cleanly

The .throw() method is particularly powerful. Instead of crashing your pipeline when bad data appears, you can inject the error directly at the coroutine's pause point and let it handle recovery internally.

Building a Finite State Machine with `yield`

A coroutine's "current line number" is its state. No state variables. No if state == "WATCHING" branching at the top. The control flow itself encodes the state.

from enum import Enum, auto

class BotState(Enum):
    WATCHING = auto()
    ACTIVE   = auto()

def trading_bot(entry_threshold: float = 150.0,
                exit_threshold:  float = 200.0) -> None:
    """
    A coroutine FSM with two states:
      WATCHING → waiting for a low price to enter a position
      ACTIVE   → holding a position, waiting to exit at profit
    """
    print("[BOT] Initialized. State: WATCHING")
    entry_price: float = 0.0

    while True:
        try:
            # ── STATE: WATCHING ──────────────────────────────
            while True:
                price: float = (yield)               # Wait for next tick
                print(f"[WATCHING] AAPL @ ${price:.2f}")
                if price <= entry_threshold:
                    entry_price = price
                    print(f"[SIGNAL] Entry at ${entry_price:.2f} → switching to ACTIVE")
                    break                            # Transition to ACTIVE

            # ── STATE: ACTIVE ────────────────────────────────
            while True:
                price = (yield)                      # Wait for next tick
                pnl   = price - entry_price
                print(f"[ACTIVE]   AAPL @ ${price:.2f} | PnL: ${pnl:+.2f}")
                if price >= exit_threshold:
                    print(f"[SIGNAL] Exit at ${price:.2f} | Profit: ${pnl:.2f} → switching to WATCHING")
                    break                            # Transition back to WATCHING

        except ValueError as e:
            # Bad tick injected via .throw() — reset to WATCHING without crashing
            print(f"[ERROR] Bad data received: {e}. Resetting to WATCHING.")
            entry_price = 0.0
            # Loop continues: back to WATCHING state

4. The Grand Finale: The High-Frequency Trading Bot

Let's wire everything together. Four components. One elegant pipeline.

The Architecture

┌─────────────────────────────────────────────────────────┐
│                    PIPELINE OVERVIEW                     │
│                                                          │
│  [Source]          [Filter]           [Sink]             │
│  market_ticker()──►aapl_stream ──────►trading_bot()      │
│  (Generator)       (Gen. Expression)  (Coroutine FSM)    │
│       │                  │                  │            │
│   Produces all       Passes only        Consumes AAPL   │
│   symbols lazily     AAPL ticks         ticks, manages  │
│                                         state internally │
│                                                          │
│                   [Bridge]                               │
│                   for loop with                          │
│                   .send() / .throw()                     │
└─────────────────────────────────────────────────────────┘

The Complete System

import random
import itertools
import sys

# ── COMPONENT 1: THE SOURCE ──────────────────────────────────────
def market_ticker():
    """Infinite stream of market ticks. Never terminates."""
    symbols = ['AAPL', 'GOOG', 'MSFT', 'AMZN']
    for _ in itertools.count():
        yield {
            'symbol': random.choice(symbols),
            'price':  round(random.uniform(100, 300), 2),
        }

# ── COMPONENT 2: THE FILTER ──────────────────────────────────────
def build_pipeline():
    ticker    = market_ticker()
    # Occasionally inject a bad tick to test error handling
    aapl_stream = (
        t for t in ticker
        if t['symbol'] == 'AAPL'
    )
    return aapl_stream

# ── COMPONENT 3: THE CONSUMER (FSM Coroutine) ────────────────────
# (trading_bot as defined in Section 3 above)

# ── COMPONENT 4: THE BRIDGE ──────────────────────────────────────
def run(tick_limit: int = 30) -> None:
    stream = build_pipeline()
    bot    = trading_bot(entry_threshold=150.0, exit_threshold=200.0)

    # Prime the coroutine — advance it to the first yield
    next(bot)

    ticks_processed = 0
    for tick in stream:
        if ticks_processed >= tick_limit:
            break

        price = tick['price']

        # Simulate occasional bad data (1-in-10 chance)
        if random.random() < 0.1:
            bad_price = -abs(price)  # Corrupt tick: negative price
            try:
                bot.throw(ValueError(f"Negative price: {bad_price}"))
                next(bot)            # Re-prime after error recovery
            except StopIteration:
                print("[BRIDGE] Bot shut down during error recovery.")
                break
        else:
            try:
                bot.send(price)      # Normal operation: push price to bot
            except StopIteration:
                print("[BRIDGE] Bot has shut down.")
                break

        ticks_processed += 1

    bot.close()  # Send GeneratorExit — clean shutdown
    print(f"\n[BRIDGE] Pipeline complete. Processed {ticks_processed} AAPL ticks.")

if __name__ == "__main__":
    run(tick_limit=30)

Why `.throw()` is the Pro Pattern

Most tutorials show .send() and call it a day. But .throw() is what makes a coroutine-based FSM production-grade.

The alternative to .throw() is sentinel values:

# Naive approach: use magic values to signal errors
bot.send(-1)  # Hope the bot understands -1 means "bad data"

This is fragile. It poisons your data channel with control signals. What if -1 is a legitimate (if unusual) price? What if you need to distinguish between different error types?

.throw() keeps the error channel and the data channel separate:

# The Bridge: clear separation of concerns
for tick in stream:
    if tick['price'] < 0:
        bot.throw(ValueError(f"Corrupt tick: {tick}"))  # Error channel
    else:
        bot.send(tick['price'])                          # Data channel

The coroutine catches it in a try/except at its yield point—exactly like a normal function. The state machine resets cleanly. The pipeline keeps running. Zero sentinel values. Zero ambiguity.

Conclusion: Why This Matters

Let's close with three concrete reasons this mental model changes how you write code.

1. Memory Efficiency: O(1) Space for Infinite Streams

# This processes a 10GB log file in constant memory
def find_errors(path: str):
    with open(path) as f:
        yield from (line for line in f if "ERROR" in line)

for error_line in find_errors("application.log"):
    alert(error_line)

No list. No .readlines(). A single line lives in memory at a time.

2. State Management: Your Line Number Is Your State

The trading_bot() coroutine has zero explicit state variables for its FSM transitions. The while True loop it's currently executing in is the state. Python's own call stack manages it.

Compare that to the class-based alternative:

# The non-generator version: manual state management
class TradingBot:
    def __init__(self):
        self.state = "WATCHING"    # Explicit state
        self.entry_price = 0.0

    def process(self, price: float) -> None:
        if self.state == "WATCHING":   # State checks everywhere
            ...
        elif self.state == "ACTIVE":
            ...

More code. More surface area for bugs. More branching to read and maintain.

3. Composability: UNIX Pipes for Your Data

Each component in our pipeline does exactly one thing: the source generates ticks, the filter screens symbols, the bot manages trades. They're connected by convention—the iterator protocol—not by inheritance or tight coupling.

You can swap any component without touching the others:

# Swap source: real broker API instead of random data
ticker = broker_api.stream()           # Same interface, different source

# Swap filter: multiple symbols
stream = (t for t in ticker if t['symbol'] in {'AAPL', 'MSFT'})

# Swap sink: logging bot instead of trading bot
bot = audit_logger(output="trades.log")  # Same .send() interface

Small tools. Single responsibilities. Glued by protocol.

Python Internals: The Iterator Protocol

aykhlf yassir — Sun, 15 Feb 2026 17:34:27 +0000

Stop Thinking in Lists. Start Thinking in Streams.

Here's a question that separates junior Python developers from senior ones:

What happens when you run for item in my_list?

If your answer was "Python loops over the list," you're not wrong—but you're describing the what, not the how. And the how is where the real engineering lives.

The for loop is syntactic sugar. Under the hood, it's a precise protocol—a contract between your code and Python's runtime. Once you understand that contract, you'll never load a 10GB file into memory again, your pipelines will run in constant memory, and you'll finally understand why generators feel like magic.

By the end of this post, you'll know:

The difference between an iterable and an iterator
What Python actually executes when you write a for loop
Why the same iterator can't be looped twice
How to build your own memory-efficient data pipelines from scratch

Let's pull back the curtain.

1. The Definitions:

The words "iterable" and "iterator" are used interchangeably in most tutorials. This can lead to confusion. Let's define them precisely.

The Analogy

Think of a book and a bookmark.

The book (Iterable) contains all the data. It can be read from the beginning any number of times. You can hand it to anyone, and they can start reading from page one. A list, a tuple, a str—these are all books.

The bookmark (Iterator) tracks where you are in a specific reading session. It has state. It knows you're on page 47. It can tell you the next word, then advance one position. Crucially: there is only one reading session. When you reach the last page, the bookmark is spent. Generators, file objects, map() results—these are all bookmarks.

The Interface

Python makes this concrete through two dunder methods:

Concept	Must Implement	Behavior
Iterable	`__iter__`	Returns a new Iterator
Iterator	`__iter__` AND `__next__`	`__next__` returns the next value, raises `StopIteration` when exhausted

# A list is an ITERABLE: it implements __iter__
my_list = [1, 2, 3]
print(hasattr(my_list, '__iter__'))  # True
print(hasattr(my_list, '__next__'))  # False - not an iterator!

# Calling __iter__ on a list creates an ITERATOR
my_iterator = iter(my_list)
print(hasattr(my_iterator, '__iter__'))  # True
print(hasattr(my_iterator, '__next__'))  # True - now we have both

The iter() built-in calls __iter__. The next() built-in calls __next__. Everything else in Python is built on top of these two primitives.

2. The Mechanics: Deconstructing the `for` Loop

Here is the lie Python tells you every day:

# What you write (The Sugar)
for item in [1, 2, 3]:
    print(item)

Here is what Python actually executes:

# What Python runs (The Reality)
_iter = iter([1, 2, 3])   # Step 1: Call __iter__, get an iterator

while True:                # Step 2: Loop forever
    try:
        item = next(_iter) # Step 3: Call __next__, get the next item
        print(item)        # Step 4: Execute the loop body
    except StopIteration:  # Step 5: Iterator is exhausted
        break              # Step 6: Exit the loop

That's it. That's the entire for loop. No magic. No special knowledge of lists or tuples or strings. Just a mechanical protocol: call __iter__ once, call __next__ repeatedly until StopIteration is raised.

The Critical Insight: The Loop is Blind

The for loop does not care about your data structure. It cannot see whether you gave it a list, a database cursor, a network socket, or a custom class you wrote this morning.

It only asks one question: "Does this object speak the iterator protocol?"

If __iter__ and __next__ exist, the loop works. This is Python's duck typing philosophy applied at the language level. No inheritance required. No registration required. Just implement two methods.

# COMPLETELY custom iteration - no list, no built-in involved
class Countdown:
    def __init__(self, start: int) -> None:
        self.current = start

    def __iter__(self):
        return self  # More on this shortly

    def __next__(self) -> int:
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

# The for loop has NO IDEA this is a custom class
for n in Countdown(5):
    print(n)  # 5, 4, 3, 2, 1

3. Why Does `iter` Return `self`?

If you look carefully at the Countdown class above, you'll notice something odd: __iter__ returns self. An iterator returning itself as an iterator.

If an iterable is a factory that creates iterators, and an iterator is a consumer that gets exhausted... why does an iterator pretend to be an iterable?

The Answer: Polymorphism

Consider this function:

def print_first_three(iterable):
    it = iter(iterable)  # Always call iter() first
    print(next(it))
    print(next(it))
    print(next(it))

If we follow the "always call iter() first" convention, this function works with both iterables and iterators:

# Works with an iterable (list)
print_first_three([10, 20, 30, 40, 50])

# Works with an iterator (generator, file, network stream)
my_stream = iter([10, 20, 30, 40, 50])
print_first_three(my_stream)

The second call works because when you call iter() on an iterator, it returns self. The protocol stays consistent. The caller doesn't need to know—or care—whether it received a fresh list or a half-consumed stream.

This is the rule, codified:

Iterable:  __iter__() → returns a NEW Iterator (fresh reading session)
Iterator:  __iter__() → returns SELF     (I am already the session)

Object	`__iter__`	`__next__`
`list`, `tuple`, `str`	Returns a new list_iterator	✗ Not present
`list_iterator`	Returns `self`	✓ Advances position
Generator	Returns `self`	✓ Runs until `yield`
File object	Returns `self`	✓ Reads next line

In short: Every iterator is also iterable. Not every iterable is an iterator.

4. The "One-Shot" Trap:

Here is a bug that has shipped to production more times than anyone will admit:

squares = map(lambda x: x**2, [1, 2, 3])

print(list(squares))  # [1, 4, 9] ✓
print(list(squares))  # []        ✗ Silent failure

The second call returns an empty list. No error. No warning. Just silently wrong data.

Why? Because map() returns an iterator—a bookmark, not a book. The first list() call consumes every item, advancing the bookmark to the end. The second call finds the iterator exhausted. StopIteration is raised immediately. list() catches it and returns [].

This trap catches developers with:

# All of these are ONE-SHOT iterators:
squares = map(lambda x: x**2, range(5))
evens = filter(lambda x: x % 2 == 0, range(10))
pairs = zip([1, 2, 3], ['a', 'b', 'c'])

lines = open("data.txt")  # File objects too!

The Fix: Know What You Have

The decision tree is simple:

Do I need to iterate this data multiple times?
├── YES → Store it: data = list(my_iterator)
└── NO  → Stream it: consume it once, discard it

# If you need multiple passes: materialize to list
squares = list(map(lambda x: x**2, [1, 2, 3]))
print(squares)  # [1, 4, 9]
print(squares)  # [1, 4, 9] - Works! It's a list now.

# If one pass is enough: stay lazy, save memory
for square in map(lambda x: x**2, range(10_000_000)):
    process(square)  # Never loads 10M items into RAM

The Generator Version of the One-Shot Trap

Generator functions are iterators, so they have the same behavior:

def squares(n: int):
    for i in range(n):
        yield i ** 2

gen = squares(5)

print(sum(gen))   # 30 ✓
print(sum(gen))   # 0  ✗ - exhausted!

Every time you call squares(5), you get a new generator object—a fresh bookmark. But if you assign it to a variable and reuse that variable, you're sharing one bookmark between two consumers.

5. The Toolkit: `itertools` — Don't Reinvent the Wheel

Once you think in streams, you'll want to transform, combine, and slice them. Python's itertools module is the standard library for doing exactly this—all lazily, all in O(1) memory.
itertools are

The itertools module is high performance C code exposed to Python. Using it is almost always faster than writing your own loop logic.

Here are the three tools you'll reach for every week:

`itertools.count` — Infinite Streams

import itertools

# Infinite counter: 10, 11, 12, 13, ...
counter = itertools.count(start=10, step=1)

# Always pair with islice or a break condition
for n in counter:
    if n > 15:
        break
    print(n)  # 10, 11, 12, 13, 14, 15

Use this anywhere you'd use while True with a manual counter.

`itertools.chain` — Linking Streams

import itertools

# Process multiple files as one continuous stream
# NO intermediate list created
log_lines = itertools.chain(
    open("january.log"),
    open("february.log"),
    open("march.log"),
)

for line in log_lines:
    process(line)  # Streams through all three files sequentially

The alternative—open("jan") + open("feb") — doesn't work on file objects. chain is the correct tool for combining any iterables without materializing them, good stuff.

`itertools.islice` — Safe Consumption

import itertools

# Take the first 5 from any iterator, safely
def squares():
    n = 0
    while True:
        yield n ** 2
        n += 1

# Without islice, this would run forever
first_five = list(itertools.islice(squares(), 5))
print(first_five)  # [0, 1, 4, 9, 16]

islice is the lazy equivalent of my_list[:5]. It doesn't generate all items first—it stops consuming after n items.

The Combinators at a Glance

Tool	What it does	Memory
`count(n)`	Infinite incrementing stream	O(1)
`cycle(it)`	Repeat iterable forever	O(n)
`repeat(x, n)`	Yield x exactly n times	O(1)
`chain(*its)`	Concatenate iterables	O(1)
`islice(it, n)`	Take first n items lazily	O(1)
`takewhile(pred, it)`	Take while condition holds	O(1)
`dropwhile(pred, it)`	Skip while condition holds	O(1)
`batched(it, n)`	Group into n-sized chunks	O(n)

Conclusion:

Let's close with the mental model shift.

Junior mindset: A for loop iterates over a collection.

Senior mindset: A for loop drives a protocol. Any object that speaks the protocol—regardless of what it is or where the data lives—can be iterated. The data could be in a list in RAM, a file on disk, a socket on the network, a database cursor, or computed on the fly from a mathematical formula. The loop doesn't care.

This is the power:

# These all work identically with the for loop:
for item in [1, 2, 3]:                          # In memory
for line in open("data.csv"):                   # On disk
for record in database.execute("SELECT ..."):   # In database
for packet in socket.recv_packets():            # On network
for value in itertools.count():                 # Computed infinitely

Same protocol. Same syntax. Radically different resource profiles.

And the payoff in memory:

# Junior code: loads entire file
def count_errors(path: str) -> int:
    lines = open(path).readlines()  # 10GB in RAM 💀
    return sum(1 for line in lines if "ERROR" in line)

# Senior code: streams the file
def count_errors(path: str) -> int:
    with open(path) as f:           # ~constant memory ✓
        return sum(1 for line in f if "ERROR" in line)

Both produce the same number. One will kill your server at 3AM.

Your Action Item

Go find that one function in your codebase that does this:

data = list(some_query_or_file_or_api_call())
for item in data:
    process(item)

If you never use data again after the loop, that list() call is pure waste. Delete it. Stream directly. Your memory graph will thank you.

The for loop doesn't need the list. It never did.

This is part of an ongoing series on Python internals. If this post changed how you think about loops, follow along.

Python Internals: Type Hints

aykhlf yassir — Sat, 07 Feb 2026 19:46:45 +0000

From dynamic chaos to static certainty: mastering MyPy and strict mode

The "I Forgot What This Returns" Problem

You're six months into a project. You need to call a function someone else wrote:

def process(data):
    # ... 50 lines of code ...
    return result

What does data need to be? A list? A dict? A pandas DataFrame? A custom class?

What does result actually contain? A boolean? A tuple? None?

You have three options:

Read all 50 lines of implementation code
Find example usage somewhere else in the codebase
Run it and hope for the best

All three are terrible.

The Solution

def process(data: pd.DataFrame) -> tuple[bool, str]:
    # ... 50 lines of code ...
    return result

Now you know instantly:

Input: pandas DataFrame
Output: tuple of (bool, str)

No runtime cost. No reading code.

The Misconception

Many developers think type hints are for Python—that they change how the code runs. They don't.

def add(a: int, b: int) -> int:
    return a + b

print(add("hello", "world"))  # "helloworld" - Python doesn't care!

Python completely ignores type hints at runtime. They're not constraints; they're documentation with a superpower: they can be automatically verified.

Type hints are for:

Your IDE - Autocomplete and error detection
MyPy - Static type checker that finds bugs before runtime
Other developers - Understanding your code without reading it
Future you - Remembering what you meant six months ago

The Modern Syntax (Python 3.10+)

If you learned Python type hints before 2021, forget what you know. The syntax changed dramatically in Python 3.9 and 3.10.

The Old Way (Verbose and Ugly)

from typing import List, Dict, Union, Optional, Tuple

def get_user(user_id: Union[int, None]) -> Dict[str, Union[str, int]]:
    pass

def process_items(items: Optional[List[str]]) -> Tuple[bool, str]:
    pass

This required importing half of the typing module and using capital letters for built-in types (List instead of list).

The New Way (Clean and Native)

def get_user(user_id: int | None) -> dict[str, str | int]:
    pass

def process_items(items: list[str] | None) -> tuple[bool, str]:
    pass

What changed:

Union syntax: Union[int, None] → int | None
Optional is dead: Optional[str] → str | None
Lowercase collections: List[int] → list[int], Dict[str, int] → dict[str, int]
Tuple syntax: Tuple[bool, str] → tuple[bool, str]

The Compatibility Timeline

Python 3.9: Can use list[int], dict[str, int] but not | for unions
Python 3.10+: Can use | for unions, making Union and Optional obsolete
Python 3.12+: New generic syntax for classes (we'll cover this)

If you're on Python 3.10+, never import Union or Optional again.

Generic Collections: The Cheat Sheet

# Lists - homogeneous collection
numbers: list[int] = [1, 2, 3]
names: list[str] = ["Alice", "Bob"]
mixed: list[int | str] = [1, "two", 3]

# Dictionaries - key and value types
scores: dict[str, int] = {"Alice": 100, "Bob": 95}
config: dict[str, str | int | bool] = {"debug": True, "port": 8080}

# Tuples - fixed length and types
point: tuple[float, float] = (1.0, 2.0)
user: tuple[int, str, bool] = (1, "Alice", True)

# Tuples - variable length, same type
coordinates: tuple[float, ...] = (1.0, 2.0, 3.0, 4.0)  # Any number of floats

# Sets
unique_ids: set[int] = {1, 2, 3}

The ellipsis (...) in tuple types means "zero or more elements of this type":

tuple[int, int] - exactly 2 integers
tuple[int, ...] - any number of integers

The "Chicken and Egg" Problem: Forward References

Try to write a method that returns the same type as the class:

class Money:
    def __init__(self, amount: float, currency: str):
        self.amount = amount
        self.currency = currency

    def __add__(self, other: Money) -> Money:  # NameError!
        return Money(self.amount + other.amount, self.currency)

Boom. NameError: name 'Money' is not defined.

Why? Because when Python is executing the class body, the class doesn't exist yet. It's being defined. So other: Money tries to reference a name that doesn't exist.

The Old Fix (Quotes)

The traditional workaround was to use strings:

class Money:
    def __add__(self, other: "Money") -> "Money":  # Works!
        return Money(self.amount + other.amount, self.currency)

Strings aren't evaluated at import time they're just stored. MyPy evaluates them later when the class exists.

But this is ugly and error-prone (typos don't cause errors until type checking).

The New Fix (Future Annotations)

Add one line at the top of your file:

from __future__ import annotations

class Money:
    def __init__(self, amount: float, currency: str):
        self.amount = amount
        self.currency = currency

    def __add__(self, other: Money) -> Money:  # Works!
        return Money(self.amount + other.amount, self.currency)

This directive tells Python: "Treat all type annotations as strings automatically."

Under the hood, Python stores Money as the string "Money", but you write it without quotes. MyPy evaluates it later when type checking.

Why This Matters

Without this, you'd need quotes everywhere:

class TreeNode:
    def __init__(self, value: int):
        self.value = value
        self.left: "TreeNode | None" = None
        self.right: "TreeNode | None" = None

    def add_child(self, node: "TreeNode") -> None:
        pass

With from __future__ import annotations:

from __future__ import annotations

class TreeNode:
    def __init__(self, value: int):
        self.value = value
        self.left: TreeNode | None = None
        self.right: TreeNode | None = None

    def add_child(self, node: TreeNode) -> None:
        pass

Always use this import in every file with type hints. It will become the default behavior in Python 4.0.

The `Any` vs `object` Distinction

When you want to accept "anything," you have two options with very different meanings.

`object` - The Safe Universal Type

def print_anything(value: object) -> None:
    print(value)

print_anything(42)        # OK
print_anything("hello")   # OK
print_anything([1, 2, 3]) # OK

When you type something as object, you're saying:

"I accept any type"
"But I can only do things that work on ANY object"

What can you do with an object?

def process(value: object) -> None:
    print(value)           # OK - all objects can be printed
    str(value)            # OK - all objects can be converted to string
    value == 5            # OK - all objects support equality

    # These are errors:
    value + 5             # Error! Not all objects support addition
    value.some_method()   # Error! Not all objects have this method
    value[0]              # Error! Not all objects are indexable

MyPy knows that object only supports the universal protocol: __str__, __repr__, __eq__, etc.

`Any` - The "Turn Off Type Checking" Escape Hatch

from typing import Any

def process(value: Any) -> None:
    print(value)           # OK
    value + 5             # OK - MyPy assumes you know what you're doing
    value.some_method()   # OK - MyPy trusts you
    value[0]              # OK - MyPy allows everything

Any tells MyPy: "Don't check this variable. I know what I'm doing."

It's dangerous because it disables type safety:

def broken(x: Any) -> int:
    return x + 5  # MyPy says this is fine

broken("hello")  # Runtime error: can't add string and int

When to Use Each

Use object when:

You truly accept any type
You only use universal operations
Example: __eq__ methods that compare with anything

def __eq__(self, other: object) -> bool:
    if not isinstance(other, Money):
        return NotImplemented
    return self.amount == other.amount

Use Any when:

You're working with truly dynamic data (JSON from API)
You're gradually typing legacy code
You're interfacing with untyped libraries
You've exhausted other options

import json

def load_config(path: str) -> Any:  # JSON can be anything
    with open(path) as f:
        return json.load(f)

The Gradual Typing Strategy

When retrofitting types to legacy code:

# Stage 1: Everything is Any
def process_data(data: Any) -> Any:
    result = transform(data)
    return result

# Stage 2: Narrow the outputs
def process_data(data: Any) -> dict[str, int]:
    result = transform(data)
    return result

# Stage 3: Narrow the inputs
def process_data(data: list[str]) -> dict[str, int]:
    result = transform(data)
    return result

# Stage 4: No more Any!

Start with Any everywhere, then gradually replace it with specific types as you understand the code.

MyPy Configuration

Type hints are useless unless you enforce them. That's where MyPy comes in, a static type checker that analyzes your code without running it.

Installing MyPy

pip install mypy

Basic Usage

# Check a single file
mypy my_script.py

# Check a directory
mypy src/

# Check and show error codes
mypy --show-error-codes .

The Configuration File

Create a pyproject.toml file (or mypy.ini) in your project root:

[tool.mypy]
python_version = "3.10"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true

# Strict mode - the final boss
strict = true

Let's break down what these do:

disallow_untyped_defs = true - Every function MUST have type hints:

# Error: Function is missing a type annotation
def process(data):
    return data.upper()

# OK
def process(data: str) -> str:
    return data.upper()

warn_return_any = true - Functions can't return Any:

import json

# Error: Returning Any from function declared to return int
def load_number() -> int:
    return json.load(open("config.json"))["number"]  # json.load returns Any

# OK
def load_number() -> int:
    value: Any = json.load(open("config.json"))["number"]
    return int(value)  # Explicit conversion

strict = true - Enables ALL strict checks:

# Error: Need to handle None case
def get_first(items: list[str]) -> str:
    return items[0]  # What if items is empty?

# OK
def get_first(items: list[str]) -> str | None:
    return items[0] if items else None

The `NotImplemented` Problem

Remember that arithmetic operations should return NotImplemented, not raise exceptions? MyPy doesn't like this:

def __add__(self, other: Money) -> Money:
    if isinstance(other, Money):
        return Money(self.amount + other.amount, self.currency)
    return NotImplemented  # Error: Incompatible return type!

MyPy says: "You declared this returns Money, but you're returning NotImplemented!"

The fix is to be explicit about both return types:

from typing import NotImplementedType

def __add__(self, other: object) -> Money | NotImplementedType:
    if isinstance(other, Money):
        return Money(self.amount + other.amount, self.currency)
    return NotImplemented

Note we also changed other: Money to other: object—we accept any type and check it at runtime.

Handling Third-Party Libraries

Many packages don't have type stubs. MyPy will complain:

error: Skipping analyzing "requests": module is installed, but missing library stubs

Solutions:

[tool.mypy]
# Option 1: Ignore missing imports globally
ignore_missing_imports = true

# Option 2: Configure per-package
[[tool.mypy.overrides]]
module = "requests.*"
ignore_missing_imports = true

# Option 3: Install type stubs
# pip install types-requests

Python Internals: Reference Cycles and Garbage Collection

aykhlf yassir — Thu, 05 Feb 2026 16:59:44 +0000

Understanding reference counting, garbage collection, and why your objects won't die

The Box vs. The Label

Quick: what does this code do?

a = [1, 2, 3]
b = a
b.append(4)
print(a)  # What prints?

If you're coming from C++ or Java, you might think "I assigned a to b, so they're separate copies. a should still be [1, 2, 3]."

But run this code and you'll see [1, 2, 3, 4].

Why? Because your mental model is wrong.

Variables Are Not Boxes

In C++, a variable is like a box with a name written on it. When you write int x = 5;, you create a box labeled "x" and put the value 5 inside it. When you assign int y = x;, you create a new box labeled "y" and copy the value into it.

Python doesn't work this way.

In Python, objects live in the heap. Variables are not boxes that contain objects; they're labels attached to objects.

a = [1, 2, 3]

What actually happens:

Python creates a list object [1, 2, 3] somewhere in memory
Python creates a label (reference) named a
Python sticks that label onto the object

When you write:

b = a

You're not copying the object. You're creating a second label b and sticking it onto the same object that a is pointing to.

    [1, 2, 3, 4] ← Object in memory
      ↑     ↑
      a     b   ← Two labels on one object

This is why modifying through b affects what you see through a, they're both pointing at the same object.

Proving It With `id()`

Every object in Python has a unique identifier its memory address:

a = [1, 2, 3]
b = a

print(id(a))  # 140234567890
print(id(b))  # 140234567890 - Same address!
print(a is b) # True - Same object

c = [1, 2, 3]
print(id(c))  # 140234567999 - Different address!
print(a is c) # False - Different objects
print(a == c) # True - Same contents

The is operator checks if two labels point to the same object, not whether the objects have the same value.

The Immutability Exception

This model explains why immutable types seem to behave differently:

x = 5
y = x
y = 10
print(x)  # Still 5!

Did the label model break? No! When you write y = 10, you're not modifying the object, you're unsticking the label y from the object 5 and re-sticking it to a different object 10. The original object 5 is unchanged, and x still points to it.

Before y = 10:
    5 ← Object
    ↑
   x,y

After y = 10:
    5 ← Object    10 ← New object
    ↑              ↑
    x              y

Immutable objects can't be modified, so every operation that looks like modification is actually creating a new object and moving the label.

Mutable Default Arguments

Now that you understand the label model, let's look at Python's most infamous gotcha:

def add_passenger(passenger, manifest=[]):
    manifest.append(passenger)
    return manifest

flight1 = add_passenger("Alice")
print(flight1)  # ['Alice']

flight2 = add_passenger("Bob")
print(flight2)  # ['Alice', 'Bob'] - WHAT?!

Why is Alice on Bob's flight?

The Hidden Function Object

When Python executes a def statement, it doesn't just "define a function and forget it." It creates a function object that lives in memory. Default argument values are created once when the function is defined and stored inside that function object.

Let's prove it:

def add_passenger(passenger, manifest=[]):
    manifest.append(passenger)
    return manifest

# The function object has a __defaults__ attribute
print(add_passenger.__defaults__)  # ([],)

# Call it once
add_passenger("Alice")
print(add_passenger.__defaults__)  # (['Alice'],)

# Call it again
add_passenger("Bob")
print(add_passenger.__defaults__)  # (['Alice', 'Bob'],)

The default list [] is created once, when def executes. Every call to the function that doesn't provide a manifest argument reuses that same list object.

The Execution Timeline

# DEFINITION TIME (happens once)
def add_passenger(passenger, manifest=[]):  # Create empty list, store in __defaults__
    manifest.append(passenger)
    return manifest

# CALL TIME 1
add_passenger("Alice")  # No manifest provided, use default (the list in __defaults__)
# The list is now ['Alice']

# CALL TIME 2  
add_passenger("Bob")    # No manifest provided, use default (SAME list!)
# The list is now ['Alice', 'Bob']

The Idiomatic Fix

Never use mutable objects as default arguments. Use None as a sentinel:

def add_passenger(passenger, manifest=None):
    if manifest is None:
        manifest = []  # Create a NEW list for each call
    manifest.append(passenger)
    return manifest

flight1 = add_passenger("Alice")
print(flight1)  # ['Alice']

flight2 = add_passenger("Bob")
print(flight2)  # ['Bob'] - Separate list!

Now each call that doesn't provide manifest creates a fresh, independent list.

When to Use Mutable Defaults (Intentionally)

There are rare cases where you want to preserve state across calls:

def cache_result(key, cache={}):
    if key not in cache:
        print(f"Computing {key}...")
        cache[key] = expensive_computation(key)
    return cache[key]

cache_result(5)  # Computing 5...
cache_result(5)  # (no output - cached!)

But this is almost always better expressed with a class or explicit cache variable.

The Life and Death of an Object: Reference Counting

So we know objects are created in memory and variables are labels pointing to them. But when does an object die?

The Reference Counter

Every Python object has a hidden field called ob_refcnt, a counter tracking how many labels are currently stuck to it.

typedef struct {
    Py_ssize_t ob_refcnt;   // Reference count
    PyTypeObject *ob_type;  // Type
    // ... actual data
} PyObject;

The rules are simple:

Create a label → Count +1
Remove a label → Count -1
Count reaches 0 → Immediate death

Let's trace the life cycle:

import sys

x = object()  # Create object, refcount = 1
print(sys.getrefcount(x))  # 2 (why? read on...)

y = x  # New label, refcount = 2 (+ 1 from getrefcount call = 3)
print(sys.getrefcount(x))  # 3

del y  # Remove label, refcount = 2
print(sys.getrefcount(x))  # 2

del x  # Refcount = 1 (from getrefcount), then 0 when function returns
# Object is immediately destroyed

The `sys.getrefcount()` Trap

Notice the reference count is always higher than expected? This is because calling sys.getrefcount(x) creates a temporary reference when passing x as an argument!

x = []
# Actual refs: x
print(sys.getrefcount(x))  # 2
# Why 2? Because during the call, refs are: x, argument to getrefcount()

The real count is always getrefcount(x) - 1.

What Creates References?

Understanding what increases the reference count is critical:

import sys

obj = object()
print(sys.getrefcount(obj))  # 2 (variable + function arg)

# Assignment creates a reference
another = obj
print(sys.getrefcount(obj))  # 3

# Lists/tuples/dicts create references
lst = [obj]
print(sys.getrefcount(obj))  # 4

# Function arguments create temporary references
def foo(x):
    print(sys.getrefcount(x))  # +1 from function call

foo(obj)  # Will print 5

# Deleting removes references
del another
del lst
print(sys.getrefcount(obj))  # Back to 2

Immediate Destruction

The beauty of reference counting is deterministic cleanup. The moment the last reference disappears, the object dies:

class Mortal:
    def __init__(self, name):
        self.name = name
        print(f"{name} is born")

    def __del__(self):
        # Called when refcount hits 0
        print(f"{name} dies")

print("Creating object...")
x = Mortal("Socrates")  # Socrates is born

print("Deleting reference...")
del x  # Socrates dies - immediately!

print("After deletion")

This is why Python doesn't need explicit free() or delete calls like C/C++. Objects clean up automatically when they're no longer needed.

The Reference Cycles Problem

Reference counting sounds perfect and intuitive. But it has a fatal weakness:

class Node:
    def __init__(self, name):
        self.name = name
        self.partner = None

    def __del__(self):
        print(f"Deleting {self.name}")

# Create two nodes
alice = Node("Alice")
bob = Node("Bob")

# Create a circular reference
alice.partner = bob
bob.partner = alice

# Delete our references
del alice
del bob

# ... crickets ...
# __del__ is never called!

What happened? Let's trace the references:

Before del:
    Node("Alice") ← alice variable
          ↕ partner
    Node("Bob") ← bob variable

After del alice, del bob:
    Node("Alice") 
          ↕ partner
    Node("Bob")

Even after we delete the variables alice and bob, the Node objects still hold references to each other. Their reference counts are still 1 (from the partner attribute), so they never get destroyed.

These objects are now unreachable from our code (we have no variables pointing to them), but they're still in memory, causing a memory leak.

The Real-World Impact

This isn't just theoretical. Reference cycles are common:

# Parent-child relationships
class Parent:
    def __init__(self):
        self.children = []

class Child:
    def __init__(self, parent):
        self.parent = parent
        parent.children.append(self)

# Event listeners
class Button:
    def __init__(self):
        self.click_handler = None

    def on_click(self, handler):
        self.click_handler = handler

# Handler holds button, button holds handler
button = Button()
button.on_click(lambda: print(f"Clicked {button}"))

# Data structures
class TreeNode:
    def __init__(self, value):
        self.value = value
        self.parent = None
        self.left = None
        self.right = None

root = TreeNode(1)
root.left = TreeNode(2)
root.left.parent = root  # Cycle!

The Garbage Collector

Python's solution is the Generational Garbage Collector, which exists specifically to find and destroy reference cycles.

How It Works

The GC periodically:

Scans all objects looking for those that reference each other
Builds a graph of references
Finds cycles (strongly connected components)
Determines if the cycle is reachable from any external reference
If not reachable, destroys the entire cycle

You can trigger it manually:

import gc

class Node:
    def __init__(self, name):
        self.name = name
        self.partner = None

    def __del__(self):
        print(f"Deleting {self.name}")

alice = Node("Alice")
bob = Node("Bob")
alice.partner = bob
bob.partner = alice

del alice, bob
print("After del...")

# Force garbage collection
gc.collect()  # Deleting Alice
              # Deleting Bob
print("After gc.collect()")

Generational Hypothesis

The GC is "generational" because it's based on an empirical observation: most objects die young.

Python divides objects into three generations:

Generation 0: New objects (checked frequently)
Generation 1: Objects that survived one GC pass (checked less often)
Generation 2: Long-lived objects (checked rarely)

import gc

# See the current thresholds
print(gc.get_threshold())  # (700, 10, 10)
# Meaning: Run gen0 after 700 allocations,
#          Run gen1 after 10 gen0 collections,
#          Run gen2 after 10 gen1 collections

# See collection stats
print(gc.get_count())  # (453, 5, 2)
# Current counts in each generation

The Cost of Cycles

The garbage collector isn't free. It has to:

Pause your program to scan objects
Build reference graphs
Traverse cycles

This is why avoiding cycles improves performance:

# Slower: Creates cycles
class Child:
    def __init__(self, parent):
        self.parent = parent
        parent.children.append(self)

# Faster: No cycles
class Child:
    def __init__(self, parent_name):
        self.parent_name = parent_name  # Store name, not reference

Weak References

Imagine building a cache:

cache = {}

def get_user(user_id):
    if user_id not in cache:
        cache[user_id] = load_from_database(user_id)
    return cache[user_id]

Problem: objects in cache never die. Even if nobody else needs them, the cache keeps them alive. This is a memory leak.

Strong vs. Weak References

Think of references like this:

Strong Reference (normal):

Like holding a dog's leash
The dog cannot leave while you hold the leash
The dog exists because you're holding it

Weak Reference:

Like pointing at a dog
You don't prevent the dog from leaving
If the owner (strong reference) leaves, the dog disappears
You're now pointing at nothing (None)

The `weakref` Module

import weakref

class HeavyObject:
    def __init__(self, data):
        self.data = data

    def __del__(self):
        print(f"Deleting {self.data}")

# Strong reference
obj = HeavyObject("important")
print(obj.data)  # "important"

# Weak reference
weak = weakref.ref(obj)
print(weak())  # <__main__.HeavyObject object> - obj still alive

# Delete strong reference
del obj  # Deleting important

# Weak reference now points to nothing
print(weak())  # None

The weak reference doesn't count toward ob_refcnt. When the last strong reference dies, the object is destroyed, even if weak references still exist.

The Optimized Cache: `WeakValueDictionary`

import weakref

cache = weakref.WeakValueDictionary()

class User:
    def __init__(self, user_id, name):
        self.user_id = user_id
        self.name = name

    def __del__(self):
        print(f"User {self.name} deleted from memory")

# Add to cache
user = User(1, "Alice")
cache[1] = user

print(cache[1].name)  # "Alice" - cache works

# Delete the strong reference
del user  # User Alice deleted from memory

# Cache automatically removed the entry!
print(1 in cache)  # False

When user was deleted, its reference count hit 0 (the cache only held a weak reference). The object was destroyed, and WeakValueDictionary automatically removed the entry.

This is how you build caches that don't leak memory.

Use Cases for Weak References

Caches: Store objects without preventing their cleanup
Observer patterns: Listeners shouldn't keep subjects alive
Circular references: Parent doesn't prevent child GC

# Observer pattern with weak refs
class Subject:
    def __init__(self):
        self._observers = []

    def attach(self, observer):
        # Store weak reference, not strong
        self._observers.append(weakref.ref(observer))

    def notify(self):
        # Filter out dead references
        self._observers = [obs for obs in self._observers if obs() is not None]
        for obs_ref in self._observers:
            obs = obs_ref()
            if obs is not None:
                obs.update()

When NOT to Use Weak References

Weak references have limitations:

# Can't create weak refs to basic types
try:
    weak = weakref.ref(42)
except TypeError:
    print("Can't weakref int")

try:
    weak = weakref.ref("hello")
except TypeError:
    print("Can't weakref str")

# Weak refs add overhead
# Don't use them everywhere - only where you need them

Recommended Resource

I highly recommend Nina Zakharenko's "Memory Management in Python" talk from PyCon 2016.

Watch on YouTube

Summary:

We've learned how Python manages object lifecycles:

Mental Models

Variables are labels, not boxes → b = a creates two labels on one object
Mutable defaults are shared → Use None sentinel pattern
Reference counting is immediate → Objects die when count hits 0

The Reference Counting System

Every object has ob_refcnt tracking references
Assignment increments count, deletion decrements
Count = 0 triggers immediate destruction via __del__

The Garbage Collector

Purpose: Find and destroy reference cycles
Strategy: Generational collection (young objects die fast)
Trigger: Automatically or via gc.collect()
Cost: Pause program to scan objects

Weak References

Strong reference: Keeps object alive (normal)
Weak reference: Points to object without preventing cleanup
WeakValueDictionary: Auto-cleaning cache
Use for: Caches, observers, avoiding cycles

The Professional Checklist

When designing classes:

# Avoid cycles in data structures
class TreeNode:
    def __init__(self, value):
        self.value = value
        self.children = []
        # DON'T store self.parent - creates cycle
        # Store parent_id or use weakref instead

# Use weak refs for caches
import weakref
class ResourceManager:
    def __init__(self):
        self._cache = weakref.WeakValueDictionary()

# Clean up resources explicitly
class FileHandler:
    def __enter__(self):
        return self

    def __exit__(self, *args):
        self.cleanup()  # Don't rely on __del__

When writing functions:

# NEVER use mutable defaults
def process_items(items, buffer=[]):  # BAD
    pass

def process_items(items, buffer=None):  # GOOD
    if buffer is None:
        buffer = []

Python Internals: Operator Protocols

aykhlf yassir — Wed, 04 Feb 2026 19:29:05 +0000

How to make your custom objects behave exactly like native types through protocol adherence

The "Native Citizen" Test

Here's how you spot a junior Python developer:

# Junior code
m1 = Money(10, "USD")
m2 = Money(5, "USD")
result = m1.add(m2)  # Java flashbacks intensify

And here's a senior:

# Senior code
m1 = Money(10, "USD")
m2 = Money(5, "USD")
result = m1 + m2  # Feels like native Python

The difference isn't just aesthetics it's understanding Python's protocol-oriented design philosophy.

Python doesn't care if your class inherits from int or float. It only cares if your object behaves like a number. This is duck typing at the language level: if it implements the arithmetic protocol, it's a number. If it implements the sequence protocol, it's a sequence.

Today, we're going to make our custom classes first-class citizens of the Python ecosystem by implementing the magic methods that let them integrate seamlessly with operators, loops, and built-in functions.

The Arithmetic Negotiation: `add` vs `radd`

Let's build a Money class that supports addition:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __add__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot add different currencies")
            return Money(self.amount + other.amount, self.currency)
        return NotImplemented

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

m1 = Money(10, "USD")
m2 = Money(5, "USD")
print(m1 + m2)  # Money(15, 'USD')

Great! But what about this?

m = Money(10, "USD")
result = m + 5  # What should this do?

We could support it:

def __add__(self, other):
    if isinstance(other, Money):
        if self.currency != other.currency:
            raise ValueError("Cannot add different currencies")
        return Money(self.amount + other.amount, self.currency)
    if isinstance(other, (int, float)):
        return Money(self.amount + other, self.currency)
    return NotImplemented

Now m + 5 works. But what about 5 + m?

m = Money(10, "USD")
result = 5 + m  # TypeError: unsupported operand type(s) for +: 'int' and 'Money'

The Dispatch Sequence

Here's what actually happens when you write a + b:

Python calls a.__add__(b)
If that returns NotImplemented, Python calls b.__radd__(a)
If that also returns NotImplemented, Python raises TypeError

So when you do 5 + m:

Python calls (5).__add__(m) → int.__add__ doesn't know about Money, returns NotImplemented
Python calls m.__radd__(5) → We haven't implemented this yet!

The solution is reflected operations:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __add__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot add different currencies")
            return Money(self.amount + other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount + other, self.currency)
        return NotImplemented

    def __radd__(self, other):
        # Reflected addition: other + self
        # Just delegate to __add__ since addition is commutative
        return self.__add__(other)

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

m = Money(10, "USD")
print(m + 5)   # Money(15, 'USD')
print(5 + m)   # Money(15, 'USD') - now works!

The Critical Distinction: `NotImplemented` vs `NotImplementedError`

This is where juniors crash and seniors cooperate:

# WRONG - Don't do this!
def __add__(self, other):
    if isinstance(other, Money):
        return Money(self.amount + other.amount, self.currency)
    raise NotImplementedError("Cannot add Money and " + type(other).__name__)

# RIGHT - Be a good citizen
def __add__(self, other):
    if isinstance(other, Money):
        return Money(self.amount + other.amount, self.currency)
    return NotImplemented

What's the difference?

NotImplementedError: An exception. Your program crashes. You're saying "I can't do this and nobody else can either."
NotImplemented: A sentinel value. You're saying "I don't know how to handle this, but maybe the other object does."

When you return NotImplemented, you're participating in Python's cooperative operator dispatch. You're giving the other object a chance to handle the operation.

class Discount:
    def __init__(self, percent):
        self.percent = percent

    def __radd__(self, other):
        if isinstance(other, Money):
            discount_amount = other.amount * (self.percent / 100)
            return Money(other.amount - discount_amount, other.currency)
        return NotImplemented

    def __repr__(self):
        return f"Discount({self.percent}%)"

m = Money(100, "USD")
d = Discount(10)

# This works because Money.__add__ returns NotImplemented,
# so Python tries Discount.__radd__
result = m + d  # Money(90.0, 'USD')

If Money.__add__ had raised NotImplementedError, this would never work.

The Full Arithmetic Protocol

For a complete numeric type, you need:

__add__, __radd__ - Addition (a + b)
__sub__, __rsub__ - Subtraction (a - b)
__mul__, __rmul__ - Multiplication (a * b)
__truediv__, __rtruediv__ - Division (a / b)
__floordiv__, __rfloordiv__ - Floor division (a // b)
__mod__, __rmod__ - Modulo (a % b)
__pow__, __rpow__ - Exponentiation (a ** b)

And the in-place variants:

__iadd__ - In-place addition (a += b)
__isub__, __imul__, etc.

class Money:
    # ... __init__, __repr__ ...

    def __add__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot add different currencies")
            return Money(self.amount + other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount + other, self.currency)
        return NotImplemented

    def __radd__(self, other):
        return self.__add__(other)

    def __sub__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot subtract different currencies")
            return Money(self.amount - other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount - other, self.currency)
        return NotImplemented

    def __rsub__(self, other):
        # Subtraction is NOT commutative: other - self
        if isinstance(other, (int, float)):
            return Money(other - self.amount, self.currency)
        return NotImplemented

    def __mul__(self, other):
        if isinstance(other, (int, float)):
            return Money(self.amount * other, self.currency)
        return NotImplemented

    def __rmul__(self, other):
        return self.__mul__(other)

The Comparison Shortcut: `@total_ordering`

Now we need comparison operators. All six of them:

__eq__ (equality: ==)
__ne__ (inequality: !=)
__lt__ (less than: <)
__le__ (less than or equal: <=)
__gt__ (greater than: >)
__ge__ (greater than or equal: >=)

That's a lot of boilerplate. Fortunately, Python provides a shortcut:

from functools import total_ordering

@total_ordering
class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __eq__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        return self.amount == other.amount and self.currency == other.currency

    def __lt__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        if self.currency != other.currency:
            raise ValueError("Cannot compare different currencies")
        return self.amount < other.amount

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

# Now ALL of these work:
m1 = Money(10, "USD")
m2 = Money(20, "USD")

print(m1 < m2)   # True
print(m1 <= m2)  # True
print(m1 > m2)   # False
print(m1 >= m2)  # False
print(m1 == m2)  # False
print(m1 != m2)  # True

The @total_ordering decorator automatically generates __le__, __gt__, __ge__, and __ne__ based on your __eq__ and __lt__ implementations.

The Mathematical Inference

How does it work? Mathematics:

a <= b is equivalent to a < b or a == b
a > b is equivalent to not (a <= b)
a >= b is equivalent to not (a < b)
a != b is equivalent to not (a == b)

The decorator generates these derived methods automatically.

The Hashability Warning

If your object is hashable (implements __hash__), you must be careful with equality.

Never allow this if your object is hashable:

m = Money(10, "USD")
print(m == 10)  # Should return False or NotImplemented, NEVER True

Why? Because if Money(10, "USD") == 10 returns True, but hash(Money(10, "USD")) != hash(10), you violate the hashability contract:

If a == b, then hash(a) must equal hash(b)

This is why our __eq__ returns NotImplemented for non-Money objects:

def __eq__(self, other):
    if not isinstance(other, Money):
        return NotImplemented  # Let Python figure it out
    return self.amount == other.amount and self.currency == other.currency

The Sequence Protocol: Free Features Through Convention

Here's something magical. Implement just two methods, and Python gives you iteration, indexing, slicing, and membership testing for free.

class Vector:
    def __init__(self, *components):
        self._components = list(components)

    def __len__(self):
        return len(self._components)

    def __getitem__(self, index):
        return self._components[index]

    def __repr__(self):
        return f"Vector{tuple(self._components)}"

v = Vector(1, 2, 3, 4, 5)

# Indexing - we implemented this explicitly
print(v[0])     # 1
print(v[-1])    # 5

# Slicing - we get this for FREE
print(v[1:3])   # [2, 3]

# Iteration - we get this for FREE
for component in v:
    print(component)  # 1, 2, 3, 4, 5

# Membership - we get this for FREE
print(3 in v)   # True
print(10 in v)  # False

# Length - we implemented this explicitly
print(len(v))   # 5

How is this possible? When you write for x in v:, Python first checks if v has an __iter__ method. If not, it falls back to the sequence protocol:

Call v[0], assign to x, execute loop body
Call v[1], assign to x, execute loop body
Keep incrementing until v[n] raises IndexError
Catch the IndexError and stop iteration

Similarly, when you write 3 in v, Python:

Checks if v has a __contains__ method
If not, iterates through v checking item == 3 for each item

Making It More Pythonic

We can add more magic methods to make our Vector even more powerful:

class Vector:
    def __init__(self, *components):
        self._components = list(components)

    def __len__(self):
        return len(self._components)

    def __getitem__(self, index):
        return self._components[index]

    def __setitem__(self, index, value):
        self._components[index] = value

    def __add__(self, other):
        if isinstance(other, Vector):
            if len(self) != len(other):
                raise ValueError("Vectors must have same length")
            return Vector(*(a + b for a, b in zip(self, other)))
        return NotImplemented

    def __mul__(self, scalar):
        if isinstance(scalar, (int, float)):
            return Vector(*(x * scalar for x in self))
        return NotImplemented

    def __rmul__(self, scalar):
        return self.__mul__(scalar)

    def __repr__(self):
        return f"Vector{tuple(self._components)}"

v1 = Vector(1, 2, 3)
v2 = Vector(4, 5, 6)

print(v1 + v2)    # Vector(5, 7, 9)
print(v1 * 2)     # Vector(2, 4, 6)
print(3 * v1)     # Vector(3, 6, 9)

# We can modify vectors
v1[0] = 10
print(v1)         # Vector(10, 2, 3)

The Iterator Protocol

If you want more control over iteration, you can implement __iter__:

class Vector:
    def __init__(self, *components):
        self._components = list(components)

    def __len__(self):
        return len(self._components)

    def __getitem__(self, index):
        return self._components[index]

    def __iter__(self):
        # Return an iterator object
        return iter(self._components)

    def __repr__(self):
        return f"Vector{tuple(self._components)}"

Now iteration uses __iter__ instead of repeated __getitem__ calls, which can be more efficient for large sequences.

The Missing Link: How `defaultdict` Actually Works

You've probably used collections.defaultdict:

from collections import defaultdict

counts = defaultdict(int)
counts['a'] += 1  # No KeyError!
counts['b'] += 1
print(counts)  # defaultdict(<class 'int'>, {'a': 1, 'b': 1})

But how does it work? Through a special hook called __missing__.

The `missing` Protocol

The __missing__ method is exclusive to dict subclasses. It's called automatically when a key lookup fails:

class DefaultDict(dict):
    def __init__(self, default_factory):
        super().__init__()
        self.default_factory = default_factory

    def __missing__(self, key):
        # This is called when self[key] raises KeyError
        if self.default_factory is None:
            raise KeyError(key)

        # Create the default value
        value = self.default_factory()

        # Store it in the dict
        self[key] = value

        # Return it
        return value

# Now we've built our own defaultdict!
counts = DefaultDict(int)
counts['a'] += 1  # __missing__ creates 0, then +=1 makes it 1
counts['b'] += 1
print(counts)  # {'a': 1, 'b': 1}

The Execution Flow

When you do counts['a'] += 1 and 'a' doesn't exist:

Python calls counts.__getitem__('a')
The inherited dict.__getitem__ looks for 'a', doesn't find it
dict.__getitem__ calls counts.__missing__('a')
__missing__ creates 0, stores it at counts['a'], returns 0
Python increments the returned 0 to get 1
Python calls counts.__setitem__('a', 1)

Critical detail: __missing__ is only called by __getitem__. It's not called by .get():

counts = DefaultDict(int)
print(counts['x'])      # 0 - __missing__ called
print(counts.get('y'))  # None - __missing__ NOT called

A Practical Use Case

Let's build a case-insensitive dictionary:

class CaseInsensitiveDict(dict):
    def __setitem__(self, key, value):
        # Always store with lowercase keys
        super().__setitem__(key.lower(), value)

    def __getitem__(self, key):
        # Always lookup with lowercase keys
        return super().__getitem__(key.lower())

    def __missing__(self, key):
        # Provide a helpful error message
        raise KeyError(f"Key '{key}' not found (case-insensitive)")

config = CaseInsensitiveDict()
config['Server'] = 'localhost'
print(config['SERVER'])  # 'localhost'
print(config['server'])  # 'localhost'
print(config['SeRvEr'])  # 'localhost'

Or a tree structure that auto-creates nested dictionaries:

class TreeDict(dict):
    def __missing__(self, key):
        # Create a new TreeDict for missing keys
        value = TreeDict()
        self[key] = value
        return value

tree = TreeDict()
tree['a']['b']['c'] = 42  # No KeyErrors!
print(tree)  # {'a': {'b': {'c': 42}}}

The Complete Protocol Checklist

Here's your reference for making objects feel native:

Arithmetic Protocol

__add__, __radd__ → +
__sub__, __rsub__ → -
__mul__, __rmul__ → *
__truediv__, __rtruediv__ → /
__floordiv__, __rfloordiv__ → //
__mod__, __rmod__ → %
__pow__, __rpow__ → **

Always return NotImplemented for unsupported types, never raise NotImplementedError.

Comparison Protocol

__eq__ → ==
__ne__ → !=
__lt__ → <
__le__ → <=
__gt__ → >
__ge__ → >=

Use @total_ordering to generate most of these from __eq__ and __lt__.

Sequence Protocol

__len__ → len(obj)
__getitem__ → obj[key]
__setitem__ → obj[key] = value
__delitem__ → del obj[key]
__contains__ → x in obj (optional if __getitem__ + __iter__ exist)
__iter__ → for x in obj (optional, falls back to __getitem__)

Dictionary Protocol

Inherit from dict
__missing__ → handles missing key lookups

Summary:

Today we've learned how to make custom objects integrate seamlessly with Python's syntax:

Key Principles

Return NotImplemented, not NotImplementedError → Enables cooperative dispatch
Implement reflected operations (__radd__, etc.) → Supports 5 + obj in addition to obj + 5
Use @total_ordering → Generate six comparison methods from two
Implement __len__ + __getitem__ → Get iteration, slicing, and membership for free
Use __missing__ in dict subclasses → Intercept missing key lookups

The Professional Template

from functools import total_ordering

@total_ordering
class Money:
    __slots__ = ['_amount', '_currency']

    def __init__(self, amount, currency):
        object.__setattr__(self, '_amount', amount)
        object.__setattr__(self, '_currency', currency)

    # Representation
    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

    # Properties
    @property
    def amount(self):
        return self._amount

    @property
    def currency(self):
        return self._currency

    # Arithmetic
    def __add__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot add different currencies")
            return Money(self.amount + other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount + other, self.currency)
        return NotImplemented

    def __radd__(self, other):
        return self.__add__(other)

    def __sub__(self, other):
        if isinstance(other, Money):
            if self.currency != other.currency:
                raise ValueError("Cannot subtract different currencies")
            return Money(self.amount - other.amount, self.currency)
        if isinstance(other, (int, float)):
            return Money(self.amount - other, self.currency)
        return NotImplemented

    def __mul__(self, other):
        if isinstance(other, (int, float)):
            return Money(self.amount * other, self.currency)
        return NotImplemented

    def __rmul__(self, other):
        return self.__mul__(other)

    # Comparison (total_ordering generates <=, >, >=, !=)
    def __eq__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        return self.amount == other.amount and self.currency == other.currency

    def __lt__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        if self.currency != other.currency:
            raise ValueError("Cannot compare different currencies")
        return self.amount < other.amount

    # Hashability
    def __hash__(self):
        return hash((self.amount, self.currency))

Python Internals: init is Not the Constructor

aykhlf yassir — Tue, 03 Feb 2026 19:23:58 +0000

Mastering __new__, __repr__, and __hash__

The Constructor Myth

Pop quiz: What method creates a Python object?

If you answered __init__, you're in good company, and you're wrong!

class Point:
    def __init__(self, x, y):
        print(f"__init__ called with {self}")
        self.x = x
        self.y = y

p = Point(3, 4)
# Output: __init__ called with <__main__.Point object at 0x7f8b4c>

Notice that inside __init__, we already have self. The object already exists. So what actually created it?

The answer is __new__, a method so fundamental that Python calls it automatically, and most developers never even know it exists.

The Truth About Object Creation

Here's what actually happens when you call Point(3, 4):

class Point:
    def __new__(cls, x, y):
        print(f"__new__ called with class {cls}")
        instance = super().__new__(cls)
        print(f"__new__ created {instance}")
        return instance

    def __init__(self, x, y):
        print(f"__init__ called with {self}")
        self.x = x
        self.y = y

p = Point(3, 4)
# Output:
# __new__ called with class <class '__main__.Point'>
# __new__ created <__main__.Point object at 0x7f8b4c>
# __init__ called with <__main__.Point object at 0x7f8b4c>

The execution flow is:

__new__(cls, ...) - The Architect
- Allocates memory for a new instance
- Returns the newly created object
- Receives the class as first parameter, not an instance
__init__(self, ...) - The Interior Decorator
- Receives the instance created by __new__
- Populates it with data
- Returns None (always!)

The Mental Model:

Point(3, 4)
    ↓
__new__(Point, 3, 4) → creates empty object → instance
    ↓
__init__(instance, 3, 4) → populates instance.x, instance.y
    ↓
return instance

99% of the time, you don't need to touch __new__. Python's default implementation (inherited from object) handles memory allocation perfectly. But there's one critical use case where __new__ is not just useful, it's essential.

Deep Dive: The Singleton Pattern

Imagine you're building a database connection pool, a configuration manager, or a logger. You want exactly one instance of the class to exist, no matter how many times someone calls the constructor.

# What we want:
db1 = Database()
db2 = Database()
print(db1 is db2)  # Should be True!

Can we do this with __init__? Let's try:

class Database:
    _instance = None

    def __init__(self):
        if Database._instance is not None:
            # Too late! Memory is already allocated
            # We can't "un-create" this object
            pass
        Database._instance = self

db1 = Database()
db2 = Database()
print(db1 is db2)  # False - we created two objects!

The problem: by the time __init__ runs, __new__ has already allocated memory for a new object. We can't prevent the creation—only configure what's already been created.

The Solution: Intercept at `new`

class Database:
    _instance = None

    def __new__(cls):
        if cls._instance is None:
            print("Creating the one true Database instance...")
            cls._instance = super().__new__(cls)
        else:
            print("Returning existing instance...")
        return cls._instance

    def __init__(self):
        print(f"__init__ called on {id(self)}")

db1 = Database()
# Output:
# Creating the one true Database instance...
# __init__ called on 140234567890

db2 = Database()
# Output:
# Returning existing instance...
# __init__ called on 140234567890

print(db1 is db2)  # True!
print(id(db1), id(db2))  # Same memory address

What's happening:

First call: _instance is None, so we call super().__new__(cls) to actually allocate memory
We cache this instance in _instance
Second call: _instance exists, so we return the cached object
__init__ still runs every time (be careful with this!)

The Critical Detail: `super().new(cls)`

This line is calling object.__new__(cls), the base implementation that actually talks to Python's memory allocator. You're delegating the "real" work of memory allocation to Python's core object class.

Do NOT do this:

def __new__(cls):
    if cls._instance is None:
        cls._instance = cls()  # RECURSION ERROR!
    return cls._instance

Calling cls() inside __new__ calls __new__ again, which calls __new__ again... infinite recursion.

Singleton Best Practice

If __init__ shouldn't run multiple times, use a flag:

class Database:
    _instance = None
    _initialized = False

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

    def __init__(self, connection_string="localhost"):
        if not Database._initialized:
            self.connection_string = connection_string
            Database._initialized = True
            print(f"Connected to {connection_string}")

db1 = Database("prod-server")  # Connected to prod-server
db2 = Database("dev-server")   # (no output - already initialized)
print(db1.connection_string)   # prod-server

The Representation Layer: `str` vs `repr`

You've built a beautiful class. Now it looks like this in the debugger:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

m = Money(10, "USD")
print(m)  # <__main__.Money object at 0x7f8b4c>

Useless. Let's fix it.

The Two Faces of Representation

Python has two methods for converting objects to strings:

__str__ - The User-Friendly Version

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __str__(self):
        return f"${self.amount} {self.currency}"

m = Money(10, "USD")
print(m)  # $10 USD
print(str(m))  # $10 USD

__repr__ - The Developer Version

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"Money({self.amount}, {self.currency})"

m = Money(10, "USD")
print(repr(m))  # Money(10, USD)
print([m])  # [Money(10, USD)] - repr is used in containers!

The Golden Rule of `repr`

The output should be valid Python code that recreates the object.

This is often stated as: eval(repr(obj)) == obj

m = Money(10, "USD")
code = repr(m)  # "Money(10, USD)"
m2 = eval(code)  # Recreate the object!
print(m2.amount)  # 10

Wait... did that actually work? Let's test it:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"Money({self.amount}, {self.currency})"

m = Money(10, "USD")
print(repr(m))  # Money(10, USD)
eval(repr(m))  # NameError: name 'USD' is not defined

The problem: USD without quotes isn't a string—it's treated as a variable name!

The `!r` Trick

Python's f-strings have a special formatter that automatically calls repr() on values:

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

m = Money(10, "USD")
print(repr(m))  # Money(10, 'USD') - notice the quotes!
m2 = eval(repr(m))  # Works perfectly!

The !r format specifier calls repr() on each value, which for strings adds the quotes. This ensures the output is valid Python syntax.

Pro comparison:

amount = 10
currency = "USD"

# Without !r
print(f"Money({amount}, {currency})")  # Money(10, USD)

# With !r
print(f"Money({amount!r}, {currency!r})")  # Money(10, 'USD')

When to Use Which

__repr__: Always implement this. It's used by debuggers, logs, and the interactive interpreter. Make it unambiguous.
__str__: Optional. Only implement if you need a user-friendly format. If not defined, Python falls back to __repr__.

class Money:
    def __init__(self, amount, currency):
        self.amount = amount
        self.currency = currency

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

    def __str__(self):
        symbols = {"USD": "$", "EUR": "€", "GBP": "£"}
        symbol = symbols.get(self.currency, self.currency)
        return f"{symbol}{self.amount}"

m = Money(10, "USD")
print(str(m))   # $10 (user-friendly)
print(repr(m))  # Money(10, 'USD') (code-like)
print(m)        # $10 (print uses str)
print([m])      # [Money(10, 'USD')] (containers use repr)

The Hashability Contract: Making Objects Dictionary Keys

You've probably used strings and tuples as dictionary keys:

cache = {}
cache["user:123"] = {"name": "Alice"}  # String key - works
cache[(1, 2)] = "point"  # Tuple key - works

But try this:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

p = Point(1, 2)
cache = {}
cache[p] = "point"  # TypeError: unhashable type: 'Point'

Why can't we use our custom object as a key? Because it's not hashable.

What Does Hashable Mean?

To be used as a dictionary key or stored in a set, an object must:

Have a __hash__ method that returns an integer
Have an __eq__ method to check equality
Follow the hashability contract

The Hashability Contract

Rule 1: Equal objects must have equal hashes

If a == b, then hash(a) MUST equal hash(b)

Rule 2: The hash must never change

Once created, an object's hash must remain constant for its entire lifetime. This is why lists aren't hashable—you can modify them!

# This is why lists fail:
lst = [1, 2, 3]
hash(lst)  # TypeError: unhashable type: 'list'

# But tuples work:
tpl = (1, 2, 3)
hash(tpl)  # 529344067295497451

Implementing Hashability

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __eq__(self, other):
        if not isinstance(other, Point):
            return NotImplemented
        return self.x == other.x and self.y == other.y

    def __hash__(self):
        return hash((self.x, self.y))

p1 = Point(1, 2)
p2 = Point(1, 2)
p3 = Point(3, 4)

print(p1 == p2)  # True
print(hash(p1) == hash(p2))  # True - contract satisfied!

cache = {p1: "origin"}
print(cache[p2])  # "origin" - found it using p2!

Why Delegate to a Tuple?

The line return hash((self.x, self.y)) is the idiomatic way to hash objects. Here's why:

Tuples are immutable - Their hash is guaranteed stable
Python's tuple hash is well-designed - It combines element hashes efficiently
It's simple - You don't have to write your own hash combining logic

Under the hood, Python's tuple hash does something like:

# Simplified version of what Python does
def hash_tuple(items):
    result = 0x345678
    for item in items:
        result = (1000003 * result) ^ hash(item)
    return result

But you don't need to know that—just pack your state into a tuple and let Python handle it.

The `NotImplemented` Pattern

Notice this line in __eq__:

if not isinstance(other, Point):
    return NotImplemented

Don't return False here! Returning NotImplemented tells Python "I don't know how to compare with this type—ask the other object."

class Point:
    def __eq__(self, other):
        if not isinstance(other, Point):
            return NotImplemented
        return self.x == other.x and self.y == other.y

p = Point(1, 2)
print(p == 5)  # False (Python tries both p.__eq__(5) and (5).__eq__(p))

If you returned False instead, you'd be claiming "a Point is definitely not equal to an integer," which might not be true if someone subclasses Point and adds custom comparison logic.

The Immutability Trap

Remember: hashable objects should be immutable. If you allow modification, weird things happen:

class MutablePoint:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __eq__(self, other):
        if not isinstance(other, MutablePoint):
            return NotImplemented
        return self.x == other.x and self.y == other.y

    def __hash__(self):
        return hash((self.x, self.y))

p = MutablePoint(1, 2)
cache = {p: "original"}

print(cache[p])  # "original" - works

# Mutate the object
p.x = 99

# Now the hash changed!
print(cache[p])  # KeyError: MutablePoint object not found

The object is now "lost" in the dictionary because its hash changed. The dictionary is looking in the wrong bucket!

Best practice: If you implement __hash__, make your object immutable using __slots__ and properties:

class ImmutablePoint:
    __slots__ = ['_x', '_y']

    def __init__(self, x, y):
        object.__setattr__(self, '_x', x)
        object.__setattr__(self, '_y', y)

    @property
    def x(self):
        return self._x

    @property
    def y(self):
        return self._y

    def __setattr__(self, name, value):
        raise AttributeError("ImmutablePoint is immutable")

    def __eq__(self, other):
        if not isinstance(other, ImmutablePoint):
            return NotImplemented
        return self.x == other.x and self.y == other.y

    def __hash__(self):
        return hash((self.x, self.y))

    def __repr__(self):
        return f"ImmutablePoint({self.x!r}, {self.y!r})"

Summary: The Professional Object Checklist

Today we've learned the lifecycle methods that make Python objects behave like first-class types:

Creation & Representation

__new__(cls, ...) creates the object; __init__(self, ...) configures it
Use __new__ for Singletons and other creation-control patterns
__repr__ is for developers (make it code-like with !r)
__str__ is for users (optional, human-friendly)

The Hashability Contract

__eq__ defines equality (return NotImplemented for unknown types)
__hash__ enables dictionary/set usage (delegate to tuple)
Rule: If a == b, then hash(a) == hash(b)
Immutability: The hash must never change

The Professional Class Template

class Money:
    __slots__ = ['_amount', '_currency']

    def __init__(self, amount, currency):
        object.__setattr__(self, '_amount', amount)
        object.__setattr__(self, '_currency', currency)

    @property
    def amount(self):
        return self._amount

    @property
    def currency(self):
        return self._currency

    def __setattr__(self, name, value):
        raise AttributeError("Money is immutable")

    def __repr__(self):
        return f"Money({self.amount!r}, {self.currency!r})"

    def __str__(self):
        return f"${self.amount} {self.currency}"

    def __eq__(self, other):
        if not isinstance(other, Money):
            return NotImplemented
        return self.amount == other.amount and self.currency == other.currency

    def __hash__(self):
        return hash((self.amount, self.currency))

This class is memory-efficient (__slots__), immutable (read-only properties), debuggable (__repr__), user-friendly (__str__), and can be used in sets and dicts (__eq__ + __hash__).

Python Internals: The Object Memory Model

aykhlf yassir — Tue, 03 Feb 2026 18:28:55 +0000

Why your Python objects are just dictionaries and when that becomes a problem

Identity vs. Equality:

First, let's clarify what is actually does. Most Python developers learn that == checks equality and is checks identity, but what does "identity" really mean?

x = [1, 2, 3]
y = [1, 2, 3]

print(x == y)  # True - same contents
print(x is y)  # False - different objects in memory
print(id(x), id(y))  # Different memory addresses

The is operator checks if two variables point to the exact same object in memory the same address.

The Integer Interning Optimization

Python uses an optimization called interning, it pre-allocates objects at startup and reuses them throughout your program's lifetime.

a = 256
b = 256
print(id(a) == id(b))  # True - same memory address

Every time you use the number 42 in your code, you're pointing to the same integer object. This saves memory and improves performance for commonly-used values.

The takeaway: Python manages memory in ways you don't see until you look. And this is just the beginning.

The Core:

Here's the real secret: a standard Python object is essentially a wrapper around a hash map. Let's prove it.

class Player:
    server = "US-East"  # Class variable

    def __init__(self, name):
        self.name = name

p = Player("Alice")
print(p.__dict__)  # {'name': 'Alice'}

Every instance has a __dict__ attribute, that is a dictionary that holds all its instance variables. When you do p.name, Python is really doing p.__dict__['name'].

The Shadowing Experiment

This dictionary model creates interesting behavior with class variables:

class Player:
    server = "US-East"

    def __init__(self, name):
        self.name = name

p1 = Player("Alice")
p2 = Player("Bob")

print(p1.server)  # "US-East" - found on the class
print(p1.__dict__)  # {'name': 'Alice'} - no 'server' key!

# Now shadow the class variable
p1.server = "EU-West"
print(p1.__dict__)  # {'name': 'Alice', 'server': 'EU-West'}
print(p2.server)  # Still "US-East"
print(Player.server)  # Still "US-East"

What happened? When you read p1.server, Python checks:

Instance dictionary (p1.__dict__) → not found
Class dictionary (Player.__dict__) → found!

But when you write p1.server = "EU-West", Python creates a new entry in the instance dictionary, shadowing the class variable for that specific instance.

The Lookup Chain

Python's attribute lookup follows a specific chain:

Instance → Class → Parent Classes (MRO)

This is why you can have class variables that all instances share by default, but can be overridden per-instance. The dictionary model makes this incredibly flexible.

The Cost of Flexibility

This flexibility comes at a steep price: memory.

A Python dictionary is a sparse, resizable hash table. It needs to handle collisions, maintain load factors, and allow dynamic resizing. For a simple Point(x, y) object, this is massive overkill.

Let's measure it:

import sys

class DictPoint:
    def __init__(self, x, y):
        self.x = x
        self.y = y

class SlotPoint:
    __slots__ = ['x', 'y']

    def __init__(self, x, y):
        self.x = x
        self.y = y

p1 = DictPoint(1, 2)
p2 = SlotPoint(1, 2)

print(f"DictPoint size: {sys.getsizeof(p1) + sys.getsizeof(p1.__dict__)} bytes")
print(f"SlotPoint size: {sys.getsizeof(p2)} bytes")

# Try to access __dict__ on slotted class
try:
    print(p2.__dict__)
except AttributeError as e:
    print(f"SlotPoint error: {e}")

On my machine, DictPoint consumes roughly 344 bytes while SlotPoint uses only 48 bytes. That's a 7x difference for storing two integers!

Now imagine creating a million of these objects. The dictionary-based approach would consume an additional ~100MB of RAM just for the dictionary overhead.

What Actually Changed?

At the C level, here's what's happening:

Standard Python Object (Dictionary-Based):

PyObject
  ↓
ob_refcnt (reference count)
ob_type (pointer to type)
__dict__ → PyDictObject (hash table)
            ↓
            Hash map with keys 'x', 'y'

Slotted Python Object:

PyObject
  ↓
ob_refcnt
ob_type
x (stored at fixed offset +0)
y (stored at fixed offset +8)

With __slots__, you're essentially telling Python: "I know exactly what attributes this object will have. Don't give me a dictionary just allocate fixed memory offsets like a C struct."

The interpreter can now access obj.x by going directly by offsetting from the object's base address. No hash lookup, no dictionary traversal just pointer arithmetic.

The Trade-Off

But this optimization comes with restrictions:

class SlotPoint:
    __slots__ = ['x', 'y']

    def __init__(self, x, y):
        self.x = x
        self.y = y

p = SlotPoint(1, 2)
p.z = 3  # AttributeError: 'SlotPoint' object has no attribute 'z'

You cannot dynamically add attributes to slotted objects. The attributes are fixed at class definition time. This is the price of performance: you trade flexibility for efficiency.

When to Use `slots`

Use __slots__ when:

You're creating many instances (thousands+) of the same class
The attributes are known and fixed
Memory is a concern (data processing, game engines, embedded systems)

Don't use __slots__ when:

You need dynamic attributes
You're creating only a few instances
The class needs to be subclassed with additional attributes (requires careful design)

Summary:

We uncovered three fundamental truths about Python's object model:

is checks memory address; == checks value → Python interns small integers for performance
Objects are usually dictionaries → The __dict__ model provides incredible flexibility at a memory cost
__slots__ turns objects into C structs → Fixed attributes, direct memory access, 60-70% memory savings

The dictionary-based model is what makes Python so dynamic and easy to use. You can add attributes on the fly, monkey-patch classes, and inspect objects at runtime. But when you're building high-performance systems or working with large datasets, knowing how to bypass this flexibility with __slots__ can be the difference between a program that runs and one that crashes.

DEV Community: aykhlf yassir

URL to Pixel: How you got Rickrolled

Layer ① — Client

Phase 1 — Input Processing

1.1 Event dispatch (UI thread)

1.2 URI structural parse (GURL)

1.3 Search vs. navigate decision (Omnibox)

Phase 2 — HSTS Enforcement & Scheme Upgrade

2.1 Preload list lookup (HSTS)

2.2 Scheme rewrite (307 Internal)

2.3 MITM downgrade prevention

Phase 3 — Inter-Process Communication (IPC)

3.1 Navigation request serialisation (Mojo IPC)

3.2 Network Service sandbox

Phase 4 — DNS Resolution — Cache Traversal

4.1 Browser DNS cache (Tier 1)

4.2 OS stub resolver — getaddrinfo (Tier 2)

4.3 Hosts file override (Tier 3)

Phase 5 — DNS Resolution — Recursive Hierarchy

5.1 DNS query datagram (UDP)

5.2 Root → TLD → Authoritative traversal (Recursive)

5.3 DNSSEC validation (optional)

5.4 A/AAAA record returned

Layer ② — Network Transit

Phase 6 — Socket Allocation & Transport Handshake

6.1 QUIC 0-RTT or 1-RTT handshake

6.2 TCP three-way handshake (fallback)

6.3 TLS 1.3 handshake (TCP path)

6.4 Cipher negotiation & symmetric key (AES-NI)

Phase 7 — HTTP Request Construction & Transmission

7.1 Header compression — HPACK/QPACK

7.2 HEADERS frame structure

7.3 TLS record layer encryption

Phase 8 — Network Transit & BGP Routing

8.1 Autonomous system routing (BGP)

8.2 Google Anycast ingress

8.3 Maglev load balancer

Layer ③ — Server

Phase 9 — NIC Ingress & Kernel DMA

9.1 Optical → electrical conversion (PHY)

9.2 PCIe DMA → sk_buff

9.3 IRQ → NET_RX_SOFTIRQ → NAPI poll

Phase 10 — Kernel TCP State Machine & Accept Queue

10.1 IP layer demultiplexing

10.2 SYN queue → Transmission Control Block (TCB)

10.3 ACK → Accept queue → accept4()

Phase 11 — User-Space TLS Termination

11.1 BoringSSL handshake

11.2 SSL_read() + AES-NI decryption

11.3 Private key security (FIDO/Titan)

Phase 12 — HTTP/2 Parsing, Header Decompression & RPC Dispatch

12.1 HPACK decompression & header extraction

12.2 Virtual host & path routing

12.3 Protocol translation → protobuf / gRPC

Phase 13 — Backend Microservice Orchestration — Fan-out RPCs

13.1 Identity / auth — Gaia service

13.2 Video metadata — Cloud Spanner / Vitess

13.3 Recommendations — TensorFlow / TPU

13.4 Monetisation — ad pre-roll check

13.5 DASH manifest assembly

Phase 14 — Response Serialisation & Egress

14.1 Brotli compression

14.2 HTTP/2 DATA frame chunking

14.3 SSL_write() → sendmsg() → NIC TX ring

Layer ④ — Rendering & Media

Phase 15 — Renderer Process Allocation & Stream Handoff

15.1 MIME sniff & response validation

15.2 Site Isolation & renderer selection

15.3 Byte stream handoff to renderer (Mojo data pipe)

Phase 16 — DOM Construction — Tokenisation & Tree Building

16.1 Byte stream → UTF-8 character stream

16.2 Tokenisation → StartTag / EndTag / Text

16.3 Token stream → DOM tree (tree builder)

Phase 17 — CSSOM Construction

17.1 CSS parsing → StyleSheetContents

17.2 Rule matching & specificity → ComputedStyle

Phase 18 — JavaScript Compilation & Execution (V8)

18.1 Source → AST (Parser)

18.2 AST → bytecode (Ignition interpreter)

18.3 Hot path → native code (TurboFan JIT)

4.2 OS stub resolver — `getaddrinfo` (Tier 2)

10.3 ACK → Accept queue → `accept4()`

11.2 `SSL_read()` + AES-NI decryption

14.3 `SSL_write()` → `sendmsg()` → NIC TX ring

21.3 Segment fetch → `SourceBuffer.appendBuffer()`

1. Core Mechanics: What `yield from` Actually Does

Without `yield from`: Manual Proxying Hell

With `yield from`: Automatic Transparent Proxying

2. Architectural Patterns: When to Use `yield from`

4.1. Memory Comparison: `sys.getsizeof` vs. `tracemalloc`

5. Historical Context: The Road to `async/await`

Conclusion: When to Reach for `yield from`

1. The Basics: What `yield` Does to a Function

`yield` as an Expression