DEV Community: prashant chouksey

Python Generators Part 2: How They Actually Work (The Magic Revealed)

prashant chouksey — Fri, 13 Feb 2026 13:25:15 +0000

Understanding the pause button: frames, state, and bidirectional communication

In Part 1, you learned:

✅ Generators produce values one at a time (lazy)
✅ The yield keyword pauses the function
✅ Generators use constant memory (112 bytes)
✅ Use for loops or next() to get values

If you haven't reviewed Part 1 yet, please refer to it here.

Now let's answer the big question: How does Python remember where it paused?

The Big Question: Where is the State Saved?

When a generator pauses at yield, it needs to remember:

📍 Where it was in the code (which line?)
🧮 All local variables (their current values)
📊 The call stack (in case there were nested function calls)

Where does Python store all this?

Answer: In a special object called a Generator Frame.

What Happens When You Call a Generator Function

Let's trace through this step-by-step:

def counter(start, end):
    current = start
    while current < end:
        value = current ** 2
        yield value
        current += 1

# Step 1: Call the function
gen = counter(0, 3)

What just happened?

❌ WRONG: The function ran and calculated squares

✅ CORRECT: Python created a "Generator Object" and did NOTHING else

The Generator Object

┌─────────────────────────────────────────┐
│    Generator Object (112 bytes)         │
├─────────────────────────────────────────┤
│  gi_frame: <Frame object>               │  ← WHERE STATE IS SAVED
│  gi_code: <counter function bytecode>   │  ← THE CODE TO RUN
│  gi_running: False                       │  ← IS IT RUNNING NOW?
│  gi_yieldfrom: None                      │  ← For yield from
└─────────────────────────────────────────┘

The most important part: gi_frame - this is where all the magic happens!

Inside the Frame: Where Variables Live

The frame (gi_frame) contains:

┌─────────────────────────────────────────┐
│           Generator Frame                │
├─────────────────────────────────────────┤
│  f_locals: {}                            │  ← Local variables
│  f_lasti: 0                              │  ← Instruction pointer
│  f_valuestack: []                        │  ← Expression stack
│  f_back: None                            │  ← Caller's frame
└─────────────────────────────────────────┘

Let's see what happens when we call next():

gen = counter(0, 3)

# FIRST next()
print(next(gen))  # 0

Frame BEFORE first next():

f_locals: {}  (empty - hasn't run yet!)
f_lasti: 0    (start of function)

Frame AFTER first next() (paused at yield):

f_locals: {
    'start': 0,
    'end': 3,
    'current': 1,     ← Updated!
    'value': 0
}
f_lasti: 58    ← Points to instruction AFTER yield

Frame AFTER second next():

f_locals: {
    'start': 0,
    'end': 3,
    'current': 2,     ← Updated again!
    'value': 1
}
f_lasti: 58    ← Same yield point

See the magic? Python saves everything and resumes exactly where it left off!

The Execution Lifecycle (States)

A generator goes through different states:

┌──────────────┐
│ GEN_CREATED  │  ← Just created, hasn't run yet
└──────────────┘
       │
       │ next() or send()
       ▼
┌──────────────┐
│ GEN_RUNNING  │  ← Currently executing code
└──────────────┘
       │
       │ hit yield
       ▼
┌──────────────┐
│GEN_SUSPENDED │  ← ⏸️ Paused, state saved
└──────────────┘
       │
       ├──── next() ────▶ (loop back to GEN_RUNNING)
       │
       └──── StopIteration or close() ────▶ GEN_CLOSED

The Four Powerful Methods

Generators aren't just for reading values! They support four methods for advanced control:

1. `next()` - Get Next Value ▶️

You already know this one!

def simple_gen():
    yield 1
    yield 2
    yield 3

gen = simple_gen()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3
print(next(gen))  # StopIteration Error!

2. `send(value)` - Send Data INTO the Generator 📨

This is where it gets interesting! You can send values into a generator while it's running.

def echo():
    while True:
        # The yield can RECEIVE a value!
        received = yield
        print(f"You sent: {received}")

gen = echo()
next(gen)  # ⚠️ MUST prime first! (advance to first yield)

gen.send("Hello")   # You sent: Hello
gen.send("World")   # You sent: World
gen.send(42)        # You sent: 42

How does send() work?

Generator code:    received = yield
                              ↑
                              │
gen.send("Hello") ────────────┘

The yield expression RECEIVES the value "Hello"
Then: received = "Hello"

Real Example: Running Average Calculator

def running_average():
    total = 0
    count = 0
    average = None

    while True:
        # Yield the current average, receive the next number
        number = yield average

        if number is None:
            break

        total += number
        count += 1
        average = total / count

# Usage
avg = running_average()
next(avg)  # Prime (get to first yield)

print(avg.send(10))  # 10.0  (running avg: 10)
print(avg.send(20))  # 15.0  (running avg: 15)
print(avg.send(30))  # 20.0  (running avg: 20)
print(avg.send(40))  # 25.0  (running avg: 25)

Step-by-step flow:

1. next(avg)
   ├─ Start function
   ├─ Set total=0, count=0, average=None
   ├─ Hit: number = yield average
   ├─ Yield None (first average)
   └─ ⏸️ PAUSE

2. avg.send(10)
   ├─ Resume: number = yield average
   ├─ number receives 10
   ├─ total = 10, count = 1
   ├─ average = 10.0
   ├─ Loop back to: number = yield average
   ├─ Yield 10.0
   └─ ⏸️ PAUSE

3. avg.send(20)
   ├─ Resume: number = yield average
   ├─ number receives 20
   ├─ total = 30, count = 2
   ├─ average = 15.0
   ├─ Yield 15.0
   └─ ⏸️ PAUSE

3. `close()` - Stop the Generator and Cleanup 🛑

When you're done with a generator, you should close it to free resources:

def file_reader(filename):
    print(f"Opening {filename}")
    f = open(filename, 'r')

    try:
        for line in f:
            yield line.strip()
    except GeneratorExit:
        # This runs when close() is called!
        print("Closing file...")
        f.close()
        raise  # Must re-raise or return

# Usage
reader = file_reader('data.txt')
print(next(reader))  # First line
print(next(reader))  # Second line

reader.close()  # Triggers cleanup!
# Output: Closing file...

What close() does:

Raises GeneratorExit exception inside the generator
Generator's except GeneratorExit block runs
Cleanup code executes
Generator marked as GEN_CLOSED

4. `throw(exception)` - Inject Errors 💥

You can raise an exception inside the generator from outside:

def error_handler():
    while True:
        try:
            value = yield
            print(f"Processing: {value}")
        except ValueError as e:
            print(f"Caught error: {e}")
            # Generator continues running!

gen = error_handler()
next(gen)  # Prime

gen.send(10)  # Processing: 10
gen.send(20)  # Processing: 20

# Inject an error from outside!
gen.throw(ValueError, "Bad value!")
# Output: Caught error: Bad value!

gen.send(30)  # Processing: 30 (still works!)

Use case: Error handling in data pipelines

def data_ ():
    while True:
        try:
            data = yield
            if data < 0:
                raise ValueError("Negative values not allowed")
            result = process(data)
            print(f"Result: {result}")
        except ValueError as e:
            print(f"Skipping bad data: {e}")

processor = data_processor()
next(processor)

processor.send(10)   # Result: ...
processor.send(-5)   # Skipping bad data: Negative values not allowed
processor.send(20)   # Result: ... (continues!)

Method Comparison Table

Method	What It Does	Returns	Use When
`next(gen)`	Get next value	Next yielded value	Basic iteration
`gen.send(val)`	Send value into generator	Next yielded value	Two-way communication
`gen.throw(exc)`	Raise exception inside	Next yield (if handled)	Error injection
`gen.close()`	Stop generator	None	Cleanup resources

Advanced: `yield from` - Generator Delegation

When you have generators calling other generators, yield from makes it seamless:

def generator1():
    yield 1
    yield 2

def generator2():
    yield 3
    yield 4

# Without yield from
def combined_old():
    for value in generator1():
        yield value
    for value in generator2():
        yield value

# With yield from (cleaner!)
def combined_new():
    yield from generator1()
    yield from generator2()

print(list(combined_new()))  # [1, 2, 3, 4]

Why is yield from better?

It transparently forwards all operations:

✅ next() calls
✅ send() values
✅ throw() exceptions
✅ close() signals

Practical Example: Multi-File Log Reader

def read_file(filename):
    """Read one file"""
    print(f"Opening {filename}")
    with open(filename) as f:
        for line in f:
            yield line.strip()
    print(f"Closed {filename}")

def read_multiple_files(filenames):
    """Read multiple files sequentially"""
    for filename in filenames:
        yield from read_file(filename)

# Usage
files = ['log1.txt', 'log2.txt', 'log3.txt']
for line in read_multiple_files(files):
    if 'ERROR' in line:
        print(line)

Output:

Opening log1.txt
... errors from log1 ...
Closed log1.txt
Opening log2.txt
... errors from log2 ...
Closed log2.txt
Opening log3.txt
... errors from log3 ...
Closed log3.txt

Understanding the Pause and Resume

Let's visualize exactly what happens:

def demo():
    print("A")
    x = yield 1
    print(f"B, x={x}")
    y = yield 2
    print(f"C, y={y}")

gen = demo()

Execution trace:

Call: gen = demo()
  ➜ Creates generator object
  ➜ NO CODE RUNS
  ➜ State: GEN_CREATED

Call: next(gen)
  ➜ Start execution
  ➜ Execute: print("A")          [Output: "A"]
  ➜ Execute: yield 1
  ➜ Return: 1
  ➜ Save state (x not assigned yet!)
  ➜ State: GEN_SUSPENDED
  ➜ Returns: 1

Call: gen.send(100)
  ➜ Resume execution
  ➜ x = 100 (yield receives sent value)
  ➜ Execute: print(f"B, x={x}")  [Output: "B, x=100"]
  ➜ Execute: yield 2
  ➜ Return: 2
  ➜ Save state (y not assigned yet!)
  ➜ State: GEN_SUSPENDED
  ➜ Returns: 2

Call: gen.send(200)
  ➜ Resume execution
  ➜ y = 200
  ➜ Execute: print(f"C, y={y}")  [Output: "C, y=200"]
  ➜ End of function
  ➜ Raise: StopIteration
  ➜ State: GEN_CLOSED

Summary: What You Learned

🎯 Key Concepts:

Generator state is saved in a "frame" (gi_frame)
Frames store local variables and instruction pointer
send() allows bidirectional communication
close() triggers cleanup with GeneratorExit
throw() injects exceptions for error handling
yield from delegates to sub-generators

🔑 Remember:

Generators go through states: CREATED → RUNNING → SUSPENDED → CLOSED
The frame remembers everything between yields
You can send data into generators, not just get data out

Happy coding! 🎉

Python Generators Deep Dive Part 1: Lazy Evaluation & Memory Optimization🚀

prashant chouksey — Fri, 13 Feb 2026 13:11:57 +0000

Python Generators Deep Dive Part 1: Lazy Evaluation & Memory Optimization🚀

The Problem: Memory Bloat in Data Processing

You've hit this before: processing a large dataset crashes your application with MemoryError. The culprit? Loading everything into memory at once.

# Processing a 50GB log file
def analyze_logs(filename):
    with open(filename) as f:
        lines = f.readlines()  # Loads entire 50GB into RAM

    return [line for line in lines if 'ERROR' in line]

# Result: MemoryError (or system freeze)

Root cause: Eager evaluation - computing all values before you need them.

Solution: Lazy evaluation with generators.

What Are Generators?

Generators are iterators that produce values on-demand using lazy evaluation. Instead of computing all values upfront, they:

Compute one value at a time
Pause execution after yielding
Preserve local state between calls
Resume from the exact pause point

Memory footprint: Constant O(1), regardless of data size.

Eager vs Lazy Evaluation

Eager (Lists)

def squares(n):
    return [i**2 for i in range(n)]

data = squares(1_000_000)
# - Computes all 1M values immediately
# - Stores all 1M values in memory
# - Memory: ~8.4 MB
# - Time to first value: Seconds

Lazy (Generators)

def squares(n):
    for i in range(n):
        yield i**2

data = squares(1_000_000)
# - Computes values on next() call
# - Stores only current state
# - Memory: 112 bytes
# - Time to first value: Microseconds

Key difference: yield instead of return

The `yield` Keyword Explained

yield transforms a function into a generator factory. When called, it returns a generator object without executing the function body.

def countdown(n):
    while n > 0:
        yield n  # Pause here, return n, save state
        n -= 1

gen = countdown(3)  # No execution yet
type(gen)  # <class 'generator'>

next(gen)  # 3 - Executes until first yield, then pauses
next(gen)  # 2 - Resumes, executes until next yield
next(gen)  # 1 - Resumes again
next(gen)  # StopIteration - No more yields

Execution model:

Call countdown(3):
  → Creates generator object
  → Saves reference to function code
  → Does NOT execute function body

Call next(gen):
  → Starts/resumes execution
  → Runs until yield statement
  → Saves execution state (locals, instruction pointer)
  → Returns yielded value
  → Pauses (GEN_SUSPENDED state)

Call next(gen) again:
  → Resumes from saved instruction pointer
  → Continues execution with preserved locals
  → Repeat process

Generator State Management

When a generator pauses, Python stores:

def example():
    x = 0      # Stored in frame locals
    while x < 3:
        y = x ** 2
        yield y  # Pause point
        x += 1

gen = example()
next(gen)  # Returns 0

# Internal state at this point:
# {
#   'x': 1,              # Updated during execution
#   'y': 0,              # Last computed value
#   'instruction': 42    # Bytecode offset (after yield)
# }

Memory breakdown:

Generator object: ~112 bytes
Local variables: Stored in frame (f_locals)
Instruction pointer: Stored in frame (f_lasti)
No data values stored (only state)

Memory Comparison: Real Numbers

import sys

# List comprehension
numbers = [i for i in range(1_000_000)]
print(sys.getsizeof(numbers))  # 8,448,728 bytes (~8.4 MB)

# Generator expression
numbers = (i for i in range(1_000_000))
print(sys.getsizeof(numbers))  # 112 bytes

# Reduction: 75,000x

Scaling:

Items	List Memory	Generator Memory	Ratio
1K	8 KB	112 B	73x
1M	8.4 MB	112 B	75,000x
1B	8.4 GB	112 B	75,000,000x

Generator Expressions

Syntactic sugar for simple generators:

# List comprehension (eager)
squares = [x**2 for x in range(1000)]

# Generator expression (lazy)
squares = (x**2 for x in range(1000))

# Usage in functions
total = sum(x**2 for x in range(1_000_000))  # Memory efficient
# Better than: sum([x**2 for x in range(1_000_000)])

Rule: Use () instead of [] for lazy evaluation.

Practical Example: Log File Streaming

Problem: Analyzing 50GB Log File

Naive approach (crashes):

def find_errors(filename):
    with open(filename) as f:
        lines = f.readlines()  # 50GB in RAM
    return [l for l in lines if 'ERROR' in l]

Generator approach (constant memory):

def find_errors(filename):
    with open(filename) as f:
        for line in f:  # File iterator (built-in generator)
            if 'ERROR' in line:
                yield line.strip()

# Usage
for error in find_errors('app.log'):
    process(error)  # Only one line in memory at a time

Benefits:

✅ Memory: O(1) - constant
✅ Starts immediately (no upfront loading)
✅ Handles files of any size
✅ Can terminate early (no wasted computation)

Iteration Protocol

Generators implement the iterator protocol:

gen = (x for x in range(3))

# Manual iteration
try:
    while True:
        value = next(gen)
        print(value)
except StopIteration:
    pass

# Automatic iteration (preferred)
for value in gen:
    print(value)  # for loop handles StopIteration

Under the hood:

# This:
for item in generator:
    process(item)

# Is equivalent to:
iterator = iter(generator)  # Generators are their own iterators
while True:
    try:
        item = next(iterator)
        process(item)
    except StopIteration:
        break

Generator Exhaustion

Important: Generators are single-use iterators.

gen = (x**2 for x in range(5))

# First iteration
list(gen)  # [0, 1, 4, 9, 16]

# Second iteration
list(gen)  # [] - Generator exhausted!

# Must recreate
gen = (x**2 for x in range(5))
list(gen)  # [0, 1, 4, 9, 16] - Works again

Design implication: Use generators when you need one pass. Use lists for multiple iterations.

Performance Characteristics

Time Complexity

Operation	List	Generator
Creation	O(n)	O(1)
Iteration	O(n)	O(n)
Random access	O(1)	N/A
Multiple passes	O(n) each	Recreate each time

Space Complexity

Operation	List	Generator
Storage	O(n)	O(1)
Iteration	O(n)	O(1)

Conclusion: Generators trade random access and re-iterability for memory efficiency.

When to Use Generators

✅ Use Generators For:

Large datasets that don't fit in memory
Streaming data (logs, network streams, sensors)
Single-pass processing (filter → transform → aggregate)
Infinite sequences (Fibonacci, primes, etc.)
Pipeline processing (chaining transformations)
Data that's expensive to compute (defer computation)

❌ Use Lists For:

Small datasets (< 10,000 items typically)
Multiple iterations required
Random access needed (items[42])
Length required (len(items))
Slicing needed (items[10:20])
Sorting/reversing in-place

Practical Examples

Example 1: Infinite Sequence

def fibonacci():
    """Infinite Fibonacci generator"""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Usage
from itertools import islice
first_10 = list(islice(fibonacci(), 10))
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Example 2: Data Pipeline

def read_file(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

def filter_comments(lines):
    for line in lines:
        if not line.startswith('#'):
            yield line

def to_uppercase(lines):
    for line in lines:
        yield line.upper()

# Chain generators
pipeline = to_uppercase(filter_comments(read_file('data.txt')))

# Memory: 3 × 112 bytes = 336 bytes
# (vs loading entire file into memory)
for line in pipeline:
    process(line)

Example 3: Batch Processing

def batch(iterable, size):
    """Yield items in batches"""
    batch = []
    for item in iterable:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

# Process database records in batches
for batch in batch(query_all_users(), 1000):
    bulk_update(batch)

Common Pitfalls

Pitfall 1: Assuming Multiple Iterations

gen = (x for x in range(5))
sum(gen)  # 10
sum(gen)  # 0 - exhausted!

Fix: Convert to list if you need multiple passes.

Pitfall 2: Debugging Difficulties

gen = (expensive_computation(x) for x in data)
# What's in gen? Can't easily inspect without consuming it

Fix: Use lists during development if you need to inspect intermediate values.

Pitfall 3: Delayed Execution Side Effects

# This doesn't execute immediately!
gen = (print(x) for x in range(5))  # No output yet

# Must iterate to trigger execution
list(gen)  # Now prints 0, 1, 2, 3, 4

Best Practices

Default to generator expressions for transformations

   # Good
   total = sum(x**2 for x in data)

   # Bad (unless you need the list)
   total = sum([x**2 for x in data])

Use itertools for common patterns

   from itertools import islice, chain

   # Take first N items
   first_100 = list(islice(large_generator(), 100))

   # Chain multiple generators
   combined = chain(gen1(), gen2(), gen3())

Close generators that manage resources

   gen = file_reader('data.txt')
   try:
       process(gen)
   finally:
       gen.close()  # Triggers cleanup

Document if your function returns a generator

   def read_logs(path):
       """
       Read log file line by line.

       Yields:
           str: Each log line
       """
       with open(path) as f:
           for line in f:
               yield line.strip()

Summary

🎯 Key Concepts:

Generators use lazy evaluation (compute on-demand)
yield pauses execution and preserves state
Memory: O(1) constant, regardless of data size
Single-use - exhaust after one iteration
Perfect for streaming large or infinite data

🔑 Remember:

Lists: Eager, multiple access, O(n) memory
Generators: Lazy, single access, O(1) memory
Use generators by default for large data processing

Quick Reference

# Generator function
def gen_func():
    yield value

# Generator expression
gen_expr = (x for x in iterable)

# Iteration
for item in generator:
    process(item)

# Manual control
value = next(generator)

# Convert to list (materializes all values)
items = list(generator)

What's Next?

In Part 2, we'll explore:

🔬 Generator internals (frames, bytecode, state storage)
📡 send() method (bidirectional communication)
🛑 close() and throw() methods
🔗 yield from delegation
⚙️ Advanced generator patterns

Next: Python Generators Part 2: How They Actually Work (The Magic Revealed)

You now understand what generators are and why they exist. Part 2 reveals how they work under the hood.

DEV Community: prashant chouksey

Python Generators Part 2: How They Actually Work (The Magic Revealed)

Understanding the pause button: frames, state, and bidirectional communication

The Big Question: Where is the State Saved?

What Happens When You Call a Generator Function

The Generator Object

Inside the Frame: Where Variables Live

The Execution Lifecycle (States)

The Four Powerful Methods

1. next() - Get Next Value ▶️

2. send(value) - Send Data INTO the Generator 📨

Real Example: Running Average Calculator

3. close() - Stop the Generator and Cleanup 🛑

4. throw(exception) - Inject Errors 💥

Method Comparison Table

Advanced: yield from - Generator Delegation

Practical Example: Multi-File Log Reader

Understanding the Pause and Resume

Summary: What You Learned

Python Generators Deep Dive Part 1: Lazy Evaluation & Memory Optimization🚀

The Problem: Memory Bloat in Data Processing

What Are Generators?

Eager vs Lazy Evaluation

Eager (Lists)

Lazy (Generators)

The yield Keyword Explained

Generator State Management

Memory Comparison: Real Numbers

Generator Expressions

Practical Example: Log File Streaming

Problem: Analyzing 50GB Log File

Iteration Protocol

Generator Exhaustion

Performance Characteristics

Time Complexity

Space Complexity

When to Use Generators

✅ Use Generators For:

❌ Use Lists For:

Practical Examples

Example 1: Infinite Sequence

Example 2: Data Pipeline

Example 3: Batch Processing

Common Pitfalls

Pitfall 1: Assuming Multiple Iterations

Pitfall 2: Debugging Difficulties

Pitfall 3: Delayed Execution Side Effects

Best Practices

Summary

Quick Reference

What's Next?

1. `next()` - Get Next Value ▶️

2. `send(value)` - Send Data INTO the Generator 📨

3. `close()` - Stop the Generator and Cleanup 🛑

4. `throw(exception)` - Inject Errors 💥

Advanced: `yield from` - Generator Delegation

The `yield` Keyword Explained