Python Generators Part 2: How They Actually Work (The Magic Revealed)

Understanding the pause button: frames, state, and bidirectional communication

In Part 1, you learned:

• ✅ Generators produce values one at a time (lazy)
• ✅ The yield keyword pauses the function
• ✅ Generators use constant memory (112 bytes)
• ✅ Use for loops or next() to get values

If you haven't reviewed Part 1 yet, please refer to it here.

Now let's answer the big question: How does Python remember where it paused?

---

The Big Question: Where is the State Saved?

When a generator pauses at yield, it needs to remember:

• 📍 Where it was in the code (which line?)
• 🧮 All local variables (their current values)
• 📊 The call stack (in case there were nested function calls)

Where does Python store all this?

Answer: In a special object called a Generator Frame.

---

What Happens When You Call a Generator Function

Let's trace through this step-by-step:

def counter(start, end):
    current = start
    while current < end:
        value = current ** 2
        yield value
        current += 1

# Step 1: Call the function
gen = counter(0, 3)

What just happened?

❌ WRONG: The function ran and calculated squares

✅ CORRECT: Python created a "Generator Object" and did NOTHING else

The Generator Object

┌─────────────────────────────────────────┐
│    Generator Object (112 bytes)         │
├─────────────────────────────────────────┤
│  gi_frame: <Frame object>               │  ← WHERE STATE IS SAVED
│  gi_code: <counter function bytecode>   │  ← THE CODE TO RUN
│  gi_running: False                       │  ← IS IT RUNNING NOW?
│  gi_yieldfrom: None                      │  ← For yield from
└─────────────────────────────────────────┘

The most important part: gi_frame - this is where all the magic happens!

---

Inside the Frame: Where Variables Live

The frame (gi_frame) contains:

┌─────────────────────────────────────────┐
│           Generator Frame                │
├─────────────────────────────────────────┤
│  f_locals: {}                            │  ← Local variables
│  f_lasti: 0                              │  ← Instruction pointer
│  f_valuestack: []                        │  ← Expression stack
│  f_back: None                            │  ← Caller's frame
└─────────────────────────────────────────┘

Let's see what happens when we call next():

gen = counter(0, 3)

# FIRST next()
print(next(gen))  # 0

Frame BEFORE first next():

f_locals: {}  (empty - hasn't run yet!)
f_lasti: 0    (start of function)

Frame AFTER first next() (paused at yield):

f_locals: {
    'start': 0,
    'end': 3,
    'current': 1,     ← Updated!
    'value': 0
}
f_lasti: 58    ← Points to instruction AFTER yield

Frame AFTER second next():

f_locals: {
    'start': 0,
    'end': 3,
    'current': 2,     ← Updated again!
    'value': 1
}
f_lasti: 58    ← Same yield point

See the magic? Python saves everything and resumes exactly where it left off!

---

The Execution Lifecycle (States)

A generator goes through different states:

┌──────────────┐
│ GEN_CREATED  │  ← Just created, hasn't run yet
└──────────────┘
       │
       │ next() or send()
       ▼
┌──────────────┐
│ GEN_RUNNING  │  ← Currently executing code
└──────────────┘
       │
       │ hit yield
       ▼
┌──────────────┐
│GEN_SUSPENDED │  ← ⏸️ Paused, state saved
└──────────────┘
       │
       ├──── next() ────▶ (loop back to GEN_RUNNING)
       │
       └──── StopIteration or close() ────▶ GEN_CLOSED

---

The Four Powerful Methods

Generators aren't just for reading values! They support four methods for advanced control:

1. next() - Get Next Value ▶️

You already know this one!

def simple_gen():
    yield 1
    yield 2
    yield 3

gen = simple_gen()
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # 3
print(next(gen))  # StopIteration Error!

---

2. send(value) - Send Data INTO the Generator 📨

This is where it gets interesting! You can send values into a generator while it's running.

def echo():
    while True:
        # The yield can RECEIVE a value!
        received = yield
        print(f"You sent: {received}")

gen = echo()
next(gen)  # ⚠️ MUST prime first! (advance to first yield)

gen.send("Hello")   # You sent: Hello
gen.send("World")   # You sent: World
gen.send(42)        # You sent: 42

How does send() work?

Generator code:    received = yield
                              ↑
                              │
gen.send("Hello") ────────────┘

The yield expression RECEIVES the value "Hello"
Then: received = "Hello"

Real Example: Running Average Calculator

def running_average():
    total = 0
    count = 0
    average = None

    while True:
        # Yield the current average, receive the next number
        number = yield average

        if number is None:
            break

        total += number
        count += 1
        average = total / count

# Usage
avg = running_average()
next(avg)  # Prime (get to first yield)

print(avg.send(10))  # 10.0  (running avg: 10)
print(avg.send(20))  # 15.0  (running avg: 15)
print(avg.send(30))  # 20.0  (running avg: 20)
print(avg.send(40))  # 25.0  (running avg: 25)

Step-by-step flow:

1. next(avg)
   ├─ Start function
   ├─ Set total=0, count=0, average=None
   ├─ Hit: number = yield average
   ├─ Yield None (first average)
   └─ ⏸️ PAUSE

2. avg.send(10)
   ├─ Resume: number = yield average
   ├─ number receives 10
   ├─ total = 10, count = 1
   ├─ average = 10.0
   ├─ Loop back to: number = yield average
   ├─ Yield 10.0
   └─ ⏸️ PAUSE

3. avg.send(20)
   ├─ Resume: number = yield average
   ├─ number receives 20
   ├─ total = 30, count = 2
   ├─ average = 15.0
   ├─ Yield 15.0
   └─ ⏸️ PAUSE

---

3. close() - Stop the Generator and Cleanup 🛑

When you're done with a generator, you should close it to free resources:

def file_reader(filename):
    print(f"Opening {filename}")
    f = open(filename, 'r')

    try:
        for line in f:
            yield line.strip()
    except GeneratorExit:
        # This runs when close() is called!
        print("Closing file...")
        f.close()
        raise  # Must re-raise or return

# Usage
reader = file_reader('data.txt')
print(next(reader))  # First line
print(next(reader))  # Second line

reader.close()  # Triggers cleanup!
# Output: Closing file...

What close() does:

1. Raises GeneratorExit exception inside the generator
2. Generator's except GeneratorExit block runs
3. Cleanup code executes
4. Generator marked as GEN_CLOSED

---

4. throw(exception) - Inject Errors 💥

You can raise an exception inside the generator from outside:

def error_handler():
    while True:
        try:
            value = yield
            print(f"Processing: {value}")
        except ValueError as e:
            print(f"Caught error: {e}")
            # Generator continues running!

gen = error_handler()
next(gen)  # Prime

gen.send(10)  # Processing: 10
gen.send(20)  # Processing: 20

# Inject an error from outside!
gen.throw(ValueError, "Bad value!")
# Output: Caught error: Bad value!

gen.send(30)  # Processing: 30 (still works!)

Use case: Error handling in data pipelines

def data_ ():
    while True:
        try:
            data = yield
            if data < 0:
                raise ValueError("Negative values not allowed")
            result = process(data)
            print(f"Result: {result}")
        except ValueError as e:
            print(f"Skipping bad data: {e}")

processor = data_processor()
next(processor)

processor.send(10)   # Result: ...
processor.send(-5)   # Skipping bad data: Negative values not allowed
processor.send(20)   # Result: ... (continues!)

---

Method Comparison Table

Method	What It Does	Returns	Use When
next(gen)	Get next value	Next yielded value	Basic iteration
gen.send(val)	Send value into generator	Next yielded value	Two-way communication
gen.throw(exc)	Raise exception inside	Next yield (if handled)	Error injection
gen.close()	Stop generator	None	Cleanup resources

---

Advanced: yield from - Generator Delegation

When you have generators calling other generators, yield from makes it seamless:

def generator1():
    yield 1
    yield 2

def generator2():
    yield 3
    yield 4

# Without yield from
def combined_old():
    for value in generator1():
        yield value
    for value in generator2():
        yield value

# With yield from (cleaner!)
def combined_new():
    yield from generator1()
    yield from generator2()

print(list(combined_new()))  # [1, 2, 3, 4]

Why is yield from better?

It transparently forwards all operations:

• ✅ next() calls
• ✅ send() values
• ✅ throw() exceptions
• ✅ close() signals

---

Practical Example: Multi-File Log Reader

def read_file(filename):
    """Read one file"""
    print(f"Opening {filename}")
    with open(filename) as f:
        for line in f:
            yield line.strip()
    print(f"Closed {filename}")

def read_multiple_files(filenames):
    """Read multiple files sequentially"""
    for filename in filenames:
        yield from read_file(filename)

# Usage
files = ['log1.txt', 'log2.txt', 'log3.txt']
for line in read_multiple_files(files):
    if 'ERROR' in line:
        print(line)

Output:

Opening log1.txt
... errors from log1 ...
Closed log1.txt
Opening log2.txt
... errors from log2 ...
Closed log2.txt
Opening log3.txt
... errors from log3 ...
Closed log3.txt

---

Understanding the Pause and Resume

Let's visualize exactly what happens:

def demo():
    print("A")
    x = yield 1
    print(f"B, x={x}")
    y = yield 2
    print(f"C, y={y}")

gen = demo()

Execution trace:

Call: gen = demo()
  ➜ Creates generator object
  ➜ NO CODE RUNS
  ➜ State: GEN_CREATED

Call: next(gen)
  ➜ Start execution
  ➜ Execute: print("A")          [Output: "A"]
  ➜ Execute: yield 1
  ➜ Return: 1
  ➜ Save state (x not assigned yet!)
  ➜ State: GEN_SUSPENDED
  ➜ Returns: 1

Call: gen.send(100)
  ➜ Resume execution
  ➜ x = 100 (yield receives sent value)
  ➜ Execute: print(f"B, x={x}")  [Output: "B, x=100"]
  ➜ Execute: yield 2
  ➜ Return: 2
  ➜ Save state (y not assigned yet!)
  ➜ State: GEN_SUSPENDED
  ➜ Returns: 2

Call: gen.send(200)
  ➜ Resume execution
  ➜ y = 200
  ➜ Execute: print(f"C, y={y}")  [Output: "C, y=200"]
  ➜ End of function
  ➜ Raise: StopIteration
  ➜ State: GEN_CLOSED

---

Summary: What You Learned

🎯 Key Concepts:

1. Generator state is saved in a "frame" (gi_frame)
2. Frames store local variables and instruction pointer
3. send() allows bidirectional communication
4. close() triggers cleanup with GeneratorExit
5. throw() injects exceptions for error handling
6. yield from delegates to sub-generators

🔑 Remember:

• Generators go through states: CREATED → RUNNING → SUSPENDED → CLOSED
• The frame remembers everything between yields
• You can send data into generators, not just get data out

Happy coding! 🎉