Python Generator

Generator functions are a special kind of function that return a lazy iterator. These are objects that you can loop over like a list. However, unlike lists, lazy iterators do not store their contents in memory.

A generator expression (also called a generator comprehension) looks almost identical to a list comprehension - but instead of creating a full list in memory, it creates a generator object that produces values lazily (one at a time).

List Comprehension vs Generator Expression
List comprehension (creates full list in memory)

squares = [x*x for x in range(5)]

print(squares)

Output:

[0, 1, 4, 9, 16]

All values are computed immediately
Stored in memory

Generator expression (lazy evaluation)

squares = (x*x for x in range(5))

print(squares)

***Output: *
Nothing is computed yet
Values are generated only when needed

How to use a generator expression?
You must iterate over it
for num in squares:
print(num)

Or convert it:
print(list(squares))

Feature	List Comprehension	Generator Expression
Syntax	[x for x in ...]	(x for x in ...)
Memory	High	Low
Execution	Immediate	Lazy
Type	list	generator

Memory Example (Important)

import sys

lst = [x for x in range(1000000)]
gen = (x for x in range(1000000))

print(sys.getsizeof(lst))  # large
print(sys.getsizeof(gen))  # small

Generator uses much less memory.

Why "without calling a function"?
Normally, to create a generator, you'd write:

def my_generator():
    for x in range(5):
        yield x*x

But with a generator expression:

gen = (x*x for x in range(5))

No function definition needed.

Very Common Use Case
Passing directly into functions like sum():

total = sum(x*x for x in range(1000000))

Notice:
No brackets
Memory efficient
Clean syntax

Python Yield Statement

On the whole, yield is a fairly simple statement. Its primary job is to control the flow of a generator function in a way that’s similar to return statements.
When you call a generator function or use a generator expression, you return a special iterator called a generator. You can assign this generator to a variable in order to use it. When you call special methods on the generator, such as next(), the code within the function is executed up to yield.

When the Python yield statement is hit, the program suspends function execution and returns the yielded value to the caller. (In contrast, return stops function execution completely.) When a function is suspended, the state of that function is saved. This includes any variable bindings local to the generator, the instruction pointer, the internal stack, and any exception handling.

yield can be used in many ways to control your generator’s execution flow. The use of multiple Python yield statements can be leveraged as far as your creativity allows.

When to use generator expressions?
✔ Large datasets
✔ Streaming data
✔ When you only iterate once
✔ Memory-sensitive applications
❌ When you need indexing or multiple passes

How lazy evaluation works internally?
Lazy evaluation means values are computed only when they are actually needed, not in advance.
In Python, this is implemented mainly through iterators and generators.

Eager vs Lazy (mental model)
Eager evaluation

data = [x * 2 for x in range(5)]

What happens internally:

Loop runs immediately
All values computed
Stored in memory as a list

Lazy evaluation

What happens:

1. Nothing is computed
2. Only a generator object is created
3. Values are computed one at a time, on demand

What a generator really is internally
A generator is:

• A state machine
• With an instruction pointer
• And saved local variables When Python sees yield, it:

1. 1. Pauses execution
2. 2. Saves local state
3. 3. Returns a value
4. Resumes later from the same spot

Step-by-step execution

def squares():
    for i in range(3):
        yield i * i

Internal flow
g = squares() # generator created (no execution)
next(g) # runs until first yield → returns 0
next(g) # resumes → returns 1
next(g) # resumes → returns 4
next(g) # StopIteration raised

At each next():
Python resumes from the last saved instruction
Executes until next yield
Saves state again

Why generators are memory-efficient?

range(1_000_000)

Stores start, stop, step only
No list of numbers

(x*x for x in range(1_000_000))

Stores:
Reference to range
Current index
Execution state

Memory usage is constant, not proportional to size.

Lazy evaluation in built-ins
Many Python functions are lazy:

Function	Lazy?
range()	✅
map()	✅
filter()	✅
zip()	✅
sum()	❌ (consumes lazily)

Example:
map(lambda x: x*x, range(10))

No computation until iterated.

How StopIteration ends lazy evaluation?
When generator finishes:

• Python raises StopIteration
• Iteration protocol catches it
• Loop stops This is how for loops work internally.

Why lazy evaluation is single-pass?

Once consumed:

gen = (x for x in range(3))
list(gen)   # [0, 1, 2]
list(gen)   # []

Why?
State machine has reached the end
No reset unless recreated

“Lazy evaluation in Python works by using iterators and generators, which compute values only when requested. Internally, a generator is a state machine that pauses execution at each yield, saves its local state, and resumes later. This allows Python to process large or infinite data streams efficiently with constant memory usage.”

Python generators support advanced control methods that let you send data into, raise exceptions inside, and terminate a generator from the outside.

These are:

.send(value)
.throw(exception)
.close()

.send(value) — send data into a generator

Normally, generators only yield values out.
.send() allows you to send a value back into the generator, which becomes the result of the last yield expression.

def counter():
    value = yield 0
    while True:
        value = yield value + 1

gen = counter()

print(next(gen))       # start generator → yields 0
print(gen.send(10))    # sends 10 into generator → yields 11
print(gen.send(20))    # yields 21

Key rules
First call must be next(gen) or gen.send(None)
send(x) assigns x to the last yield expression

.throw() — raise an exception inside the generator
.throw() injects an exception at the point where the generator is paused.

def generator():
    try:
        while True:
            yield "running"
    except ValueError:
        yield "ValueError handled"

gen = generator()
print(next(gen))                 # running
print(gen.throw(ValueError))     # ValueError handled

Use cases

Cancel work
Signal error conditions
Interrupt long-running generators
If the generator does not catch the exception → it propagates outward.

.close() — stop the generator gracefully
.close() raises a GeneratorExit inside the generator.

def generator():
    try:
        while True:
            yield "working"
    finally:
        print("Cleaning up resources")

gen = generator()
print(next(gen))
gen.close()

Output
working
Cleaning up resources

**FastAPI dependency that yields a session. Automatically commits if no exception occurs,or rolls back if an exception is raised.**

async def get_db()-> AsyncGenerator[AsyncSession]:
    async with SessionLocal() as session:
       try:
           yield session
           await session.commit()
       except Exception:
           await session.rollback()  # Central rollback
           raise

Lifecycle Summary
Method Purpose
next() Resume generator
send(x) Resume + send value
throw(e) Resume + raise exception
close() Terminate generator

To Summarize

A generator in Python is a function that:
Uses the yield keyword
Produces values one at a time
Remembers its state between executions
Unlike a normal function, it does not return all results at once.

When Python sees yield:
It returns the value
Pauses execution
Saves local state
Resumes from that point on next iteration