Python Generators Finally Made Sense When I Stopped Thinking About Lists

For a long time, I thought Python generators were just a complicated way to create lists.

I knew yield made something "lazy", but I didn't really understand what that meant. Most examples showed a function yielding a few numbers, and I couldn't see why I'd use that instead of returning a list.

Then two things finally clicked.

A list stores values. A generator produces values when you ask for them.

And more importantly:

A generator is a function that pauses.

When you call a generator function, Python doesn't immediately run its body. It gives you a generator object. That object runs the function a little at a time, pausing at each yield until you ask for another value.

That's the part I was missing.

Let me show you what actually happens.

Step 1: Calling a generator doesn't run it

Here's a simple countdown:

def countdown(n):
    print("starting")

    while n > 0:
        yield n
        n -= 1

    print("done")

gen = countdown(3)

print("generator created")

Output:

generator created

Notice something?

There's no "starting" in the output.

Even though we called countdown(3), Python hasn't executed the function body yet.

It created a generator object and handed it back.

Now we have to ask it for a value.

Step 2: next() runs it until the next yield

Let's call next() a few times:

print(next(gen))
print(next(gen))
print(next(gen))

Output:

starting
3
2
1

Here's what happened:

1. The first next() starts executing the function, prints "starting", and runs until yield n. It gives us 3.
2. The second next() resumes after that yield, decrements n, and gives us 2.
3. The third does the same thing and gives us 1.

The important part is that the function doesn't start over each time.

It remembers where it stopped, including the values of its local variables.

That's what yield does. It hands back a value without finishing the function.

What happens if we call next() again?

try:
    next(gen)
except StopIteration:
    print("no more values")

Output:

done
no more values

The function finishes, and Python raises StopIteration because there are no more values to produce.

That's the basic generator mechanism.

Step 3: A for loop does all of this for you

In everyday Python code, you rarely need to call next() manually.

A for loop handles it:

for number in countdown(3):
    print(number)

Output:

starting
3
2
1
done

The loop keeps requesting values and automatically stops when the generator is exhausted.

From the outside, it looks almost identical to looping over a list.

But there's a big difference in when the work happens.

Step 4: Stop early, and the remaining work never happens

This is where generators started feeling genuinely useful to me.

Imagine we're converting strings into integers.

I want to find the first number greater than 10.

Here's a generator that prints whenever it processes something:

def parse(values):
    for value in values:
        print(f"  parsing {value!r}")
        yield int(value)

data = ["4", "8", "15", "16", "23", "42"]

for number in parse(data):
    if number > 10:
        print("found", number)
        break

Output:

  parsing '4'
  parsing '8'
  parsing '15'
found 15

Look at that output.

We had six strings, but Python only converted three.

The moment we found 15, we stopped asking for more values.

The remaining three strings were never processed.

Now compare that with building a list first:

numbers = [int(value) for value in data]

for number in numbers:
    if number > 10:
        print("found", number)
        break

This produces the same result, but the list comprehension converts all six strings before the loop begins.

With six values, that difference barely matters.

With six million records, an expensive calculation, or an API you're paging through, avoiding unnecessary work becomes much more interesting.

And this is also where the memory benefit comes in.

The generator doesn't need to store every converted value in a new list.

It produces them as they're requested.

Step 5: Something realistic — processing application logs

Let's use a more practical example.

Suppose we have a large application log and want to extract only the error messages.

First, let's create a tiny log file so you can run the entire example yourself:

with open("app.log", "w", encoding="utf-8") as file:
    file.write(
        "09:00:01 INFO server started\n"
        "09:00:05 ERROR database timeout\n"
        "09:00:09 INFO request ok\n"
        "09:01:12 ERROR disk almost full\n"
        "09:02:30 ERROR database timeout\n"
    )

Now we'll build three generators, each with one responsibility.

One reads lines, one parses them into records, and one filters by log level.

def read_lines(source):
    for line in source:
        yield line.rstrip("\n")


def parse_log(lines):
    for line in lines:
        time, level, message = line.split(" ", 2)

        yield {
            "time": time,
            "level": level,
            "message": message,
        }


def only(level, records):
    for record in records:
        if record["level"] == level:
            yield record

Now connect them:

with open("app.log", encoding="utf-8") as file:
    errors = only(
        "ERROR",
        parse_log(read_lines(file))
    )

    for record in errors:
        print(record["time"], record["message"])

Output:

09:00:05 database timeout
09:01:12 disk almost full
09:02:30 database timeout

The interesting part is how these generators work together.

The for loop asks only() for a record.

only() asks parse_log() for a parsed record.

parse_log() asks read_lines() for a line.

And read_lines() gets that line from the file.

The line passes through the pipeline before another is requested.

We're not loading the entire file into a list, parsing every record upfront, and then building another list containing errors.

Each stage processes data incrementally.

That's useful whether the log contains five lines or millions.

What if we only want the first two errors?

Python's itertools.islice lets us stop after a certain number of results.

from itertools import islice

with open("app.log", encoding="utf-8") as file:
    errors = only(
        "ERROR",
        parse_log(read_lines(file))
    )

    for record in islice(errors, 2):
        print(record["message"])

Output:

database timeout
disk almost full

Once the second matching record is produced, iteration stops. The remaining lines aren't read by our pipeline.

The with open(...) block also makes sure the file is closed when we're done, including when we stop early.

This is the kind of problem where generators really earn their place.

And notice how each stage is just a small, reusable function.

parse_log() doesn't care whether its lines came from a file, a list, or another generator.

Step 6: Generator expressions — the one-line version

You've probably used a list comprehension like this:

squares = [x * x for x in range(5)]

That builds a list containing all five results.

Change the square brackets to parentheses:

squares = (x * x for x in range(5))

Now it's a generator expression.

The values are produced when you ask for them.

print(next(squares))  # 0
print(next(squares))  # 1
print(next(squares))  # 4

Here's something interesting.

What happens if we turn that same generator into a list now?

print(list(squares))

Output:

[9, 16]

Not [0, 1, 4, 9, 16].

Why?

Because we already consumed the first three values.

The generator continues from where it left off.

That little example explains one of the biggest differences between generators and lists.

Generator expressions are also handy with functions like sum() and any():

print(sum(x * x for x in range(5)))
# 30

print(any(x > 10 for x in range(20)))
# True

sum() consumes all the generated values.

any() stops as soon as it finds a truthy result, so it doesn't need to check all 20 numbers.

The gotchas, so you don't learn them the hard way

Generators are useful, but there are a few behaviors that can catch you off guard.

1. You can't reuse an exhausted generator

numbers = (x * x for x in range(3))

print(list(numbers))
print(list(numbers))

Output:

[0, 1, 4]
[]

The second list is empty because the generator has already finished.

If you need another pass, create a new generator or store the results in a list.

2. An empty generator is still truthy

This one is surprisingly easy to miss:

numbers = (x for x in [])

if numbers:
    print("this still prints")

Output:

this still prints

A generator object itself is truthy, regardless of whether it will produce any values.

So if numbers: doesn't check whether the generator has data.

If you need to check for at least one item, you can use next(numbers, None) when None isn't a valid value in your data.

Just remember that checking this way also consumes the first value, if one exists.

3. No indexing or len()

This doesn't work:

numbers = (x for x in range(5))

print(numbers[0])  # TypeError
print(len(numbers))  # TypeError

Generators don't support random-access indexing or len().

If you need those operations, a list is usually a better fit.

4. Errors can appear later than you expect

Remember our parse() generator?

numbers = parse(["4", "oops"])

No error yet.

Now:

print(next(numbers))  # 4
print(next(numbers))  # ValueError

The second call fails because "oops" can't be converted to an integer.

The error didn't happen when we created the generator. It happened when execution reached the invalid value.

This also matters when testing generators.

If your test only creates a generator without iterating over it, you haven't actually tested its body.

5. Turning everything back into lists

numbers = list(x * x for x in range(1_000_000))

This is valid Python.

But now you've stored all one million values.

Sometimes that's exactly what you need. Other times, you've lost the memory benefit you were trying to get.

The important question isn't whether you can use a generator.

It's whether you need all the results stored at once.

So, list or generator?

I don't think generators should replace lists everywhere.

They solve different problems.

List	Generator
Stores its elements	Produces values on demand
Can be iterated repeatedly	Gets consumed as you iterate
Supports indexing	No direct indexing
Supports len()	No built-in length
Convenient when you need all results	Useful when processing values one at a time
Holds its elements in memory	Avoids storing the full generated sequence

Use a list when you need to access elements repeatedly, index into them, sort them, or keep the results around.

Use a generator when you're dealing with large data, might stop early, or want to connect processing steps without building intermediate lists.

And one thing worth remembering: generators aren't automatically faster.

Producing values one at a time has overhead too. Their advantage is often reduced memory usage and avoiding work you never needed to do.

The thing that finally made it click

I used to think of a generator as a different kind of list.

Now I think of it as a paused computation that knows how to produce the next value.

That's why yield makes sense.

That's why next() resumes execution instead of starting over.

That's why a generator can be exhausted.

And that's why generators work so well for reading files, processing records, and chaining transformations.

A list is a collection of values you already have.

A generator is a way to get the next value when you need it.

Once I understood that difference, generators stopped feeling complicated.

Go deeper, hands-on

Generators are one of those Python features that make much more sense when you actually write a few and watch what happens.

I'm building Runtime, a free hands-on learning platform for Python and SQL.

There's an interactive lesson on generator functions where you can experiment with yield, write your own generators, and work through coding challenges in your browser.

Practice Python generators on Runtime

The lessons that follow explore yield from, lazy processing pipelines, and measuring memory usage with tracemalloc.

If you've been learning generators, try the stop-early example from this post. Change the condition, add a few more values, and see exactly when Python does the work.

When did Python generators finally click for you?