DEV Community: Krishna

Python Generators Finally Made Sense When I Stopped Thinking About Lists

Krishna — Mon, 26 Oct 2026 18:39:59 +0000

For a long time, I thought Python generators were just a complicated way to create lists.

I knew yield made something "lazy", but I didn't really understand what that meant. Most examples showed a function yielding a few numbers, and I couldn't see why I'd use that instead of returning a list.

Then two things finally clicked.

A list stores values. A generator produces values when you ask for them.

And more importantly:

A generator is a function that pauses.

When you call a generator function, Python doesn't immediately run its body. It gives you a generator object. That object runs the function a little at a time, pausing at each yield until you ask for another value.

That's the part I was missing.

Let me show you what actually happens.

Step 1: Calling a generator doesn't run it

Here's a simple countdown:

def countdown(n):
    print("starting")

    while n > 0:
        yield n
        n -= 1

    print("done")

gen = countdown(3)

print("generator created")

Output:

generator created

Notice something?

There's no "starting" in the output.

Even though we called countdown(3), Python hasn't executed the function body yet.

It created a generator object and handed it back.

Now we have to ask it for a value.

Step 2: `next()` runs it until the next `yield`

Let's call next() a few times:

print(next(gen))
print(next(gen))
print(next(gen))

Output:

starting
3
2
1

Here's what happened:

The first next() starts executing the function, prints "starting", and runs until yield n. It gives us 3.
The second next() resumes after that yield, decrements n, and gives us 2.
The third does the same thing and gives us 1.

The important part is that the function doesn't start over each time.

It remembers where it stopped, including the values of its local variables.

That's what yield does. It hands back a value without finishing the function.

What happens if we call next() again?

try:
    next(gen)
except StopIteration:
    print("no more values")

Output:

done
no more values

The function finishes, and Python raises StopIteration because there are no more values to produce.

That's the basic generator mechanism.

Step 3: A `for` loop does all of this for you

In everyday Python code, you rarely need to call next() manually.

A for loop handles it:

for number in countdown(3):
    print(number)

Output:

starting
3
2
1
done

The loop keeps requesting values and automatically stops when the generator is exhausted.

From the outside, it looks almost identical to looping over a list.

But there's a big difference in when the work happens.

Step 4: Stop early, and the remaining work never happens

This is where generators started feeling genuinely useful to me.

Imagine we're converting strings into integers.

I want to find the first number greater than 10.

Here's a generator that prints whenever it processes something:

def parse(values):
    for value in values:
        print(f"  parsing {value!r}")
        yield int(value)

data = ["4", "8", "15", "16", "23", "42"]

for number in parse(data):
    if number > 10:
        print("found", number)
        break

Output:

  parsing '4'
  parsing '8'
  parsing '15'
found 15

Look at that output.

We had six strings, but Python only converted three.

The moment we found 15, we stopped asking for more values.

The remaining three strings were never processed.

Now compare that with building a list first:

numbers = [int(value) for value in data]

for number in numbers:
    if number > 10:
        print("found", number)
        break

This produces the same result, but the list comprehension converts all six strings before the loop begins.

With six values, that difference barely matters.

With six million records, an expensive calculation, or an API you're paging through, avoiding unnecessary work becomes much more interesting.

And this is also where the memory benefit comes in.

The generator doesn't need to store every converted value in a new list.

It produces them as they're requested.

Step 5: Something realistic — processing application logs

Let's use a more practical example.

Suppose we have a large application log and want to extract only the error messages.

First, let's create a tiny log file so you can run the entire example yourself:

with open("app.log", "w", encoding="utf-8") as file:
    file.write(
        "09:00:01 INFO server started\n"
        "09:00:05 ERROR database timeout\n"
        "09:00:09 INFO request ok\n"
        "09:01:12 ERROR disk almost full\n"
        "09:02:30 ERROR database timeout\n"
    )

Now we'll build three generators, each with one responsibility.

One reads lines, one parses them into records, and one filters by log level.

def read_lines(source):
    for line in source:
        yield line.rstrip("\n")


def parse_log(lines):
    for line in lines:
        time, level, message = line.split(" ", 2)

        yield {
            "time": time,
            "level": level,
            "message": message,
        }


def only(level, records):
    for record in records:
        if record["level"] == level:
            yield record

Now connect them:

with open("app.log", encoding="utf-8") as file:
    errors = only(
        "ERROR",
        parse_log(read_lines(file))
    )

    for record in errors:
        print(record["time"], record["message"])

Output:

09:00:05 database timeout
09:01:12 disk almost full
09:02:30 database timeout

The interesting part is how these generators work together.

The for loop asks only() for a record.

only() asks parse_log() for a parsed record.

parse_log() asks read_lines() for a line.

And read_lines() gets that line from the file.

The line passes through the pipeline before another is requested.

We're not loading the entire file into a list, parsing every record upfront, and then building another list containing errors.

Each stage processes data incrementally.

That's useful whether the log contains five lines or millions.

What if we only want the first two errors?

Python's itertools.islice lets us stop after a certain number of results.

from itertools import islice

with open("app.log", encoding="utf-8") as file:
    errors = only(
        "ERROR",
        parse_log(read_lines(file))
    )

    for record in islice(errors, 2):
        print(record["message"])

Output:

database timeout
disk almost full

Once the second matching record is produced, iteration stops. The remaining lines aren't read by our pipeline.

The with open(...) block also makes sure the file is closed when we're done, including when we stop early.

This is the kind of problem where generators really earn their place.

And notice how each stage is just a small, reusable function.

parse_log() doesn't care whether its lines came from a file, a list, or another generator.

Step 6: Generator expressions — the one-line version

You've probably used a list comprehension like this:

squares = [x * x for x in range(5)]

That builds a list containing all five results.

Change the square brackets to parentheses:

squares = (x * x for x in range(5))

Now it's a generator expression.

The values are produced when you ask for them.

print(next(squares))  # 0
print(next(squares))  # 1
print(next(squares))  # 4

Here's something interesting.

What happens if we turn that same generator into a list now?

print(list(squares))

Output:

[9, 16]

Not [0, 1, 4, 9, 16].

Why?

Because we already consumed the first three values.

The generator continues from where it left off.

That little example explains one of the biggest differences between generators and lists.

Generator expressions are also handy with functions like sum() and any():

print(sum(x * x for x in range(5)))
# 30

print(any(x > 10 for x in range(20)))
# True

sum() consumes all the generated values.

any() stops as soon as it finds a truthy result, so it doesn't need to check all 20 numbers.

The gotchas, so you don't learn them the hard way

Generators are useful, but there are a few behaviors that can catch you off guard.

1. You can't reuse an exhausted generator

numbers = (x * x for x in range(3))

print(list(numbers))
print(list(numbers))

Output:

[0, 1, 4]
[]

The second list is empty because the generator has already finished.

If you need another pass, create a new generator or store the results in a list.

2. An empty generator is still truthy

This one is surprisingly easy to miss:

numbers = (x for x in [])

if numbers:
    print("this still prints")

Output:

this still prints

A generator object itself is truthy, regardless of whether it will produce any values.

So if numbers: doesn't check whether the generator has data.

If you need to check for at least one item, you can use next(numbers, None) when None isn't a valid value in your data.

Just remember that checking this way also consumes the first value, if one exists.

3. No indexing or `len()`

This doesn't work:

numbers = (x for x in range(5))

print(numbers[0])  # TypeError
print(len(numbers))  # TypeError

Generators don't support random-access indexing or len().

If you need those operations, a list is usually a better fit.

4. Errors can appear later than you expect

Remember our parse() generator?

numbers = parse(["4", "oops"])

No error yet.

Now:

print(next(numbers))  # 4
print(next(numbers))  # ValueError

The second call fails because "oops" can't be converted to an integer.

The error didn't happen when we created the generator. It happened when execution reached the invalid value.

This also matters when testing generators.

If your test only creates a generator without iterating over it, you haven't actually tested its body.

5. Turning everything back into lists

numbers = list(x * x for x in range(1_000_000))

This is valid Python.

But now you've stored all one million values.

Sometimes that's exactly what you need. Other times, you've lost the memory benefit you were trying to get.

The important question isn't whether you can use a generator.

It's whether you need all the results stored at once.

So, list or generator?

I don't think generators should replace lists everywhere.

They solve different problems.

List	Generator
Stores its elements	Produces values on demand
Can be iterated repeatedly	Gets consumed as you iterate
Supports indexing	No direct indexing
Supports `len()`	No built-in length
Convenient when you need all results	Useful when processing values one at a time
Holds its elements in memory	Avoids storing the full generated sequence

Use a list when you need to access elements repeatedly, index into them, sort them, or keep the results around.

Use a generator when you're dealing with large data, might stop early, or want to connect processing steps without building intermediate lists.

And one thing worth remembering: generators aren't automatically faster.

Producing values one at a time has overhead too. Their advantage is often reduced memory usage and avoiding work you never needed to do.

The thing that finally made it click

I used to think of a generator as a different kind of list.

Now I think of it as a paused computation that knows how to produce the next value.

That's why yield makes sense.

That's why next() resumes execution instead of starting over.

That's why a generator can be exhausted.

And that's why generators work so well for reading files, processing records, and chaining transformations.

A list is a collection of values you already have.

A generator is a way to get the next value when you need it.

Once I understood that difference, generators stopped feeling complicated.

Go deeper, hands-on

Generators are one of those Python features that make much more sense when you actually write a few and watch what happens.

I'm building Runtime, a free hands-on learning platform for Python and SQL.

There's an interactive lesson on generator functions where you can experiment with yield, write your own generators, and work through coding challenges in your browser.

Practice Python generators on Runtime

The lessons that follow explore yield from, lazy processing pipelines, and measuring memory usage with tracemalloc.

If you've been learning generators, try the stop-early example from this post. Change the condition, add a few more values, and see exactly when Python does the work.

When did Python generators finally click for you?

Python Decorators Finally Clicked When I Saw This One Line

Krishna — Fri, 23 Oct 2026 14:44:29 +0000

Decorators confused me for an embarrassingly long time. Every tutorial jumped straight to @app.route or @lru_cache, and I nodded along without actually getting it.

Then someone showed me the one line that makes it click.

This:

@deco
def f():
    pass

is equivalent to:

def f():
    pass

f = deco(f)

That's it.

A decorator takes a function and returns a replacement. Usually, that replacement is another function that adds behavior around the original.

Everything else builds on that idea. Let me prove it by building one from scratch.

Step 1: Functions are values

In Python, a function is just another value. You can pass it into another function:

def add_exclamation(text):
    return text + "!"

def apply_twice(func, value):
    return func(func(value))

print(apply_twice(add_exclamation, "hello"))
# hello!!

We passed add_exclamation into apply_twice without calling it ourselves. Then apply_twice called it twice.

A decorator uses exactly this ability: it takes a function, builds a wrapper around it, and returns the wrapper.

Step 2: Build a decorator by hand

Say I want to count how many times a function gets called:

def count_calls(func):
    def wrapper(*args, **kwargs):
        wrapper.calls += 1
        return func(*args, **kwargs)

    wrapper.calls = 0
    return wrapper

def parse_magnitude(text):
    return float(text)

parse_magnitude = count_calls(parse_magnitude)

print(parse_magnitude("5.5"))    # 5.5
print(parse_magnitude(" 4.7 "))  # 4.7
print(parse_magnitude.calls)     # 2

What actually happened?

count_calls received the original parse_magnitude function.
It created a new function called wrapper.
The wrapper increments a counter, calls the original function, and returns its result.
count_calls returns the wrapper, which replaces the original function under the name parse_magnitude.

Three things to notice:

wrapper accepts *args, **kwargs and passes them through unchanged, so it can handle different function signatures.
State (the count) lives on the wrapper itself.
The wrapper returns the original's result. Forget that return and every call silently returns None.

One subtle detail: the counter increments before calling the original function. That means failed calls count too.

Step 3: The @ shorthand

That last assignment is so common Python gave it its own syntax:

@count_calls
def place_name(feature):
    return feature["properties"]["place"]

This is the same thing as:

def place_name(feature):
    return feature["properties"]["place"]

place_name = count_calls(place_name)

The name now refers to the wrapper.

There's no special execution engine hiding behind @. Python defines the function, passes it to the decorator, and assigns the returned object to the name.

Once I understood that, decorators stopped looking like magic.

Step 4: Don't lose the original function's identity

There's one catch.

Because the decorated name now points at the wrapper, inspecting the function can give you the wrapper's metadata instead of the original's.

Using our first decorator:

print(place_name.__name__)  # wrapper

That gets confusing when you're debugging, logging, or inspecting functions.

The standard library gives us functools.wraps to fix this:

import functools

def count_calls(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        wrapper.calls += 1
        return func(*args, **kwargs)

    wrapper.calls = 0
    return wrapper

@count_calls
def place_name(feature):
    """Return the place name from a feature."""
    return feature["properties"]["place"]

print(place_name.__name__)  # place_name
print(place_name.__doc__)
# Return the place name from a feature.

@functools.wraps(func) preserves useful metadata from the original function, including its name and docstring. It also keeps a reference to the wrapped function.

Make it a habit when writing wrappers.

And yes, @wraps is itself a decorator.

Step 5: A realistic one — logging with failures

Wrappers can run code before and after a function call, and they can see exceptions too.

Imagine you're processing incoming data and want to know which functions were called, what arguments they received, and whether they succeeded.

Instead of adding logging to every function, we can write one reusable decorator.

import functools

log = []

def log_calls(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        parts = [repr(a) for a in args]
        parts += [
            f"{k}={v!r}"
            for k, v in kwargs.items()
        ]

        call = f"{func.__name__}({', '.join(parts)})"

        try:
            result = func(*args, **kwargs)
        except Exception as err:
            log.append(
                f"{call} raised {type(err).__name__}"
            )
            raise

        log.append(f"{call} -> {result!r}")
        return result

    return wrapper

Now let's apply it to a parser that converts depth measurements into kilometers:

@log_calls
def parse_depth(text, unit="km"):
    value = float(text)
    return value / 1000 if unit == "m" else value

parse_depth("22")
parse_depth("10", unit="m")

try:
    parse_depth("n/a")
except ValueError:
    pass

for line in log:
    print(line)

Output:

parse_depth('22') -> 22.0
parse_depth('10', unit='m') -> 0.01
parse_depth('n/a') raised ValueError

Notice what happened.

Our decorator:

Recorded the function name and arguments, including keyword arguments.
Logged the result when the call succeeded.
Captured the exception type when parsing failed.
Re-raised the exception so the caller still knew something went wrong.

The raise matters. Without it, our wrapper would hide errors and make failed operations appear successful.

And the actual parse_depth function knows nothing about logging.

That's the useful part of decorators: we added behavior without modifying the function's main job.

When to actually use them

Decorators are useful for behavior that multiple functions need in the same way, but that isn't their main responsibility.

Think logging, timing, caching, retrying, validating arguments, or checking permissions.

If you find yourself copy-pasting the same five lines into every function, that's a decorator worth considering.

And a few classic gotchas, so you don't learn them the hard way:

Forgetting return wrapper — the decorated name becomes None.
Calling the original without returning its result — callers receive None.
Using fixed parameters instead of *args, **kwargs — your wrapper only supports certain function signatures.
Swallowing exceptions — failures disappear when callers may need to handle them.
Skipping @functools.wraps — the decorated function exposes the wrapper's name and other metadata.

You don't need a decorator for everything. Sometimes a normal function is simpler.

But when the same supporting behavior belongs around many functions, decorators are a clean way to reuse it.

The one line to remember

If you forget everything else from this post, remember:

@decorator
def my_function():
    pass

means:

def my_function():
    pass

my_function = decorator(my_function)

A function goes in. A replacement comes out.

Once that clicked, the rest of decorators started making sense.

Go deeper, hands-on

Reading about decorators is one thing. Writing them yourself and seeing exactly what breaks is another.

I built a free interactive lesson on this. You can run Python directly in your browser and work through two coding challenges that test the concepts we covered.

Practice Python decorators interactively on Runtime

It's part of Runtime, a free, hands-on learning platform I'm building for Python and SQL.

The idea is simple: learn a concept, write real code, and test whether you actually understand it.

If decorators finally make sense, the next lesson covers decorators that take their own settings, like:

@retry_on(TimeoutError, attempts=3)
def fetch_data():
    ...

That's where things get even more interesting.

What was the moment Python decorators finally clicked for you?

DEV Community: Krishna

Python Generators Finally Made Sense When I Stopped Thinking About Lists

Step 1: Calling a generator doesn't run it

Step 2: next() runs it until the next yield

Step 3: A for loop does all of this for you

Step 4: Stop early, and the remaining work never happens

Step 5: Something realistic — processing application logs

What if we only want the first two errors?

Step 6: Generator expressions — the one-line version

The gotchas, so you don't learn them the hard way

1. You can't reuse an exhausted generator

2. An empty generator is still truthy

3. No indexing or len()

4. Errors can appear later than you expect

5. Turning everything back into lists

So, list or generator?

The thing that finally made it click

Go deeper, hands-on

Python Decorators Finally Clicked When I Saw This One Line

Step 1: Functions are values

Step 2: Build a decorator by hand

Step 3: The @ shorthand

Step 4: Don't lose the original function's identity

Step 5: A realistic one — logging with failures

When to actually use them

The one line to remember

Go deeper, hands-on

Step 2: `next()` runs it until the next `yield`

Step 3: A `for` loop does all of this for you

3. No indexing or `len()`