Generators in Python

In Python, a generator is a special type of iterable, similar to lists or tuples, but with a key distinction: it doesn't store all of its values in memory at once. Instead, it generates values on the fly as you iterate over it. This makes generators memory-efficient, especially when dealing with large datasets or when generating an indefinite sequence of values.

Image by Peter H from Pixabay

Generators are defined using functions or generator expressions, and they make use of the yield keyword to produce values one at a time. When the yield statement is encountered in a generator function, it temporarily suspends the function's execution, yielding the value to the caller. The function's state is preserved, so it can resume from where it left off the next time it's called.

Here's a simple example of a generator function:

    def countdown(n):
        while n > 0:
            yield n
            n -= 1

    # Using the generator function
    for i in countdown(5):
        print(i)

    # Result
    # 5
    # 4
    # 3
    # 2
    # 1

In this example, the countdown function generates a countdown from n to 1. Each time the yield statement is encountered, it produces the current value of n, and the function's state is saved. When you iterate over the generator with a for loop, it will yield values one at a time.
Generator expressions are another way to create generators more compactly. They have a syntax similar to list comprehensions, but they are enclosed in parentheses instead of square brackets:

    squares = (x * x for x in range(1, 6))
    for square in squares:
        print(square)

    # Result
    # 1
    # 4
    # 9
    # 16
    # 25

Generator expressions work in the same way as generator functions, producing values on the fly.
The benefits of using generators include reduced memory usage and improved performance when working with large datasets, as you can process elements one by one without loading the entire dataset into memory. Generators are also useful when working with infinite sequences or when you don't know in advance how many values you need to generate.

Remember that once you iterate through all the values generated by a generator, it's typically exhausted, and you'll need to recreate the generator to iterate through it again.

No length

Generators do not have length. This is part of memory saving but it means that len(countdown) will give you a type error.

An example of the type error

Using outside a loop

Well, generators are mostly used within a for-loop you can use them outside of one with __next__() function. This is not a common use case. Every time you call the function you get the next generated element. If you call it more than possible you will get a StopIteration exception raised.

Rule of thumb: if you call __next__() more then once you might just need a for-loop

Example of the next function that is called to many times

Get the last element

The -1 index will not work on generators, given they have no length. You can use this one-liner. However, it will loop through all to get to the end!

    def countdown(n):
        while n > 0:
            print("generating ", n)
            yield n
            n -= 1
    *_, last = countdown(5)
    print(last)

    # Result
    # generating  5
    # generating  4
    # generating  3
    # generating  2
    # generating  1
    # 1

Conclusion

In summary, Python generators are memory-efficient tools for managing dynamic data. They’re ideal for large datasets and infinite sequences, as they generate values on the fly. Generators lack a fixed length, so len() won't work, and they're typically used within for loops. While you can use __next__() outside loops, overuse may lead to a StopIteration exception. In Python, generators are a valuable solution for efficient data processing.