Iterators & Generators in Python

This post aims to describe the basic mechanisms behind iterators and generators.

Iterator protocol

As in many programming languages, Python allows to iterate over a collection. The iteration mechanism is often useful when we need to scan a sequence, operation that is very common in programming.
In Python the iterator protocol involves two components: an iterable and an iterator.

Iterable

The iterable is the container through which we want to iterate. It is the object that needs to be scanned to retrieve all the elements (or some of them). Some of well-known iterables are lists, tuples, dictionaries, ranges. In the iterator protocol the iterable exposes a iter method that returns an iterator object.

Iterator

The iterator is the data structure that allows the scan through the container. It could seem a complication, but actually, with the separation of concerns, it lets the developer separate the concept of container, from the concept of iteration. The container object doesn't need to keep the state of an iteration and furthermore, on the same object, many iterations at the same time can take place, so keeping the iteration state in a different object is a must. The container is a collection of elements, while the iterator is kind of an handler of the container, it exposes the same elements (owned by the container) one-by-one with a specific order and . In the iterator protocol, the iterator exposes two methods: iter and next. While the first one return the object itself (it allows the usage of both container and iterator in for and in statements), the latter return the next item from the container. What does it make the iterator end the iteration? The StopIteration exception.

Iterable and Iterator Examples

Below an example of iterator protocol implementation:

And its usage:

Generator

Generators are methods with yield statements. The yield statement has the power to suspend the function execution and store its state, so that it can be resumed. Behind the scenes, python returns the control to the function caller and save the function state; in this way, at the next execution, the function will start where it left, without letting the developer worrying about the state of the function. Generators ARE iterators, but not vice-versa.

Here an example of generator:

and its usage:

Async generator

Like generators they are async function with a yield statement.

Conclusions

• Iterators are iterables.
• Iterators are objects that implement the iterator protocol consisting in implementing both __iter__ and __next__.
• Iterator iterations stop when StopIteration is raised.
• Generators are methods with yield statements.
• Async generators are async methods with yield statements.
• Whenever possible, generator must be the preferred method due to its simplicity, while the protocol implementation gives much more control.

Resources

• Iterator types
• Generator types
• Yield expression
• Generators and iterators