Behind The Scenes Of For Loop

A For Loop is a control flow statement for specifying iteration in computer science. In simpler terms, a for loop is a section or block of code which runs repeatedly until a certain condition has been satisfied. It is the most basic concept, without which every programming language will be incomplete.

While exploring the ever-surprising python language, I uncovered that it is an Iterator that plays a huge part in the For Loop and many other higher-order functions such as the range function. This article is written to understand what is an iterator, what is a generator, and how it helps us to understand the crux behind the For Loop.

What is an Iterator in python?

An iterator is any python object which can be iterated upon and returns data one at a time. They are implemented in comprehension, loops, and generators but are not readily seen in plain sight. They are the pillars behind most of the beforementioned functionality of python.

An iterator object must support two methods namely iter() and next(). This is collectively called an Iterator protocol.

• iter() is a function used to return an iterator by calling the iter() method on an Iterable object.

• next() is used to iterate through each element. StopIteration is an exception raised whenever the end is reached.

Wait, iter() method is called on an Iterable object? What is the difference between an Iterator and an Iterable?

Well, an Iterable is an object, that one can iterate over. It generates an Iterator when passed to iter() method. Example of iterable objects includes string, list, tuple, dictionary, etc.

Remember, every iterator is also an iterable, but not every iterable is an iterator.

Now that we have some theoretical knowledge about an iterator let’s see them in action.

Let us first see how the iterable object can be converted into an iterator.

    # We know string can be iterated over.
    s = "Loreum Ipsum"
    for word in s:
        print(word)

Output:

    L
    o
    r
    e
    u
    m

    I
    p
    s
    u
    m

What will happen if we call, next() on the string

    s = "Loreum Ipsum"
    next(s)

Output:

    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    ~\AppData\Local\Temp/ipykernel_41560/3883344095.py in <module>
          1 # Calling next() on string
    ----> 2 next(s)

    TypeError: 'str' object is not an iterator

As we mentioned earlier we first need to call the iter method to create an iterator from the iterable object then only we can use this next method to access the element one by one. See the code snippet below which corrects that,

    s = "Loreum Ipsum"
    myIter = iter(s)
    print(next(myIter))
    print(next(myIter))
    print(next(myIter))
    print(next(myIter))

Output:

    L
    o
    r
    e

Now we will try to create an iterator that will print numbers from 1 to n.

    class PrintNumber:
        def __init__(self, max):
            self.max = max

        def __iter__(self):
            self.num = 0
            return self

        def __next__(self):
            if(self.num >= self.max):
                raise StopIteration
            self.num += 1
            return self.num

    print_num = PrintNumber(3)

    print_num_iter = iter(print_num)
    print(next(print_num_iter))  
    print(next(print_num_iter))  
    print(next(print_num_iter)) 

Output:

    1
    2
    3

Notice, we have called next only 3 times, since we explicitly provided max or n as 3. If we give another call to next it will throw StopIteration.

How does For loop work?

Equipped with our knowledge of iterator and how it needs to be implemented it is very easy now to understand how exactly the for loop works. For Loop calls an iter method to get an iterator and then calls method next over and over until a StopIteration exception is raised, which it handles gracefully to end the loop without breaking the code. That’s all, we have debunked the behind the scene of the For loop.

Still, something feels incomplete, inefficient. To create an iterator we have to implement an Iterator Protocol, a class with iter() and next() methods. We also need to keep track of the internal states and handle StopIteration exception so that the code is not broken. It seems like lots of work. This is where Generator comes in.

Generator in Python,

Simply speaking, a generator is a function that returns an iterator. It is very easy to create a generator in python. We simply need to define a normal function with one change being, using the keyword yield instead of return. If a function contains at least one yield statement then it is a generator.

The difference between return and yield is, while return terminates a function entirely, yield temporarily halts a function while retaining its local values and later continues from there on successive calls.

Below is the generator class which iterates the number from 0 to n and yields a number that is divisible by 7

    class divisible_by_7_generator:
        def __init__(self, num):
            self.num = num
        #Generator
        def get_nums_divisible_by_7(self):
            for i in range(0, self.num):
                if (i % 7 == 0):
                    yield i

    n = 100
    result = divisible_by_7_generator(n)
    print(f"Numbers which are divisible between 0 to {n} are:")
    for num in result.get_nums_divisible_by_7():
        print(num, end = ",")

Output:

    Numbers which are divisible between 0 to 100 are:
    0,7,14,21,28,35,42,49,56,63,70,77,84,91,98,

Python Generator Expression

A simple generator can be easily created on the fly using generator expressions. This is very similar to lambda functions with a syntax resembling to the list comprehension in python, with square brackets replaced by round parentheses. Unlike list comprehensions, generator expressions follow lazy execution, meaning producing items only when asked for and thus hugely beneficial for memory management.

Check out the below example, finding a cube of each element in a list.

    myList = [1,2,3,4,5,6]
    #list Comprehension
    cubeList = [x**3 for x in myList]
    # generator expressions are surrounded by parenthesis ()
    cubeGenerator = (x**3 for x in myList)

    print(cubeList)
    print(cubeGenerator)

Output:

    [1, 8, 27, 64, 125, 216]
    <generator object <genexpr> at 0x0000022F5D3EBD60>

We can use for loop to access the values from the generator as follow,

    for i in cubeGenerator: 
        print(i)

Output:

    1
    8
    27
    64
    125
    216

Notice, that list comprehension returns a list while the generator returns elements one at a time.

Conclusion,

In this article, we dug deep into the For Loop and in the process learned about very interesting and useful features such as Iterators, Generators. We also get to understand what is yield and how a generator can be implemented.

Finally, I would like to list a few frequently asked questions related to this topic in interviews:

1. What is the difference between an Iterator and an Iterable?

2. Can next() can be called on a string or on an iterable object?

3. What is an Iterator Protocol?

4. What is a Generator? What is the difference between an Iterator and a Generator?

5. What is the Difference between Generator and a Function?

6. What is Yield and what is the difference between Yield and Return?

If you like this article please follow me and don’t forget to like and give feedback. Every feedback will motivate me to explore more and come up with many more interesting topics in the future. All the code snippets from the article are available on my GitHub repo if anyone wants to play around. You can get in touch with me on my LinkedIn.

Comments

[Si Dunn]

In this example: "Notice, we have called next only 3 times, since we explicitly provided max or n as 3. If we give another call to next it will throw StopIteration."

When I run the code, I can raise StopIteration if I change PrintNumber(3) to PrintNumber(2), but if I change PrintNumber to (4) or (5), etc., I get output
1
2
3
and no StopIteration message.

Did I get something wrong?

[Ashutosh Vaidya]

If you followed the code exactly, this is anticipated behaviour.If you notice, I manually wrote the print statement three times in the code sample.

  print_num = PrintNumber(3)

    print_num_iter = iter(print_num)
    print(next(print_num_iter))  
    print(next(print_num_iter))  
    print(next(print_num_iter)) 

Here, PrintNumber(2) successfully prints 1 and 2 but throws an exception for the third print statement since it exceeds the limit of 2.

The PrintNumber(4) or (5) program does not throw an exception since we are only utilizing three print instructions, which are within the limit of 4 and 5. To register an exception, you must repeat the print statement more than 4 and 5 times, respectively.

This is OK till you have a small number, but it soon becomes repetitive as the number gets bigger. Hence, the best approach would be to use the loop; see the code below that has been altered to use the for loop.

class PrintNumber:
    def __init__(self, max):
        self.max = max

    def __iter__(self):
        self.num = 0
        return self

    def __next__(self):
        if(self.num >= self.max):
            raise StopIteration
        self.num += 1
        return self.num

n = 4
print_num = PrintNumber(n)

print_num_iter = iter(print_num)

for i in range(n):
    print(next(print_num_iter))

# raises StopIteration
print(next(print_num_iter))

[Si Dunn]

Thanks for the explanation and the new code example. And thanks also for the For Loop post and its teachable examples. I've learned from each of them.