Anik Sikder

Posted on Sep 21

Iterators: The Waiters With One-Way Tickets 🍽️🧑‍🍳

#python #datastructures #webdev #machinelearning

In Part 1 we met the buffet (iterables) the endless source of food. But buffets don’t serve themselves. You need a waiter who walks the line, remembers where you left off, and hands you the next dish.

That’s an iterator:
👉 A waiter with a one-way ticket who can’t walk backwards.

By the end of this post, you’ll know:

Exactly what makes something an iterator (__iter__ and __next__).
How StopIteration actually ends a for loop.
How CPython represents iterators in memory (lightweight, cursor-based).
Why generators are just fancy waiters powered by yield.
Fun quirks, pitfalls, and debugging tips.

Grab a plate, let’s go.

🍴 Opening scene the waiter’s life

Imagine you’re at the buffet:

The buffet (iterable) says: “Here’s a waiter.”
The waiter (iterator) says: “Let me serve you the first dish.”
Each time you call next(), the waiter walks one more step.
When no more food? Waiter drops the tray and shouts: StopIteration!

Key point:
👉 The waiter is stateful. He remembers where he left off.

⚙️ What is an iterator? (the contract)

Python defines an iterator with 2 simple rules:

__iter__()   # must return self
__next__()   # must return next item, or raise StopIteration

Example:

class Countdown:
    def __init__(self, start):
        self.current = start
    def __iter__(self):
        return self
    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        val = self.current
        self.current -= 1
        return val

c = Countdown(3)
print(list(c))  # [3, 2, 1]

👉 Note the weirdness: __iter__() returns self.
Because the waiter is both the iterable and the iterator.

This is why iterators are one-shot once they’re exhausted, they’re done.

🧠 Behind the curtain CPython’s iterator objects

In CPython (the reference Python implementation):

A list_iterator object has:
- ob_ref: pointer to the original list.
- index: an integer cursor (starts at 0).

Each next() does:

Look at list[index].
Increment index.
Return the object.
If index >= len(list), raise StopIteration.

Memory footprint:

The iterator is just a tiny struct (pointer + index).
It does not copy the list.

That’s why iterators are so lightweight a few bytes instead of duplicating your data.

🛑 StopIteration the waiter’s “sorry, no more food”

When an iterator ends, it raises StopIteration.

But you almost never see it because for loops and comprehensions handle it silently.

Example under the hood:

it = iter([1, 2])
while True:
    try:
        item = next(it)
    except StopIteration:
        break
    print(item)

That’s what for x in [1,2]: compiles to.
The StopIteration is the signal that breaks the loop.

🔬 Dissection: Iterables vs Iterators

Let’s recap with buffet metaphors:

Thing	Who is it in the restaurant?	Protocol	Multiple passes?
Iterable	The buffet table	`__iter__`	Yes (new waiter every time)
Iterator	The waiter with the plate	`__iter__` + `__next__`	No (one trip only)

⚡ Generators: Waiters on autopilot

Writing __next__ by hand feels clunky. That’s why Python gave us generators.

A generator is just a special function with yield that remembers its state between calls.

def countdown(n):
    while n > 0:
        yield n   # pause here
        n -= 1

c = countdown(3)
print(next(c))  # 3
print(next(c))  # 2
print(next(c))  # 1
print(next(c))  # StopIteration

Under the hood:

When you call countdown(3), Python creates a generator object.
That object has a stack frame and an instruction pointer.
Each next() resumes execution until the next yield.

It’s like a waiter with a save point in time.

🔍 Fun fact: Iterators are everywhere

Files → for line in open('file.txt'): is an iterator over lines.
Dictionaries → for key in d: iterates keys lazily.
range → returns a range_iterator, no giant list in memory.
zip, map, filter → all return iterators.
itertools → factory of infinite or lazy iterators.

Basically: whenever Python can avoid making a giant list, it uses an iterator.

💾 Memory Showdown list vs iterator vs generator

import sys

nums = [i for i in range(1_000_000)]
print(sys.getsizeof(nums))  # ~8 MB

gen = (i for i in range(1_000_000))
print(sys.getsizeof(gen))   # ~112 bytes 🤯

👉 A list holds all 1M references in memory.
👉 A generator just stores a tiny frame object.

That’s the power of laziness.

🚨 Common pitfalls

Iterator exhaustion

   it = iter([1,2,3])
   print(list(it))  # [1,2,3]
   print(list(it))  # [] (already empty!)

Sharing an iterator across functions

   def f(it): return list(it)
   def g(it): return list(it)

   it = iter([1,2,3])
   print(f(it))  # [1,2,3]
   print(g(it))  # [] (oops)

Infinite loops

   import itertools
   for x in itertools.count():
       print(x)   # will never stop without break

🎨 ASCII mental model

[Iterator (waiter)]
   |
   | next()
   V
 item → item → item → StopIteration

A waiter walks forward, dropping plates. Once empty-handed, game over.

🧭 When to use iterators

✅ Streaming large datasets (logs, CSVs, DB queries).
✅ Lazily combining pipelines (map, filter, itertools).
✅ Infinite sequences with controlled break conditions.
❌ Don’t use them if you need random access or multiple passes (use lists/tuples).

🎬 Wrap-up

We learned:

Iterators are stateful waiters: one trip, no rewinds.
Protocol = __iter__() (returns self) + __next__() (returns item or StopIteration).
CPython iterators are tiny, cursor-based objects.
Generators are just syntactic sugar for making iterators.
Iterators save tons of memory but can trip you up with exhaustion.

👉 Next up (Part 3): Advanced Iteration Tricks
we’ll explore itertools, yield from, generator delegation, tee’ing an iterator, and how to build custom lazy pipelines like a pro chef designing a menu.

DEV Community