Timothy had been using for loops for months, never questioning how they worked. But when he tried to iterate over a custom data structure he'd built—a specialized book index—Python refused with a cryptic error: 'BookIndex' object is not iterable.
Margaret found him puzzling over the error message. "You've been using the conveyor belt system without understanding its mechanics," she said, leading him to the library's basement where massive brass gears and leather belts powered the book delivery system. "It's time to learn the Iterator Protocol—the contract that makes iteration work."
The Mystery of For Loops
Timothy had always taken for loops for granted:
# Note: Examples use placeholder functions like process_chunk(), add_book_to_catalog()
# In practice, replace these with your actual implementation
books = ["1984", "Dune", "Foundation"]
for book in books:
print(book)
"How does Python know how to iterate over a list?" Timothy asked. "And why doesn't it work with my BookIndex?"
Margaret showed him what Python did behind the scenes:
books = ["1984", "Dune", "Foundation"]
# What for loop actually does:
iterator = iter(books) # Get an iterator from the iterable
while True:
try:
book = next(iterator) # Get next item
print(book)
except StopIteration: # No more items
break
"Every for loop," Margaret explained, "is syntactic sugar for this pattern. Python calls iter() on the object to get an iterator, then repeatedly calls next() until StopIteration is raised."
Iterables vs Iterators
Timothy learned the crucial distinction:
Iterable: An object that can produce an iterator (has __iter__ method)
Iterator: An object that produces values one at a time (has __next__ method)
books = ["1984", "Dune", "Foundation"] # This is an ITERABLE
iterator = iter(books) # This is an ITERATOR
print(type(books)) # <class 'list'> - iterable
print(type(iterator)) # <class 'list_iterator'> - iterator
Margaret drew the analogy: "The iterable is like the library's card catalog—it contains information about books. The iterator is like a librarian walking through the stacks, fetching one book at a time."
The Memory Efficiency Advantage
Margaret showed Timothy why iterators mattered:
# Without iterators - creates entire list in memory
million_numbers = [i for i in range(1000000)] # Uses ~8MB of RAM
print(f"List size: {len(million_numbers)} items")
# With iterators - generates on demand
million_range = range(1000000) # Uses ~48 bytes!
print(f"Iterator created: {million_range}")
# Only creates values when needed
for i in million_range:
if i > 5:
break # Stopped after 6 iterations, not a million!
"Iterators," Margaret explained, "generate values lazily—only when requested. This is why range() in Python 3 doesn't create a massive list. It's an iterator that produces numbers on demand."
The Iterator Protocol
Margaret revealed the two special methods that made iteration work:
class SimpleIterator:
def __init__(self, data):
self.data = data
self.index = 0
def __iter__(self):
# Return an iterator (for iterators, return self)
return self
def __next__(self):
# Return the next value
if self.index >= len(self.data):
raise StopIteration # Signal we're done
value = self.data[self.index]
self.index += 1
return value
# Use it
iterator = SimpleIterator([1, 2, 3])
print(next(iterator)) # 1
print(next(iterator)) # 2
print(next(iterator)) # 3
print(next(iterator)) # Raises StopIteration
"The __iter__ method," Margaret explained, "returns an iterator object. The __next__ method returns the next value, or raises StopIteration when exhausted."
The Protocol Summary
Margaret drew a diagram in her notebook:
THE ITERATOR PROTOCOL:
Iterable:
- Has __iter__() method
- Returns a new iterator each call
- Examples: list, str, dict, custom collections
Iterator:
- Has __iter__() method (returns self)
- Has __next__() method (returns next value or raises StopIteration)
- Single-use: exhausted after one iteration
- Examples: list_iterator, generator, file object
"Every iterator is iterable," she noted, "but not every iterable is an iterator."
Why Iterators Return Self
Timothy asked why __iter__ returned self for iterators.
"Because iterators are already positioned in the sequence," Margaret explained. "They're stateful—they remember where they are. Returning self says 'I'm already an iterator, use me as-is.'"
iterator = iter([1, 2, 3])
# Calling iter() on an iterator returns the same object
same_iterator = iter(iterator)
print(iterator is same_iterator) # True - same object!
# This allows iterators to work in for loops
for value in iterator: # for loop calls iter(iterator), which returns itself
print(value)
"In contrast," she continued, "iterables return new iterators each time to allow independent iteration."
Iterables That Aren't Iterators
Timothy discovered that many iterables produce new iterators each time:
class BookCollection:
def __init__(self, books):
self.books = books
def __iter__(self):
# Return a NEW iterator each time
return BookCollectionIterator(self.books)
class BookCollectionIterator:
def __init__(self, books):
self.books = books
self.index = 0
def __iter__(self):
# Iterators return themselves
return self
def __next__(self):
if self.index >= len(self.books):
raise StopIteration
book = self.books[self.index]
self.index += 1
return book
# This pattern allows multiple iterations
collection = BookCollection(["1984", "Dune"])
# First iteration
for book in collection:
print(book)
# Second iteration - works because we get a new iterator
for book in collection:
print(book)
"Separating the iterable from the iterator," Margaret noted, "allows iterating multiple times. Each for loop gets a fresh iterator with its own state."
How Generators Are Iterators
Timothy had an epiphany connecting to his generator knowledge:
def book_generator():
yield "1984"
yield "Dune"
yield "Foundation"
gen = book_generator()
# Generators implement the iterator protocol!
print(hasattr(gen, '__iter__')) # True
print(hasattr(gen, '__next__')) # True
print(next(gen)) # 1984
print(next(gen)) # Dune
"Generators," Margaret explained, "are iterators created with yield. Python automatically implements __iter__ and __next__ for you. That's why they're so convenient—you get the iterator protocol without writing the boilerplate."
The iter() Built-in Function
Margaret showed Timothy two forms of the iter() function:
# Form 1: Get iterator from iterable
books = ["1984", "Dune"]
iterator = iter(books)
# Form 2: Create iterator from callable and sentinel
with open('catalog.txt') as f:
# Read until empty string (sentinel)
for line in iter(f.readline, ''):
if line.strip():
print(line)
The second form was less common but powerful: it repeatedly called the function until it returned the sentinel value.
Built-in Iterators
Timothy explored Python's built-in iterators:
# range - generates numbers on demand
for i in range(1000000): # Doesn't create a million-item list!
if i > 5:
break
# enumerate - adds index to iteration
books = ["1984", "Dune", "Foundation"]
for index, book in enumerate(books):
print(f"{index}: {book}")
# zip - iterates multiple sequences in parallel
titles = ["1984", "Dune"]
authors = ["Orwell", "Herbert"]
for title, author in zip(titles, authors):
print(f"{title} by {author}")
# reversed - iterates in reverse (requires sequence with __len__ and __getitem__)
for book in reversed(books):
print(book)
Each of these created iterators that produced values on demand, rather than building complete lists in memory.
Manual Iteration with next()
Timothy learned he could manually control iteration:
books = iter(["1984", "Dune", "Foundation"])
# Process items manually
first = next(books)
print(f"First: {first}")
second = next(books)
print(f"Second: {second}")
# Process remaining with for loop
for book in books: # Continues from where we left off
print(f"Remaining: {book}")
The iterator maintained state, so the for loop picked up where next() left off.
The Default Value Pattern
Margaret showed Timothy how to provide a default when an iterator exhausted:
books = iter(["1984"])
print(next(books)) # 1984
print(next(books, "No more")) # No more (instead of StopIteration)
The second argument to next() became the default return value when StopIteration would otherwise be raised.
Iterator Exhaustion
Timothy encountered a critical behavior:
books = iter(["1984", "Dune", "Foundation"])
# First iteration
list1 = list(books)
print(list1) # ['1984', 'Dune', 'Foundation']
# Second iteration - empty!
list2 = list(books)
print(list2) # []
"Iterators," Margaret warned, "are single-use. Once exhausted, they're done. If you need multiple iterations, either get a new iterator from the iterable, or convert to a list first."
# Solution 1: Keep the iterable, create new iterators
book_list = ["1984", "Dune", "Foundation"]
for book in book_list: # New iterator each time
print(book)
for book in book_list: # Another new iterator
print(book)
# Solution 2: Convert iterator to list
book_iterator = iter(["1984", "Dune", "Foundation"])
books = list(book_iterator) # Now we can iterate multiple times
The Iterator Chain
Timothy discovered he could chain iterators:
import itertools
classics = ["1984", "Brave New World"]
scifi = ["Dune", "Foundation"]
# Chain iterators together
for book in itertools.chain(classics, scifi):
print(book)
# 1984
# Brave New World
# Dune
# Foundation
The itertools module provided tools for working with iterators efficiently.
Infinite Iterators
Margaret demonstrated iterators that never exhausted:
import itertools
# Count forever
counter = itertools.count(start=1, step=1)
for i in counter:
print(i)
if i >= 5:
break
# Cycle through values infinitely
colors = itertools.cycle(['red', 'green', 'blue'])
for i, color in enumerate(colors):
print(color)
if i >= 7:
break
# red, green, blue, red, green, blue, red, green
Infinite iterators were safe because they produced values lazily—they only generated what was actually consumed.
The iter() Sentinel Pattern in Practice
Timothy saw a practical use of the sentinel form:
# Read file in chunks until empty
with open('large_catalog.txt', 'rb') as f:
# The lambda creates a no-argument callable that iter() calls repeatedly
# Each call executes f.read(4096), stopping when it returns b'' (the sentinel)
for chunk in iter(lambda: f.read(4096), b''):
process_chunk(chunk)
# Keep asking for input until user says "quit"
for user_input in iter(lambda: input("Enter book title (or 'quit'): "), "quit"):
add_book_to_catalog(user_input)
"The lambda," Margaret explained, "creates a no-argument callable that iter() can invoke repeatedly. Each call executes the function body, and when it returns the sentinel value, iteration stops. This pattern is perfect for reading until a condition is met."
Checking if Something is Iterable
Timothy learned to test for iteration support:
from collections.abc import Iterable, Iterator
books = ["1984", "Dune"]
book_iter = iter(books)
print(isinstance(books, Iterable)) # True
print(isinstance(books, Iterator)) # False - iterables aren't iterators
print(isinstance(book_iter, Iterable)) # True - iterators are iterable
print(isinstance(book_iter, Iterator)) # True
"All iterators are iterable," Margaret explained, "because they implement __iter__ (which returns themselves). But not all iterables are iterators—lists, for example, produce new iterators each time."
Timothy's Iterator Protocol Wisdom
Through mastering the Conveyor Belt Protocol, Timothy learned essential principles:
For loops use the iterator protocol: Every for loop calls iter() then repeatedly calls next().
Iterables have iter: They produce iterators when iter() is called.
Iterators have next: They produce the next value or raise StopIteration.
Iterators return self from iter: Because they're already positioned and stateful.
Iterables return new iterators: To allow independent, multiple iterations.
StopIteration signals exhaustion: This exception tells Python there are no more values.
Iterators are single-use: Once exhausted, they're done. Get a new iterator to iterate again.
Memory efficiency is the superpower: Iterators generate values on demand, not all upfront.
Generators are iterators: They implement the iterator protocol automatically.
Separate iterable from iterator: For multiple iterations, make the iterable produce new iterators.
Built-ins use iterators: range, enumerate, zip, and reversed all return iterators.
Manual control with next(): You can step through iterators yourself, mixing with for loops.
Check with isinstance(): Use collections.abc.Iterable and Iterator to test.
Timothy's exploration of the Iterator Protocol revealed the machinery beneath Python's most common construct—the for loop. The brass gears and leather belts of the conveyor system, once mysterious, now made perfect sense. Understanding the protocol meant he could create his own iterables, debug iteration problems, and appreciate how Python's lazy evaluation worked at the foundational level. The conveyor belts would deliver books one at a time, efficiently and elegantly, following a protocol centuries old yet perfectly modern.
Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.
Top comments (0)