Timothy understood how iterators worked, but Margaret's next challenge tested his knowledge: "The library needs a custom catalog browsing system. Patrons should view entries by decade, skip damaged records, and page through results fifty at a time. Build it using the Iterator Protocol."
Margaret led him to the workshop where specialized conveyor systems were assembled. "You know how iterators work," she said. "Now you'll build your own custom conveyors for specific library needs."
The Custom Iterator Need
Timothy's catalog needed specialized iteration that built-in types couldn't provide:
# Note: Examples use placeholder data structures and functions
# In practice, replace with your actual implementation
# The library's catalog - thousands of entries
catalog = [
{"title": "1984", "author": "Orwell", "year": 1949, "damaged": False},
{"title": "Dune", "author": "Herbert", "year": 1965, "damaged": False},
{"title": "Foundation", "author": "Asimov", "year": 1951, "damaged": True},
# ... thousands more
]
# Requirements:
# 1. Filter out damaged books
# 2. Group by decade
# 3. Page through results
# 4. Maintain position across queries
"Generic iteration won't work," Margaret explained. "You need custom iterators that embody your specific browsing logic."
Building a Filter Iterator
Timothy started with a simple filter iterator:
class DamagedBookFilter:
"""Iterator that skips damaged books"""
def __init__(self, books):
self.books = books
self.index = 0
def __iter__(self):
return self
def __next__(self):
# Skip damaged books
while self.index < len(self.books):
book = self.books[self.index]
self.index += 1
if not book.get('damaged', False):
return book
# No more undamaged books
raise StopIteration
# Use it
undamaged = DamagedBookFilter(catalog)
for book in undamaged:
print(f"{book['title']} - {book['year']}")
The iterator maintained position and filtering logic internally. Code using it didn't need to know about the damaged flag—the iterator handled that complexity.
The Pagination Iterator
Margaret showed Timothy how to implement pagination:
class PageIterator:
"""Iterator that yields pages of items"""
def __init__(self, items, page_size=50):
self.items = items
self.page_size = page_size
self.current_position = 0
def __iter__(self):
return self
def __next__(self):
# Check if we've reached the end
if self.current_position >= len(self.items):
raise StopIteration
# Get the current page
start = self.current_position
end = start + self.page_size
page = self.items[start:end]
# Move to next page
self.current_position = end
return page
# Use it
pages = PageIterator(catalog, page_size=50)
for page_number, page in enumerate(pages, start=1):
print(f"Page {page_number}: {len(page)} books")
for book in page[:3]: # Show first 3
print(f" - {book['title']}")
Each call to next() returned a complete page of results. The iterator managed the pagination math.
The Decade Grouping Iterator
Timothy built an iterator that grouped books by decade:
class DecadeIterator:
"""Iterator that yields books grouped by decade"""
def __init__(self, books):
# Sort books by year first
self.books = sorted(books, key=lambda b: b['year'])
self.index = 0
def __iter__(self):
return self
def __next__(self):
if self.index >= len(self.books):
raise StopIteration
# Get the decade of the current book
current_decade = (self.books[self.index]['year'] // 10) * 10
decade_books = []
# Collect all books from this decade
while (self.index < len(self.books) and
(self.books[self.index]['year'] // 10) * 10 == current_decade):
decade_books.append(self.books[self.index])
self.index += 1
return {
'decade': current_decade,
'books': decade_books
}
# Use it
by_decade = DecadeIterator(catalog)
for decade_group in by_decade:
print(f"{decade_group['decade']}s: {len(decade_group['books'])} books")
The iterator handled the grouping logic, yielding one decade at a time.
The Reverse Iterator
Margaret showed Timothy how to iterate backwards:
class ReverseIterator:
"""Iterator that yields items in reverse order"""
def __init__(self, items):
self.items = items
self.index = len(items) - 1
def __iter__(self):
return self
def __next__(self):
if self.index < 0:
raise StopIteration
item = self.items[self.index]
self.index -= 1
return item
# Use it - newest books first
reverse_catalog = ReverseIterator(catalog)
for book in reverse_catalog:
print(f"{book['title']} ({book['year']})")
"While Python has reversed()," Margaret noted, "building your own teaches the pattern. And sometimes you need custom reverse logic."
The Infinite Iterator Pattern
Timothy created an iterator that cycled through genres infinitely:
class GenreCycler:
"""Iterator that cycles through genres infinitely"""
def __init__(self, genres):
self.genres = genres
self.index = 0
def __iter__(self):
return self
def __next__(self):
# Never raises StopIteration - infinite!
genre = self.genres[self.index]
self.index = (self.index + 1) % len(self.genres)
return genre
# Use it with a limit
genres = GenreCycler(['Fiction', 'Non-Fiction', 'Reference'])
for i, genre in enumerate(genres):
print(genre)
if i >= 7:
break
# Fiction, Non-Fiction, Reference, Fiction, Non-Fiction, Reference, Fiction, Non-Fiction
The modulo operator ensured the index wrapped around, creating an infinite cycle.
Combining Multiple Iterators
Margaret demonstrated composing iterators:
class ChainedIterator:
"""Iterator that chains multiple iterators together"""
def __init__(self, *iterators):
self.iterators = list(iterators)
self.current_index = 0
def __iter__(self):
return self
def __next__(self):
# Try current iterator
while self.current_index < len(self.iterators):
try:
return next(self.iterators[self.current_index])
except StopIteration:
# Current iterator exhausted, move to next
self.current_index += 1
# All iterators exhausted
raise StopIteration
# Use it
classics = iter([{"title": "1984"}, {"title": "Brave New World"}])
scifi = iter([{"title": "Dune"}, {"title": "Foundation"}])
all_books = ChainedIterator(classics, scifi)
for book in all_books:
print(book['title'])
The chained iterator seamlessly moved from one source to the next.
"While this demonstrates the pattern," Margaret added, "Python's itertools.chain() provides this functionality efficiently. Build custom iterators when built-in tools don't fit your specific needs."
The Stateful Iterator
Timothy built an iterator that remembered query history:
class SearchHistoryIterator:
"""Iterator that tracks and limits search results"""
def __init__(self, books, query, max_results=10):
self.books = books
self.query = query.lower()
self.max_results = max_results
self.index = 0
self.results_returned = 0
self.search_history = []
def __iter__(self):
return self
def __next__(self):
if self.results_returned >= self.max_results:
raise StopIteration
while self.index < len(self.books):
book = self.books[self.index]
self.index += 1
if self.query in book['title'].lower():
self.results_returned += 1
self.search_history.append(book['title'])
return book
raise StopIteration
def get_history(self):
"""Access search history after iteration"""
return self.search_history.copy()
# Use it
search = SearchHistoryIterator(catalog, query="the", max_results=5)
for book in search:
print(book['title'])
print(f"Searched: {search.get_history()}")
The iterator maintained internal state beyond just position, tracking what it had returned.
The Lazy Loading Iterator
Margaret showed Timothy an iterator that loaded data on demand, but warned him about resource management:
class LazyFileIterator:
"""Iterator that reads file lines on demand
WARNING: This example demonstrates lazy loading but has a resource leak
if iteration stops early. In production, use context managers with 'with'
statements, or ensure proper cleanup with try/finally.
"""
def __init__(self, filename):
self.filename = filename
self.file_handle = None
def __iter__(self):
# Open file when iteration starts
self.file_handle = open(self.filename, 'r')
return self
def __next__(self):
if self.file_handle is None:
raise StopIteration
line = self.file_handle.readline()
if not line:
# End of file - clean up
self.close()
raise StopIteration
return line.strip()
def close(self):
"""Close the file handle - call this if you stop iterating early"""
if self.file_handle:
self.file_handle.close()
self.file_handle = None
def __del__(self):
"""Cleanup when object is destroyed"""
self.close()
# Use it - but remember to close if breaking early
iterator = LazyFileIterator('catalog.txt')
for line in iterator:
print(line)
if "stop condition" in line:
iterator.close() # Important! Clean up if stopping early
break
"This iterator," Margaret explained, "doesn't load the entire file into memory. It reads one line at a time, perfect for huge files. But notice the warning—if someone breaks out of the loop early, they should call close(). In production code, you'd use a context manager instead."
The Transformation Iterator
Timothy created an iterator that transformed items:
class UppercaseTitleIterator:
"""Iterator that yields books with uppercase titles"""
def __init__(self, books):
self.books = books
self.index = 0
def __iter__(self):
return self
def __next__(self):
if self.index >= len(self.books):
raise StopIteration
book = self.books[self.index].copy()
book['title'] = book['title'].upper()
self.index += 1
return book
# Use it
uppercase = UppercaseTitleIterator(catalog)
for book in uppercase:
print(book['title']) # All uppercase
The iterator modified data during iteration without changing the source.
When to Use Iterators vs Generators
Margaret compiled guidelines:
Use custom iterator classes when:
# You need to maintain complex state
class BookmarkableIterator:
def __init__(self, items):
self.items = items
self.index = 0
self.bookmarks = {}
def bookmark(self, name):
self.bookmarks[name] = self.index
def jump_to_bookmark(self, name):
self.index = self.bookmarks[name]
# ... __iter__ and __next__ methods
# You need public methods beyond iteration
# You want to expose internal state
Use generators when:
# Logic is simple and sequential
def undamaged_books(books):
for book in books:
if not book.get('damaged', False):
yield book
# You don't need to maintain complex state
# The iteration logic is straightforward
# You want concise, readable code
"Generators," Margaret noted, "are usually simpler. Use custom iterator classes when you need the full power of a class—state, methods, attributes."
Common Iterator Pitfalls
Margaret warned Timothy about common mistakes:
Pitfall 1: Forgetting to raise StopIteration
# BAD - infinite loop
def __next__(self):
if self.index < len(self.items):
item = self.items[self.index]
self.index += 1
return item
# Forgot to raise StopIteration!
return None # for loop never ends!
# GOOD
def __next__(self):
if self.index >= len(self.items):
raise StopIteration # Explicitly signal done
# ... return item
Pitfall 2: Returning self from iter for an iterable
# BAD - mixing iterable and iterator in one class
class BadCollection:
def __init__(self, items):
self.items = items
self.index = 0
def __iter__(self):
return self # Problem: can't iterate twice
def __next__(self):
if self.index >= len(self.items):
raise StopIteration
item = self.items[self.index]
self.index += 1
return item
# This causes bugs:
bad = BadCollection([1, 2, 3])
list1 = list(bad) # [1, 2, 3]
list2 = list(bad) # [] - Iterator exhausted! Bug!
# GOOD - separate iterable from iterator
class GoodCollection:
def __init__(self, items):
self.items = items
def __iter__(self):
return GoodCollectionIterator(self.items) # New iterator each time
class GoodCollectionIterator:
def __init__(self, items):
self.items = items
self.index = 0
def __iter__(self):
return self
def __next__(self):
if self.index >= len(self.items):
raise StopIteration
item = self.items[self.index]
self.index += 1
return item
# Now it works:
good = GoodCollection([1, 2, 3])
list1 = list(good) # [1, 2, 3]
list2 = list(good) # [1, 2, 3] - New iterator each time!
Pitfall 3: Modifying data during iteration
# BAD - modifying source during iteration
class DangerousIterator:
def __init__(self, items):
self.items = items
self.index = 0
def __next__(self):
if self.index >= len(self.items):
raise StopIteration
item = self.items[self.index]
self.items.pop(self.index) # Modifying source! Bad!
return item
# GOOD - iterate over copy or build new collection
class SafeIterator:
def __init__(self, items):
self.items = items
self.index = 0
def __next__(self):
if self.index >= len(self.items):
raise StopIteration
item = self.items[self.index]
self.index += 1
# Don't modify source - just read from it
return item
# Or if you need to build a modified collection:
class TransformIterator:
def __init__(self, items):
self.items = items
self.results = [] # Build separate results
self.index = 0
def __next__(self):
if self.index >= len(self.items):
raise StopIteration
item = self.items[self.index]
self.index += 1
transformed = process(item)
self.results.append(transformed) # Modify results, not source
return transformed
Timothy's Custom Iterator Wisdom
Through mastering the Custom Conveyor, Timothy learned essential principles:
Choose iterators for complex state: When you need bookmarks, history, or multiple methods, use classes.
Choose generators for simplicity: When iteration logic is straightforward, generators are cleaner.
Always raise StopIteration: This is how Python knows iteration is complete.
Separate iterable from iterator: Unless you want single-use iteration, create new iterators each time.
Don't modify during iteration: Changing the source data while iterating leads to bugs.
Handle resources properly: Close files, release locks, clean up in __del__ if needed.
Maintain clear state: Keep track of position and any other iteration state.
Handle edge cases: Empty collections, None values, invalid indices—plan for them.
Document the protocol: Make it clear how your iterator behaves.
Consider memory: Iterators should generate on demand, not load everything upfront.
Test exhaustion behavior: Ensure your iterator properly signals when done.
Use built-ins when available: Don't reinvent itertools.chain, filter, etc. unless you need custom behavior.
Timothy's mastery of custom iterators gave him the power to model any iteration pattern the library needed. The Custom Conveyor workshop produced specialized systems—filters, paginators, groupers, transformers—each perfectly suited to its task. The brass gears meshed smoothly, the leather belts moved steadily, and books flowed through the library exactly as needed, one at a time, following protocols Timothy now understood and could craft himself.
Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.
Top comments (0)