DEV Community: German Yamil

Python enumerate, zip, map, and filter: Loop Upgrades Every Beginner Needs

German Yamil — Sun, 24 May 2026 23:48:12 +0000

If you have been writing for i in range(len(items)): to track index positions, or manually zipping two lists with items[i] inside a loop, you are doing extra work that Python already solved. The builtins enumerate, zip, map, and filter exist precisely to replace those patterns — and once you internalize them, your loops become shorter, more readable, and less error-prone.

Why These Builtins Matter

Python's design philosophy favors expressive, intention-revealing code. The range(len(...)) pattern works, but it forces the reader to mentally reconstruct what you actually want: index plus value, or two lists in lockstep. These builtins make the intent explicit. They also tend to be faster than their manual equivalents because the iteration logic runs at the C layer, not in your Python loop body.

enumerate() — Never Write `i = 0; i += 1` Again

Before:

fruits = ["apple", "banana", "cherry"]
i = 0
for fruit in fruits:
    print(i, fruit)
    i += 1

After:

for i, fruit in enumerate(fruits):
    print(i, fruit)

enumerate wraps any iterable and yields (index, value) tuples. The start parameter lets you control where the count begins — useful when building 1-indexed output for users:

for n, fruit in enumerate(fruits, start=1):
    print(f"{n}. {fruit}")
# 1. apple
# 2. banana
# 3. cherry

Real use case: updating only specific items based on their position, or building a numbered display from a list without a separate counter variable polluting your scope.

zip() — Pair Two Lists Cleanly

Before:

titles = ["Chapter 1", "Chapter 2", "Chapter 3"]
slugs = ["ch-1", "ch-2", "ch-3"]
for i in range(len(titles)):
    print(titles[i], slugs[i])

After:

for title, slug in zip(titles, slugs):
    print(title, slug)

zip stops at the shortest iterable. If your lists can be unequal in length and you need to process all items, use itertools.zip_longest with a fillvalue:

from itertools import zip_longest
for title, slug in zip_longest(titles, slugs, fillvalue="MISSING"):
    print(title, slug)

Unzipping: pass a zipped sequence back through zip with the unpack operator to reverse the operation:

pairs = [("a", 1), ("b", 2), ("c", 3)]
letters, numbers = zip(*pairs)

map() — Apply a Function to Every Item

map(func, iterable) applies func to each element and returns a lazy iterator. It does not build a list until you ask for one.

Before:

prices = [9.99, 14.99, 4.99]
discounted = []
for p in prices:
    discounted.append(round(p * 0.9, 2))

After:

discounted = list(map(lambda p: round(p * 0.9, 2), prices))

Because map is lazy, it is memory-efficient for large sequences — values are computed on demand, not all at once.

When to prefer a list comprehension instead: if the transformation is complex or the lambda becomes hard to read at a glance, a comprehension is clearer:

# Prefer this for readability
discounted = [round(p * 0.9, 2) for p in prices]

Use map when you already have a named function to pass in — map(str, numbers) or map(slugify, titles) reads very naturally.

filter() — Keep Only Matching Items

filter(func, iterable) keeps elements where func returns truthy. Also lazy.

Before:

scores = [82, 45, 91, 67, 55, 78]
passing = []
for s in scores:
    if s >= 60:
        passing.append(s)

After:

passing = list(filter(lambda s: s >= 60, scores))

Again, if the condition is straightforward, a comprehension often wins on readability:

passing = [s for s in scores if s >= 60]

Use filter when the predicate is a named function you want to reuse, or when you are chaining it with map in a pipeline.

Combining Them: Real Chains

These builtins compose cleanly. Here is zip paired with enumerate to track position while iterating paired data:

titles = ["Intro", "Setup", "Usage"]
slugs = ["intro", "setup", "usage"]
for n, (title, slug) in enumerate(zip(titles, slugs), start=1):
    print(f"{n}. {title} → /posts/{slug}")

Here is a map + filter pipeline: normalize titles, then keep only the ones long enough to be meaningful:

raw = ["  hello world  ", "hi", "  python for beginners  ", "ok"]
cleaned = map(str.strip, raw)
meaningful = filter(lambda s: len(s) > 5, cleaned)
print(list(meaningful))
# ['hello world', 'python for beginners']

sorted() with key= — The Most Useful Builtin You Are Underusing

sorted deserves a mention here because its key parameter accepts any callable — which pairs naturally with the patterns above.

articles = [
    {"title": "Python Sets", "views": 1200},
    {"title": "Python zip", "views": 850},
    {"title": "Python map", "views": 3100},
]
by_views = sorted(articles, key=lambda a: a["views"], reverse=True)

You can pass operator.itemgetter("views") instead of the lambda if you prefer avoiding anonymous functions. The key= pattern is far more versatile than writing a custom comparison — it computes a sort value once per element, which is also more efficient.

any() and all() — Short-Circuit Evaluation for Collections

any(iterable) returns True the moment it finds a truthy element and stops. all(iterable) returns False the moment it finds a falsy element and stops. Both accept generators, so they stay lazy.

Before:

errors = ["", "missing slug", "", ""]
has_error = False
for e in errors:
    if e:
        has_error = True
        break

After:

has_error = any(errors)
all_valid = all(len(slug) > 0 for slug in slugs)

These are especially useful for validation checks in publishing pipelines — you want to know if any task failed or if all required fields are present before proceeding.

Real Pipeline Pattern

Here is how all of these come together in a practical scenario — managing an article publishing queue:

articles = [
    {"title": "Python Sets", "slug": "python-sets", "status": "published"},
    {"title": "Python zip", "slug": "python-zip", "status": "draft"},
    {"title": "Python map", "slug": "python-map", "status": "failed"},
    {"title": "Python filter", "slug": "python-filter", "status": "published"},
]

# Pair titles with slugs for published articles only
published = filter(lambda a: a["status"] == "published", articles)
paired = [(a["title"], a["slug"]) for a in published]

# Enumerate the queue for display
for n, (title, slug) in enumerate(paired, start=1):
    print(f"{n}. {title} → /posts/{slug}")

# Check if any tasks failed
failed = any(a["status"] == "failed" for a in articles)
if failed:
    failed_titles = [a["title"] for a in articles if a["status"] == "failed"]
    print(f"Failed: {failed_titles}")

Output:

1. Python Sets → /posts/python-sets
2. Python filter → /posts/python-filter
Failed: ['Python map']

This is the kind of pipeline that would take twice as many lines with manual index tracking and explicit append loops. Every builtin here does exactly one thing and composes cleanly with the others.

Quick Decision Reference

Pattern	Use builtin	Use comprehension
Index + value	`enumerate`	rarely needed
Two lists in sync	`zip`	rarely needed
Transform every item	`map(named_func, ...)`	`map(lambda ...)` — use comp instead
Keep matching items	`filter(named_func, ...)`	`filter(lambda ...)` — use comp instead
Sort by field	`sorted(key=...)`	not applicable
Any/all check	`any()` / `all()`	not applicable

The rule of thumb: if you need a lambda, a list comprehension is usually more readable. If you have a named function, map and filter read well. enumerate, zip, sorted, any, and all are almost always the right choice — no comprehension equivalent exists for them.

If you want to see how these builtins power a real automated publishing workflow, the full pipeline is documented in the AI Book Publishing Pipeline — including the queue manager, slug generator, and status checker built entirely with these patterns.

Python String Methods: The Complete Reference You'll Actually Use

German Yamil — Sat, 23 May 2026 22:08:40 +0000

Strings are everywhere in Python. Every CSV you parse, every API response you process, every user input you validate — it all comes back to text. Python ships with a rich set of built-in string methods, and knowing the right one for the job separates scripts that work from scripts that are a pleasure to maintain.

Strings Are Immutable — What That Means in Practice

Before diving into methods, one concept saves a lot of debugging time: strings in Python are immutable. Every method that appears to "change" a string actually returns a new string. The original is untouched.

name = "  alice  "
name.strip()   # returns "alice"
print(name)    # still "  alice  " — nothing changed in place

The fix is always to reassign:

name = name.strip()
print(name)    # "alice"

Keep this in mind as you work through the methods below. Every one of them follows the same rule.

Cleaning Input: strip(), lstrip(), rstrip()

The most common real-world string problem is extra whitespace. CSVs have trailing spaces. User inputs have newlines. LLM outputs sometimes start with a blank line. The strip family solves all of it.

raw = "  \n  hello world  \n  "

raw.strip()    # "hello world"
raw.lstrip()   # "hello world  \n  "   (left side only)
raw.rstrip()   # "  \n  hello world"   (right side only)

You can also strip specific characters — not just whitespace:

path = "///usr/local/bin///"
path.strip("/")    # "usr/local/bin"

csv_field = '"revenue"'
csv_field.strip('"')   # "revenue"

This is invaluable when cleaning CSV headers exported from spreadsheet software, which often wrap column names in quotation marks.

Case Conversion: upper(), lower(), title(), capitalize(), swapcase()

Case methods are simple but frequently misused. Here is what each one actually does:

text = "hello world from PYTHON"

text.upper()       # "HELLO WORLD FROM PYTHON"
text.lower()       # "hello world from python"
text.title()       # "Hello World From Python"
text.capitalize()  # "Hello world from python"  — only first char
text.swapcase()    # "HELLO WORLD FROM python"

title() is useful for formatting user-supplied names, but watch out — it treats any character after a space as a word boundary, including apostrophes:

"it's a test".title()   # "It'S A Test"  ← probably not what you want

For proper title casing with edge-case handling, consider a dedicated library or a custom function. For most automation work, title() is good enough.

Searching: find(), index(), startswith(), endswith(), in

When you need to know whether a substring exists or where it lives, Python gives you several tools with slightly different behavior:

line = "ERROR: disk quota exceeded"

"ERROR" in line          # True  — simplest check
line.startswith("ERROR") # True  — checks the beginning
line.endswith("exceeded") # True — checks the end

line.find("disk")        # 7   — returns index, or -1 if not found
line.index("disk")       # 7   — same, but raises ValueError if not found

Use in and startswith/endswith for boolean guards. Use find() when you need the position but missing is acceptable. Use index() when a missing substring is a genuine error you want to surface.

# Practical: classify log lines
def classify_log(line):
    if line.startswith("ERROR"):
        return "error"
    elif line.startswith("WARN"):
        return "warning"
    return "info"

startswith() and endswith() both accept a tuple of prefixes/suffixes, which saves nested or chains:

line.startswith(("ERROR", "CRITICAL", "FATAL"))

Splitting and Joining: split(), rsplit(), splitlines(), join()

Split and join are two sides of the same coin. split() breaks a string into a list; join() assembles a list back into a string.

row = "Alice,30,Engineer,New York"
fields = row.split(",")
# ["Alice", "30", "Engineer", "New York"]

# Limit the number of splits
"a:b:c:d".split(":", 2)   # ["a", "b", "c:d"]

# Split from the right
"a:b:c:d".rsplit(":", 1)  # ["a:b:c", "d"]  — useful for file extensions

# Split on newlines (handles \r\n, \n, \r automatically)
text = "line one\nline two\r\nline three"
text.splitlines()   # ["line one", "line two", "line three"]

join() is the correct way to build strings from lists — far more efficient than concatenation in a loop:

parts = ["usr", "local", "bin"]
"/".join(parts)     # "usr/local/bin"

words = ["Hello", "world"]
" ".join(words)     # "Hello world"

# Rebuild a CSV row
",".join(["Alice", "30", "Engineer"])   # "Alice,30,Engineer"

Replacing Text: replace() and translate()

replace() swaps every occurrence of a substring with another string:

text = "the cat sat on the cat mat"
text.replace("cat", "dog")          # "the dog sat on the dog mat"
text.replace("cat", "dog", 1)       # "the dog sat on the cat mat"  — limit to 1

For character-level bulk replacements, translate() paired with str.maketrans() is faster than chained replace() calls:

# Replace multiple characters at once
table = str.maketrans({"á": "a", "é": "e", "í": "i", "ó": "o", "ú": "u"})
"café résumé".translate(table)    # "cafe resume"

# Remove characters (map to None)
remove_table = str.maketrans("", "", "!?.,")
"Hello, world!".translate(remove_table)   # "Hello world"

This pattern is common when normalizing text before feeding it into a search index or slug generator.

Testing String Content: isdigit(), isalpha(), isalnum(), isspace()

The is* methods return True or False and are useful for validation:

"42".isdigit()      # True
"hello".isalpha()   # True
"h3llo".isalnum()   # True  — letters and digits only
"   ".isspace()     # True  — only whitespace
"HELLO".isupper()   # True
"hello".islower()   # True

A common use case: validating CSV fields before type conversion.

def safe_int(value):
    cleaned = value.strip()
    if cleaned.isdigit():
        return int(cleaned)
    return None

Note that isdigit() returns False for floats ("3.14") and negative numbers ("-5"). For those, a try/except around float() or int() is more reliable.

Padding and Alignment: zfill(), ljust(), rjust(), center()

These methods are useful for formatting fixed-width output — log files, CLI tables, zero-padded identifiers:

# Zero-padding numbers
"42".zfill(5)       # "00042"
"1234".zfill(3)     # "1234"  — never truncates

# Alignment with padding character (default: space)
"Alice".ljust(10)          # "Alice     "
"Alice".rjust(10)          # "     Alice"
"Alice".center(11)         # "   Alice   "
"Alice".center(11, "-")    # "---Alice---"

Building a quick formatted report without importing tabulate:

headers = ["Name", "Score", "Grade"]
rows = [("Alice", "95", "A"), ("Bob", "82", "B"), ("Carol", "91", "A")]

print("  ".join(h.ljust(8) for h in headers))
for row in rows:
    print("  ".join(cell.ljust(8) for cell in row))

Real Pipeline Patterns

Here is where string methods combine into something actually useful. Consider a script that reads a Markdown file with YAML frontmatter, cleans LLM-generated content, and builds URL slugs.

import re

def clean_llm_output(text: str) -> str:
    """Strip whitespace, normalize line endings, fix case artifacts."""
    text = text.strip()
    text = text.replace("\r\n", "\n").replace("\r", "\n")
    # Remove double blank lines
    while "\n\n\n" in text:
        text = text.replace("\n\n\n", "\n\n")
    return text

def parse_frontmatter(content: str) -> dict:
    """Extract key: value pairs from YAML frontmatter block."""
    meta = {}
    if not content.startswith("---"):
        return meta
    end = content.find("---", 3)
    if end == -1:
        return meta
    block = content[3:end].strip()
    for line in block.splitlines():
        if ": " in line:
            key, value = line.split(": ", 1)
            meta[key.strip()] = value.strip().strip('"')
    return meta

def build_slug(title: str) -> str:
    """Convert an article title to a URL-safe slug."""
    slug = title.lower()
    # Remove characters that aren't alphanumeric or spaces
    table = str.maketrans("", "", "!?.,:'\"")
    slug = slug.translate(table)
    slug = slug.replace(" ", "-")
    # Collapse multiple dashes
    while "--" in slug:
        slug = slug.replace("--", "-")
    return slug.strip("-")

# Example usage
title = "Python String Methods: The Complete Reference You'll Actually Use"
print(build_slug(title))
# "python-string-methods-the-complete-reference-youll-actually-use"

Each function above uses only the methods covered in this article. No regex required (except for the import that is there but unused — delete it if you like). Real automation scripts are mostly string operations stitched together.

If you want to see these patterns applied to a full publishing pipeline — parsing markdown, formatting output, and automating uploads — the full pipeline guide is at germy5.gumroad.com/l/xhxkzz (pay what you want, min $9.99).

Python datetime: Dates, Times, and Timezones Without the Confusion

German Yamil — Fri, 22 May 2026 22:00:03 +0000

If you've ever tried to add a week to today's date, parse a log timestamp, or figure out why two datetimes that look identical won't compare equal — welcome to Python's datetime module. It's powerful, it's in the standard library, and it's more confusing than it needs to be. This guide cuts straight to the 20% of features you'll actually use 80% of the time.

The Three Types: date, time, and datetime

The datetime module exports three main classes, and picking the right one matters.

from datetime import date, time, datetime

# date: year, month, day only — no time component
d = date(2025, 5, 6)
print(d)  # 2025-05-06

# time: hours, minutes, seconds — no date component
t = time(14, 30, 0)
print(t)  # 14:30:00

# datetime: both date AND time — what you'll use most
dt = datetime(2025, 5, 6, 14, 30, 0)
print(dt)  # 2025-05-06 14:30:00

When to use which:

date — scheduling events, birth dates, expiration dates, anything where the time of day is irrelevant
time — storing a recurring time (e.g., "run at 09:00 every day") independent of the date
datetime — log timestamps, scheduling with time precision, anything that combines date and time

In practice, you'll use datetime for 90% of automation work.

Creating Datetimes: now(), today(), and date.today()

from datetime import datetime, date

# Get the current date and time (local time, no timezone info)
now = datetime.now()
print(now)  # 2025-05-06 14:32:17.483921

# Get just today's date
today = date.today()
print(today)  # 2025-05-06

# datetime.today() works too, but datetime.now() is preferred
# (now() accepts a timezone argument; today() does not)

The difference between datetime.now() and datetime.today() is subtle and rarely matters for naive datetimes. Prefer datetime.now() because it can accept a tz argument when you need timezone awareness later.

strftime() — Format a Datetime to String

strftime stands for "string format time." You pass a format string with codes, and it returns a formatted string.

from datetime import datetime

now = datetime(2025, 5, 6, 14, 30, 5)

# ISO 8601 — machine-readable, great for logs and APIs
print(now.strftime("%Y-%m-%dT%H:%M:%S"))    # 2025-05-06T14:30:05

# Human-readable
print(now.strftime("%B %d, %Y at %I:%M %p"))  # May 06, 2025 at 02:30 PM

# Compact date for filenames
print(now.strftime("%Y%m%d"))               # 20250506

Common format codes:
| Code | Meaning | Example |
|------|---------|---------|
| %Y | 4-digit year | 2025 |
| %m | Month (01–12) | 05 |
| %d | Day (01–31) | 06 |
| %H | Hour 24h (00–23) | 14 |
| %M | Minutes (00–59) | 30 |
| %S | Seconds (00–59) | 05 |
| %B | Full month name | May |
| %I | Hour 12h (01–12) | 02 |
| %p | AM or PM | PM |

Tip: Use .isoformat() as a shortcut for ISO 8601 without memorizing the format string:

print(now.isoformat())  # 2025-05-06T14:30:05

strptime() — Parse a String into datetime

strptime is the reverse: string in, datetime out. The p stands for "parse."

from datetime import datetime

# Parse an ISO timestamp from a log file
raw = "2025-05-06T14:30:05"
dt = datetime.strptime(raw, "%Y-%m-%dT%H:%M:%S")
print(type(dt))   # <class 'datetime.datetime'>
print(dt.year)    # 2025

# Parse a human-readable date
raw2 = "May 06, 2025"
dt2 = datetime.strptime(raw2, "%B %d, %Y")
print(dt2)  # 2025-05-06 00:00:00

Common mistake: The format string must exactly match the input string — including separators. "2025/05/06" with the format "%Y-%m-%d" will raise a ValueError. Double-check your separators.

timedelta — Date Arithmetic

timedelta represents a duration. Add it to or subtract it from a datetime to get a new datetime.

from datetime import datetime, timedelta

now = datetime(2025, 5, 6, 14, 30, 0)

# Add 7 days
next_week = now + timedelta(days=7)
print(next_week)  # 2025-05-13 14:30:00

# Subtract 2 hours
two_hours_ago = now - timedelta(hours=2)
print(two_hours_ago)  # 2025-05-06 12:30:00

# Find the difference between two datetimes
start = datetime(2025, 5, 1, 9, 0, 0)
end = datetime(2025, 5, 6, 14, 30, 0)
delta = end - start
print(delta.days)          # 5
print(delta.total_seconds())  # 484200.0

timedelta accepts: weeks, days, hours, minutes, seconds, milliseconds, microseconds.

Comparing Datetimes — Sorting and Filtering

Datetimes support all comparison operators, which makes sorting and filtering straightforward.

from datetime import datetime

dt1 = datetime(2025, 5, 1)
dt2 = datetime(2025, 5, 6)

print(dt1 < dt2)   # True
print(dt1 == dt2)  # False

# Filter a list to events in May 2025
events = [
    datetime(2025, 4, 30),
    datetime(2025, 5, 3),
    datetime(2025, 5, 15),
    datetime(2025, 6, 1),
]

may_start = datetime(2025, 5, 1)
may_end = datetime(2025, 5, 31, 23, 59, 59)

may_events = [e for e in events if may_start <= e <= may_end]
print(may_events)
# [datetime.datetime(2025, 5, 3, 0, 0), datetime.datetime(2025, 5, 15, 0, 0)]

# Sort a list of datetimes
sorted_events = sorted(events)

Timezone-Aware Datetimes with zoneinfo (Python 3.9+)

This is where most beginners hit a wall. By default, datetime.now() returns a naive datetime — no timezone information attached. The moment your code runs on a server in a different timezone, or you compare timestamps from two sources, naive datetimes become a liability.

from datetime import datetime
from zoneinfo import ZoneInfo  # Python 3.9+

# Naive datetime (no timezone — avoid for production logs)
naive = datetime.now()
print(naive.tzinfo)  # None

# Timezone-aware datetime
aware = datetime.now(tz=ZoneInfo("America/New_York"))
print(aware)         # 2025-05-06 14:30:00-04:00
print(aware.tzinfo)  # America/New_York

# Convert between timezones
utc_dt = datetime.now(tz=ZoneInfo("UTC"))
eastern = utc_dt.astimezone(ZoneInfo("America/New_York"))
print(eastern)

The rule: Always use UTC for storage and logging. Convert to local time only for display. Never mix naive and aware datetimes — Python will raise a TypeError if you try to compare them.

# This raises TypeError — don't do this
naive = datetime(2025, 5, 6, 14, 30)
aware = datetime(2025, 5, 6, 14, 30, tzinfo=ZoneInfo("UTC"))
print(naive < aware)  # TypeError: can't compare offset-naive and offset-aware datetimes

Unix Timestamps: timestamp() and fromtimestamp()

A Unix timestamp is a float representing seconds since January 1, 1970 UTC. It's common in APIs, databases, and system logs.

from datetime import datetime
from zoneinfo import ZoneInfo

# datetime → Unix timestamp
dt = datetime(2025, 5, 6, 14, 30, 0, tzinfo=ZoneInfo("UTC"))
ts = dt.timestamp()
print(ts)   # 1746541800.0

# Unix timestamp → datetime (local timezone by default)
dt_back = datetime.fromtimestamp(ts)
print(dt_back)  # local time equivalent

# Better: specify UTC explicitly
dt_utc = datetime.fromtimestamp(ts, tz=ZoneInfo("UTC"))
print(dt_utc)  # 2025-05-06 14:30:00+00:00

Warning: datetime.fromtimestamp() without a tz argument uses your system's local timezone. Always pass tz=ZoneInfo("UTC") to get consistent results across environments.

Real Pipeline Pattern: Logging, Scheduling, and Time-Since-Last-Run

Here's how these pieces fit together in a real automation pipeline — logging timestamps, scheduling daily jobs, and calculating time since last run.

from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
import json
from pathlib import Path

STATE_FILE = Path("pipeline_state.json")
UTC = ZoneInfo("UTC")

def get_now() -> datetime:
    """Always return timezone-aware UTC datetime."""
    return datetime.now(tz=UTC)

def log_run(step: str) -> None:
    """Write a timestamped log entry."""
    ts = get_now().isoformat()
    print(f"[{ts}] {step}")

def save_last_run() -> None:
    """Persist the last run time to disk."""
    STATE_FILE.write_text(json.dumps({"last_run": get_now().isoformat()}))

def should_run_today() -> bool:
    """Return True if the pipeline hasn't run in the last 23 hours."""
    if not STATE_FILE.exists():
        return True
    state = json.loads(STATE_FILE.read_text())
    last_run = datetime.fromisoformat(state["last_run"])
    elapsed = get_now() - last_run
    return elapsed > timedelta(hours=23)

# Usage
if should_run_today():
    log_run("Starting pipeline")
    # ... do work ...
    save_last_run()
    log_run("Pipeline complete")
else:
    log_run("Skipping — already ran today")

Key patterns here:

datetime.fromisoformat() parses the ISO strings you saved with .isoformat() — clean round-trip
Storing last_run as an ISO string in JSON means it's human-readable and portable
All datetimes are UTC-aware so the comparison get_now() - last_run is always valid

If you're building automation pipelines in Python and want a complete publishing workflow — from generating content to scheduling posts and tracking runs — my pipeline guide covers the full stack: germy5.gumroad.com/l/xhxkzz (pay what you want, min $9.99).

Python Regex: re Module Patterns That Actually Make Sense

German Yamil — Fri, 22 May 2026 00:00:30 +0000

Regular expressions intimidate most beginners — and honestly, that's fair. A pattern like r'(?P<version>\d+\.\d+\.\d+)' looks like line noise until you know what each piece does. But regex is one of those tools that unlocks whole categories of automation work: log parsing, slug validation, text cleanup at scale. This article walks through the Python re module from the ground up, with concrete examples you'd actually write in a script.

Why Regex Exists (and When to Reach for It)

Python's string methods — str.find(), str.replace(), str.split() — cover a huge range of tasks. But they break down the moment the pattern isn't fixed.

# str methods: works fine for exact substrings
log_line = "2024-03-15 ERROR app crashed"
if "ERROR" in log_line:
    print("Found error")

# str methods: breaks for variable patterns
# How do you extract ANY date in YYYY-MM-DD format?
# You'd need split + indexing + validation — fragile and verbose

Regex solves this by letting you describe the shape of what you're looking for, not just the exact text. One pattern handles thousands of variants automatically.

re.search() vs re.match() vs re.fullmatch()

This is the most common source of confusion with the re module. All three find patterns — but they look in different places.

re.match() — only checks at the start of the string
re.search() — scans the entire string, returns first match
re.fullmatch() — requires the pattern to match the entire string

import re

text = "App version 2.1.0 deployed"

# match() fails — "version" isn't at position 0
print(re.match(r'\d+\.\d+\.\d+', text))   # None

# search() finds it anywhere
m = re.search(r'\d+\.\d+\.\d+', text)
print(m.group())  # '2.1.0'

# fullmatch() fails — text is more than just a version number
print(re.fullmatch(r'\d+\.\d+\.\d+', text))  # None
print(re.fullmatch(r'\d+\.\d+\.\d+', "2.1.0"))  # Match object

Rule of thumb: use re.search() by default. Switch to re.match() when you know the pattern starts at position zero, and re.fullmatch() for input validation.

re.findall() and re.finditer()

When you need all matches — not just the first — use re.findall() for a list or re.finditer() for an iterator of match objects.

log = """
ERROR 2024-03-15 disk full
INFO  2024-03-16 backup started
ERROR 2024-03-17 network timeout
"""

# findall returns a list of strings
errors = re.findall(r'\d{4}-\d{2}-\d{2}', log)
print(errors)  # ['2024-03-15', '2024-03-16', '2024-03-17']

# finditer returns match objects — useful when you need position info
for m in re.finditer(r'ERROR (\d{4}-\d{2}-\d{2})', log):
    print(f"Error on {m.group(1)} at position {m.start()}")

Use finditer() when you're processing large text and want to avoid building a full list in memory, or when you need .start(), .end(), or .span() metadata.

Basic Patterns: The Building Blocks

You don't need to memorize every metacharacter. These ten cover 80% of real-world use cases:

Pattern	Matches
`.`	Any character except newline
`\d`	Any digit (0–9)
`\w`	Word character (letters, digits, underscore)
`\s`	Whitespace (space, tab, newline)
`*`	Zero or more of the preceding
`+`	One or more of the preceding
`?`	Zero or one (makes preceding optional)
`^`	Start of string (or line with MULTILINE)
`$`	End of string (or line with MULTILINE)
`{n,m}`	Between n and m repetitions

# Matching a log timestamp: YYYY-MM-DD HH:MM:SS
pattern = r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}'
re.search(pattern, "Event at 2024-03-15 14:22:01 UTC")

Always use raw strings (r'...') for regex patterns. Without the r, Python processes backslashes first, which causes subtle bugs.

Character Classes [] and Negation [^]

Square brackets define a set of characters where any one character can match. A caret inside the brackets negates the set.

# Match any vowel
re.findall(r'[aeiou]', "python regex")  # ['o', 'e', 'e']

# Match any character that is NOT a digit
re.findall(r'[^\d]', "abc123")  # ['a', 'b', 'c']

# Match a slug: lowercase letters, digits, hyphens
slug_pattern = r'[a-z0-9-]+'
re.fullmatch(slug_pattern, "python-regex-guide")  # Match
re.fullmatch(slug_pattern, "Python Regex Guide")  # None

Character classes are especially useful for cleaning user input or validating slugs, usernames, and similar constrained strings.

Groups () and Named Groups (?P...)

Parentheses group part of a pattern and let you extract it with .group() or re.findall(). Named groups make your code self-documenting.

# Unnamed groups — accessed by index
m = re.search(r'(\d{4})-(\d{2})-(\d{2})', "Date: 2024-03-15")
print(m.group(0))  # '2024-03-15' (full match)
print(m.group(1))  # '2024'
print(m.group(2))  # '03'

# Named groups — accessed by name
m = re.search(
    r'(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})',
    "Date: 2024-03-15"
)
print(m.group("year"))   # '2024'
print(m.group("month"))  # '03'
print(m.groupdict())     # {'year': '2024', 'month': '03', 'day': '15'}

Named groups pay off quickly when you're returning data from a parsing function — the dictionary is immediately usable without counting group indices.

re.sub() — Substitution and Replacement

re.sub(pattern, replacement, string) is the regex-powered str.replace(). The replacement can reference groups with \1, \2, or named groups with \g<name>.

# Normalize date separators
text = "Dates: 2024/03/15 and 2024.04.20"
normalized = re.sub(r'(\d{4})[/.](\d{2})[/.](\d{2})', r'\1-\2-\3', text)
print(normalized)  # 'Dates: 2024-03-15 and 2024-04-20'

# Redact email addresses in log output
log = "User user@example.com logged in from 192.168.1.1"
safe_log = re.sub(r'[\w.+-]+@[\w-]+\.\w+', '[REDACTED]', log)
print(safe_log)  # 'User [REDACTED] logged in from 192.168.1.1'

# re.sub with a function — dynamic replacements
def double_number(m):
    return str(int(m.group()) * 2)

re.sub(r'\d+', double_number, "port 80 and port 443")
# 'port 160 and port 886'

Compiled Patterns with re.compile()

When you use the same pattern repeatedly — inside a loop, across many files — compile it once and reuse the object.

# Without compile — pattern is re-parsed every iteration
for line in log_lines:
    if re.search(r'\d{4}-\d{2}-\d{2}', line):
        process(line)

# With compile — pattern parsed once
DATE_RE = re.compile(r'\d{4}-\d{2}-\d{2}')
for line in log_lines:
    if DATE_RE.search(line):
        process(line)

The compiled object has the same methods: .search(), .match(), .findall(), .sub(). The performance difference matters at scale; for a one-off match it's negligible. A good pattern is to define compiled regexes at module level as constants.

Flags: IGNORECASE, MULTILINE, DOTALL

Flags modify how patterns interpret the text.

# re.IGNORECASE — case-insensitive matching
re.findall(r'error', "Error: FATAL ERROR occurred", re.IGNORECASE)
# ['Error', 'ERROR']

# re.MULTILINE — ^ and $ match start/end of each LINE
text = "first line\nsecond line\nthird line"
re.findall(r'^\w+', text, re.MULTILINE)
# ['first', 'second', 'third']

# re.DOTALL — . matches newline too
html = "<div>\nsome text\n</div>"
re.search(r'<div>(.*)</div>', html, re.DOTALL).group(1)
# '\nsome text\n'

# Combine flags with |
re.findall(r'^error', text, re.IGNORECASE | re.MULTILINE)

You can also embed flags inline in the pattern with (?i), (?m), (?s) — useful when passing patterns as strings to functions that don't expose a flags argument.

Real Pipeline Patterns

Here's where everything comes together. These are patterns you'd use in actual automation scripts.

Parse Dev.to article slugs from a markdown file:

# Before: str methods — brittle, breaks on query params or anchors
url = "https://dev.to/username/my-article-slug-abc123"
slug = url.split("/")[-1]  # fragile if URL has trailing slash or params

# After: regex — handles variations cleanly
DEVTO_RE = re.compile(r'dev\.to/[\w-]+/([\w-]+)')
m = DEVTO_RE.search(url)
slug = m.group(1) if m else None  # 'my-article-slug-abc123'

Validate email format:

EMAIL_RE = re.compile(r'^[\w.+-]+@[\w-]+\.[a-zA-Z]{2,}$')

def is_valid_email(email):
    return bool(EMAIL_RE.fullmatch(email.strip()))

print(is_valid_email("user@example.com"))    # True
print(is_valid_email("not-an-email"))        # False
print(is_valid_email("user@.com"))           # False

Extract version numbers from deployment logs:

VERSION_RE = re.compile(r'v?(?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)')

def extract_versions(log_text):
    return [m.groupdict() for m in VERSION_RE.finditer(log_text)]

log = "Deployed v2.1.0, rollback from 1.9.12 available"
print(extract_versions(log))
# [{'major': '2', 'minor': '1', 'patch': '0'},
#  {'major': '1', 'minor': '9', 'patch': '12'}]

These patterns are reusable, testable, and readable — especially with named groups and compiled constants.

If you're building automation scripts and pipelines in Python, the full pipeline guide at germy5.gumroad.com/l/xhxkzz covers how to chain regex parsing with file I/O, API calls, and structured output in a complete workflow.

How to Turn Your Python Scripts Into a Side Income Stream

German Yamil — Tue, 19 May 2026 23:50:05 +0000

You've written Python scripts that solve real problems. Someone else has the same problem and would pay to have it solved.

That gap is where side income lives.

Here's the playbook — from script to product to sale.

What Makes a Python Script Worth Selling

Not every script is a product. The ones that sell share a few properties:

Saves meaningful time. Scripts that save 30 minutes a day are worth $30-50/month to the right buyer. Scripts that save 2 hours a week are worth $100+/month.

Solves a recurring problem. One-time scripts are curiosities. Scripts that run daily or weekly are infrastructure.

Works without deep Python knowledge. Buyers aren't always developers. A script with a clean CLI or GUI reaches more buyers than one that requires editing source code.

Three Packaging Models

1. Sell the Script (One-time, $10-50)

The simplest model. Package your script as a download with documentation.

product/
├── main.py          # The script
├── README.md        # Setup + usage
├── requirements.txt # Dependencies
└── examples/        # Sample inputs/outputs

Best for: utility scripts, data processing tools, automation helpers.

2. Sell Documentation (One-time, $15-30)

Sometimes the value isn't the code — it's knowing how to build the system. A detailed guide with working examples sells to developers who want to learn, not just copy.

This is the model I use. The Python AI Publishing Pipeline sells the architecture and scripts behind a content automation system. Buyers aren't just getting code — they're getting a system design they can adapt.

3. Sell Access (Recurring, $5-20/month)

Wrap your script in a simple web API or SaaS interface. Higher build cost, recurring revenue.

Best for: scripts with ongoing maintenance, data that changes, or high-frequency use cases.

Finding Buyers

The order matters:

1. Communities first. Post in subreddits where your target buyers hang out:

r/learnpython, r/Python, r/programming (developers)
r/freelance, r/entrepreneur (business buyers)
Topic-specific subs (r/SEO for content tools, r/datascience for data scripts)

2. Dev.to and Hashnode. Write about the problem your script solves. Include the product at the end. This is the content marketing play — slower, but compounds.

3. X (Twitter) and LinkedIn. Build-in-public posts showing the script working get attention. Short videos of the output convert better than text.

What Actually Converts

After 45 articles and tracking what readers click:

Works well:

Showing real output (actual terminal output, actual numbers)
Before/after comparisons ("3 hours manually → 4 seconds automated")
Honest failure stories ("here's where my first version broke")

Works poorly:

Generic claims ("this will save you tons of time")
Long preambles before the actual code
Paywalled demos

The pattern that converts: solve a small problem in the article for free, then offer the complete system for $20.

A Python Script Worth Selling: Real Example

Here's the core of my publish queue — the script that runs every day and handles article publishing, cross-posting, and link patching:

def cmd_publish_next():
    q = load_queue()
    if not q['pending']:
        return  # Nothing to publish today

    next_item = q['pending'][0]
    filepath = os.path.join(MARKETING_DIR, next_item['filename'])

    # Publish to Dev.to
    result = publish_article(filepath)
    url = f"https://dev.to{result['path']}"

    # Cross-post everywhere
    crosspost(url, next_item['filename'])      # Hashnode
    attach_cover(result['id'], next_item['filename'])  # Cover image
    patch_stale_links(url, next_item['filename'])      # Fix old articles

    # Update queue state
    q['pending'].pop(0)
    q['published'].append({
        "filename": next_item['filename'],
        "date": str(date.today()),
        "url": url,
        "id": result['id'],
    })
    save_queue(q)

This runs at 10am every day. I haven't touched it in weeks. It's published 45 articles.

That's the kind of system worth productizing.

The Math

Price:     $9.99/sale
Platform:  Gumroad (~10% fee) → ~$18/sale
Goal:      $500/month passive
Sales needed: 28 sales/month (~1/day)

Traffic needed (at 1% conversion): 2,800 visitors/month
Articles needed (at 60 views/article): ~47 articles
Time to get there: ~6 months of daily publishing

It's not fast. But it's automatic once the system is running.

Starting This Week

Pick the script. What have you built that saves you time? Start there.
Document it. Write a README that a non-Python developer could follow.
Price it. Start at $15-20 for scripts, $19-29 for guides with architecture.
List it. Gumroad is the easiest — free account, 10% transaction fee, instant setup.
Write one article. Show the problem the script solves. Add the Gumroad link at the end.

The first sale will take longer than you expect. The tenth sale will come faster than you think.

Get the Complete System

The Python AI Publishing Pipeline — the scripts, architecture, and setup guide behind this content system.

📋 Free: AI Publishing Checklist — 7 steps to ship a Python ebook, PDF, no email required.

🚀 Full pipeline: germy5.gumroad.com/l/xhxkzz — $9.99, 30-day money-back guarantee.

If this was useful, the ❤️ button helps other developers find it.

Building a Python content pipeline? I sell the complete automation system as a one-time download — Dev.to API, Claude API, launchd, Gumroad. Check it out ($9.99)

Python functools: lru_cache, partial, reduce, and wraps

German Yamil — Tue, 19 May 2026 01:36:22 +0000

The functools module is one of Python's most practical standard-library gems. It ships a small set of higher-order function utilities — tools that operate on or return other functions — that eliminate boilerplate, improve performance, and make your code easier to reason about. You don't need to install anything; import functools is enough.

lru_cache — Stop Paying Twice for the Same Result

lru_cache (Least Recently Used cache) wraps a function and remembers its return values. Call it with the same arguments twice, and the second call returns the stored result instantly.

import functools
import time

@functools.lru_cache(maxsize=128)
def fetch_article_ids(api_key: str) -> tuple[str, ...]:
    """Fetch already-published article IDs from the Dev.to API."""
    time.sleep(1)          # simulates a real HTTP call
    return ("abc123", "def456", "ghi789")

# First call: hits the network
ids = fetch_article_ids("my-api-key")   # ~1 s

# Second call: instant — result came from cache
ids = fetch_article_ids("my-api-key")   # 0 ms

maxsize=128 keeps up to 128 distinct argument combinations. Set it to None (or use @functools.cache, available since Python 3.9) for an unbounded cache.

Inspect the cache:

print(fetch_article_ids.cache_info())
# CacheInfo(hits=1, misses=1, maxsize=128, currsize=1)

When NOT to cache: arguments must be hashable (no lists, dicts). If your function receives mutable input or has side effects you need every time (sending an email, writing a file), skip the cache.

Real pipeline example: In a publishing automation script, fetch_article_ids is called once at startup and potentially again inside a loop. With @lru_cache, the HTTP round-trip happens exactly once per process run regardless of how many times the function is called.

partial — Freeze Some Arguments Up Front

functools.partial creates a new callable with some arguments pre-filled. Think of it as a lightweight factory that adapts a general function to a specific context.

import functools
import requests

# General function
def api_get(url: str, headers: dict, timeout: int = 10):
    return requests.get(url, headers=headers, timeout=timeout)

# Pre-configured version with auth headers frozen in
devto_get = functools.partial(
    api_get,
    headers={"api-key": "my-secret-key", "Accept": "application/json"},
    timeout=15,
)

# Now every call needs only the URL
response = devto_get("https://dev.to/api/articles/me/published")

Before partial you would either duplicate the headers dict everywhere or write a wrapper function. partial gives you a clean, named callable with zero extra code.

Another classic use: adapting a two-argument function for sorted or map:

from functools import partial

def multiply(x, factor):
    return x * factor

double = partial(multiply, factor=2)
print(list(map(double, [1, 2, 3, 4])))   # [2, 4, 6, 8]

wraps — Every Decorator Must Use This

When you write a decorator, Python replaces the original function with your wrapper. Without wraps, tools like help(), inspect.signature(), and logging lose the original name and docstring.

Without wraps — broken:

def log_call(func):
    def wrapper(*args, **kwargs):
        print(f"Calling something...")
        return func(*args, **kwargs)
    return wrapper

@log_call
def publish_article(title: str) -> bool:
    """Publish an article and return True on success."""
    ...

print(publish_article.__name__)   # "wrapper"  ← wrong
print(publish_article.__doc__)    # None        ← wrong

With wraps — correct:

import functools

def log_call(func):
    @functools.wraps(func)          # ← one line fixes everything
    def wrapper(*args, **kwargs):
        print(f"Calling {func.__name__}...")
        return func(*args, **kwargs)
    return wrapper

@log_call
def publish_article(title: str) -> bool:
    """Publish an article and return True on success."""
    ...

print(publish_article.__name__)   # "publish_article"  ✓
print(publish_article.__doc__)    # "Publish an article..."  ✓

wraps copies __name__, __qualname__, __doc__, __dict__, __module__, and __wrapped__. It costs one decorator line and has no downsides — use it every time you write a wrapper.

reduce — Fold a Sequence to a Single Value

functools.reduce applies a two-argument function cumulatively across an iterable, collapsing it to a single value.

from functools import reduce

numbers = [1, 2, 3, 4, 5]
total = reduce(lambda acc, x: acc + x, numbers)   # 15

It is less common now that sum(), max(), min(), and comprehensions cover most everyday cases. But reduce shines when you need to chain function calls dynamically:

from functools import reduce

def apply_pipeline(value, transforms):
    """Apply a list of functions to a value in sequence."""
    return reduce(lambda v, fn: fn(v), transforms, value)

pipeline = [str.strip, str.lower, lambda s: s.replace(" ", "-")]
slug = apply_pipeline("  Hello World  ", pipeline)
print(slug)   # "hello-world"

This pattern — a list of single-argument functions collapsed with reduce — is a clean way to build configurable text-processing pipelines without nested calls or temporary variables.

total_ordering — Define Two Comparisons, Get Six

If you have a class that needs ordering (e.g., articles sorted by publish date), you normally must implement __eq__, __lt__, __le__, __gt__, and __ge__. total_ordering lets you define just __eq__ and one of the four inequality methods, and fills in the rest automatically.

from functools import total_ordering

@total_ordering
class Article:
    def __init__(self, title: str, word_count: int):
        self.title = title
        self.word_count = word_count

    def __eq__(self, other):
        return self.word_count == other.word_count

    def __lt__(self, other):
        return self.word_count < other.word_count

a = Article("Short post", 300)
b = Article("Deep dive", 1200)

print(a < b)    # True
print(a >= b)   # False  ← generated automatically
print(a <= b)   # True   ← generated automatically

The trade-off: auto-generated methods are slightly slower than hand-written ones. For classes compared millions of times in tight loops, implement all six manually. For most domain objects, total_ordering is fine.

cached_property — Compute Once Per Instance

functools.cached_property (Python 3.8+) works like @property but stores the result on the instance after the first access. Subsequent accesses read from __dict__ directly, bypassing the descriptor entirely.

import functools

class ArticleStats:
    def __init__(self, words: list[str]):
        self.words = words

    @functools.cached_property
    def word_count(self) -> int:
        print("Computing word count...")
        return len(self.words)

stats = ArticleStats("the quick brown fox".split())
print(stats.word_count)   # Computing word count... → 4
print(stats.word_count)   # 4  (no print — cached on instance)

Ideal for expensive computations derived from instance data: parsing, aggregation, expensive lookups. Unlike lru_cache on a method, cached_property is per-instance and cleared when the instance is garbage-collected — no unbounded global cache growth.

Which Ones You'll Use Every Day

Tool	Frequency	Reason
`lru_cache` / `cache`	Very high	Eliminate redundant I/O instantly
`wraps`	Very high	Required in every decorator
`partial`	High	Pre-configure functions for specific contexts
`cached_property`	Medium	Lazy, once-per-instance properties
`total_ordering`	Medium	Clean comparison support on data classes
`reduce`	Low-niche	Useful for dynamic pipelines; otherwise use builtins

Start with lru_cache and wraps — they pay off immediately and show up in production codebases constantly. Add partial when you find yourself repeating the same keyword arguments. Reach for reduce only when you genuinely have a list of functions to chain.

If you're building automation pipelines in Python and want a complete end-to-end example — from script structure to publishing — check out the full pipeline guide at germy5.gumroad.com/l/xhxkzz.

Python itertools: Efficient Loops Without the Boilerplate

German Yamil — Mon, 18 May 2026 05:35:16 +0000

The Python standard library ships with a module that most beginners skip right past: itertools. Once you start processing real data — CSV exports, API responses, log files, ebook chapters — you will reach for it constantly. The core idea is lazy iteration: instead of building a giant list in memory, itertools functions produce one item at a time, on demand.

What itertools Gives You

itertools is built into Python — no pip install needed. Every function in it returns an iterator, not a list. That means:

Memory stays flat no matter how large the input is.
You can chain operations together without intermediate allocations.
It composes cleanly with generators, map, and filter.

Import it once at the top of your script:

import itertools

chain() — Concatenate Iterables Without Building a List

Suppose you have multiple lists of chapter titles pulled from different sources and you want to process them in one pass.

Before:

all_titles = chapter_titles + bonus_titles + appendix_titles
for title in all_titles:
    process(title)

After:

for title in itertools.chain(chapter_titles, bonus_titles, appendix_titles):
    process(title)

The chain version never builds a combined list. If each source has thousands of items, that matters. You can also flatten a list-of-lists with chain.from_iterable:

nested = [["ch1", "ch2"], ["bonus1"], ["app1", "app2"]]
for title in itertools.chain.from_iterable(nested):
    print(title)

islice() — Take the First N Items from Any Iterable

islice is the slice operator for iterators. Generators do not support [0:10] syntax, but islice handles them cleanly.

import itertools

def load_rows(filename):
    with open(filename) as f:
        for line in f:
            yield line.strip()

# Preview first 5 rows without loading the whole file
for row in itertools.islice(load_rows("data.csv"), 5):
    print(row)

You can also skip rows with a start offset: islice(iterable, start, stop).

batched() — Process in Chunks of N

Added in Python 3.12, itertools.batched splits an iterable into fixed-size chunks. This is the pattern for rate-limited API calls, database batch inserts, or processing ebook pages in groups.

# Python 3.12+
from itertools import batched

results = range(100)
for batch in batched(results, 10):
    print(batch)  # tuple of up to 10 items

For Python 3.11 and earlier, implement the same pattern manually:

def batched_compat(iterable, n):
    it = iter(iterable)
    while chunk := list(itertools.islice(it, n)):
        yield chunk

Both versions yield tuples (or lists) of at most n items, with the final chunk smaller if the total does not divide evenly.

groupby() — Group Sorted Data by a Key

groupby scans an iterable and emits (key, group_iterator) pairs. The critical rule: the input must be sorted by the same key first, otherwise groups will be fragmented.

import itertools, csv

rows = [
    {"category": "python", "title": "Generators"},
    {"category": "python", "title": "Decorators"},
    {"category": "bash",   "title": "Cron Jobs"},
    {"category": "bash",   "title": "Pipelines"},
]

rows.sort(key=lambda r: r["category"])

for category, group in itertools.groupby(rows, key=lambda r: r["category"]):
    titles = [r["title"] for r in group]
    print(f"{category}: {titles}")

Output:

bash: ['Cron Jobs', 'Pipelines']
python: ['Generators', 'Decorators']

This pattern maps directly onto grouping log lines by severity, CSV rows by date, or API records by status code.

product() — Cartesian Product Replaces Nested Loops

Nested for loops scanning a grid of options are a common source of hard-to-read code. product collapses them:

Before:

for size in ["small", "medium", "large"]:
    for color in ["red", "blue"]:
        for format_ in ["epub", "pdf"]:
            generate(size, color, format_)

After:

for size, color, format_ in itertools.product(
    ["small", "medium", "large"],
    ["red", "blue"],
    ["epub", "pdf"],
):
    generate(size, color, format_)

Same output, one indentation level, easier to extend. The repeat parameter lets you take the cartesian power of a single iterable: product("ABC", repeat=2) produces all two-letter pairs.

combinations() and permutations()

When you need to test all pairs from a set of values, or all ordered arrangements:

tags = ["python", "automation", "publishing"]

# All 2-tag pairs (order does not matter)
for pair in itertools.combinations(tags, 2):
    print(pair)
# ('python', 'automation'), ('python', 'publishing'), ('automation', 'publishing')

# All ordered arrangements of 2 tags
for pair in itertools.permutations(tags, 2):
    print(pair)
# ('python', 'automation'), ('python', 'publishing'), ('automation', 'python'), ...

combinations_with_replacement allows repeated elements. These are useful for generating test cases, keyword combinations for SEO experiments, or cover layout variants.

repeat() and cycle() — Generating Test Data

repeat(value, n) yields the same value n times (or infinitely if n is omitted). cycle(iterable) loops through a sequence forever. Both are useful for pairing a static value with a stream:

# Tag every row with the same pipeline version
rows = ["row1", "row2", "row3"]
for row, version in zip(rows, itertools.repeat("v4.0")):
    print(row, version)

# Rotate through cover styles when generating previews
styles = itertools.cycle(["dark", "light", "minimal"])
for i in range(9):
    print(f"Cover {i}: {next(styles)}")

Combine repeat with chain to inject a separator between sections of output without building an intermediate list.

Real Pipeline Pattern: Batching 100 API Results

Here is how these tools combine in a realistic script — fetching 100 records from an API and processing them 10 at a time to stay under rate limits:

import itertools
import time

def fetch_all_records():
    """Generator that yields all 100 records lazily."""
    for i in range(100):
        yield {"id": i, "title": f"Chapter {i}"}

def process_batch(batch):
    """Send a batch to an external API or write to disk."""
    ids = [r["id"] for r in batch]
    print(f"Processing batch: {ids}")
    time.sleep(0.1)  # simulate rate limit pause

# Python 3.12+
from itertools import batched
for batch in batched(fetch_all_records(), 10):
    process_batch(batch)

# Python 3.11 and earlier
def batched_compat(iterable, n):
    it = iter(iterable)
    while chunk := list(itertools.islice(it, n)):
        yield chunk

for batch in batched_compat(fetch_all_records(), 10):
    process_batch(batch)

At no point does this script hold all 100 records in memory. The generator, the batch wrapper, and the processing function form a lazy pipeline — exactly the pattern used in the full ebook publishing pipeline where thousands of metadata rows flow through validation, transformation, and upload stages.

If you are building a document or content pipeline and want to apply this batch-processing approach at scale — including API calls, file generation, and KDP/Gumroad uploads — the full pipeline guide covers every stage end-to-end: germy5.gumroad.com/l/xhxkzz.

Python collections: defaultdict, Counter, deque, and namedtuple

German Yamil — Fri, 15 May 2026 22:00:01 +0000

Python ships with a built-in collections module that fixes the most common pain points with plain dicts and lists. Most beginners reach for a regular dict or list out of habit — and then write ten extra lines of boilerplate to work around their limitations. This article walks you through the six most useful types in collections, with clear before/after examples so you know exactly when to swap.

Why the `collections` module exists

Plain dicts raise KeyError when you access a missing key. Plain lists are slow when you need to insert or remove from the front. Tuples have no field names, so point[0] tells you nothing. The collections module exists to solve these exact problems without requiring you to write a class or import a heavy library.

from collections import defaultdict, Counter, deque, namedtuple, OrderedDict, ChainMap

That single import unlocks six specialized containers, each a drop-in replacement for a plain dict or list in the right situation.

`defaultdict` — no more "if key not in dict"

Every time you try to group or count with a plain dict, you write the same defensive check:

# Before: plain dict, grouping articles by tag
articles_by_tag = {}
for article, tag in data:
    if tag not in articles_by_tag:
        articles_by_tag[tag] = []
    articles_by_tag[tag].append(article)

defaultdict replaces those three lines with one. You pass a factory function — list, int, set — and any missing key is automatically initialized with that factory's return value:

# After: defaultdict
from collections import defaultdict

articles_by_tag = defaultdict(list)
for article, tag in data:
    articles_by_tag[tag].append(article)  # no KeyError, ever

Use defaultdict(int) for counting, defaultdict(set) for deduplication, defaultdict(list) for grouping. The factory can be any callable, including a lambda:

stats = defaultdict(lambda: {"count": 0, "views": 0})
stats["python"]["count"] += 1

`Counter` — count anything in one line

Counting the frequency of items in a list normally takes a loop and a dict. Counter collapses that entirely:

# Before
word_counts = {}
for word in words:
    word_counts[word] = word_counts.get(word, 0) + 1

# After
from collections import Counter

word_counts = Counter(words)

Counter accepts any iterable — strings, lists, generator expressions. It comes with most_common(n), which returns the top-n items sorted by count:

top5 = word_counts.most_common(5)
# [('python', 42), ('tutorial', 31), ...]

Counters also support arithmetic: add two Counters to combine tallies, subtract to find the difference, use & for intersection (minimum counts) and | for union (maximum counts):

c1 = Counter(python=10, javascript=5)
c2 = Counter(python=3, rust=8)
print(c1 + c2)  # Counter({'python': 13, 'rust': 8, 'javascript': 5})
print(c1 - c2)  # Counter({'python': 7, 'javascript': 5})

`deque` — efficient append/pop from both ends

Python lists are fast at appending to the right, but list.insert(0, x) and list.pop(0) are O(n) — they shift every element. deque (double-ended queue) does both in O(1):

# Before: using list as a queue (slow left-side operations)
queue = []
queue.append("task_a")
queue.insert(0, "urgent_task")   # O(n) — shifts everything
next_task = queue.pop(0)         # O(n) — shifts everything

# After: deque
from collections import deque

queue = deque()
queue.append("task_a")           # append to right
queue.appendleft("urgent_task")  # O(1) — no shifting
next_task = queue.popleft()      # O(1) — no shifting

The maxlen parameter turns a deque into a sliding window that automatically discards the oldest item when full — perfect for keeping the last N log lines or prices:

recent_views = deque(maxlen=10)
for view in stream:
    recent_views.append(view)    # oldest auto-dropped when len > 10

deque.rotate(n) shifts all elements by n positions, which is handy for circular buffers or round-robin scheduling:

tasks = deque(["a", "b", "c", "d"])
tasks.rotate(1)   # deque(['d', 'a', 'b', 'c'])

`namedtuple` — lightweight records with field names

Plain tuples force you to remember that point[0] is x and point[1] is y. namedtuple gives you attribute access with zero memory overhead over a regular tuple:

# Before: plain tuple, no context
article = ("Python collections", "tutorial", 1200)
print(article[2])  # What is index 2 again?

# After: namedtuple
from collections import namedtuple

Article = namedtuple("Article", ["title", "tag", "word_count"])
article = Article("Python collections", "tutorial", 1200)
print(article.word_count)  # clear and readable

Two utility methods make namedtuple practical for data pipelines:

_asdict() converts the record to an OrderedDict (or regular dict in Python 3.8+), useful for serialization.
_replace() returns a copy with specific fields swapped — since namedtuples are immutable, this is how you "update" a record.

published = article._replace(tag="python")
data = article._asdict()  # {'title': 'Python collections', 'tag': 'tutorial', ...}

For more complex records with defaults, validators, or type annotations, see the dataclasses module (link in Further Reading).

`OrderedDict` — insertion order with extras

Since Python 3.7, regular dicts preserve insertion order, so OrderedDict is rarely needed. But it still has one trick plain dicts lack: move_to_end():

from collections import OrderedDict

cache = OrderedDict()
cache["a"] = 1
cache["b"] = 2
cache["c"] = 3

cache.move_to_end("a")           # move "a" to the back
cache.move_to_end("c", last=False)  # move "c" to the front

This makes OrderedDict the natural building block for an LRU (Least Recently Used) cache.

`ChainMap` — layered configuration

ChainMap combines multiple dicts into a single view, resolving lookups from left to right. This is ideal for layered config: CLI args override environment variables, which override defaults.

from collections import ChainMap
import os

defaults = {"debug": False, "output": "stdout", "retries": 3}
env_config = {"debug": True} if os.getenv("DEBUG") else {}
cli_args = {"output": "file.log"}

config = ChainMap(cli_args, env_config, defaults)
print(config["debug"])    # True  (from env_config)
print(config["retries"])  # 3     (from defaults)
print(config["output"])   # file.log (from cli_args)

Writes go to the first map, so overrides stay isolated — the defaults dict is never mutated.

Real pipeline example: Counter + defaultdict together

Here is a compact publishing pipeline that tracks article stats and groups titles by tag using both types together:

from collections import Counter, defaultdict

articles = [
    {"title": "Python collections", "tag": "python", "views": 420},
    {"title": "Python decorators",  "tag": "python", "views": 310},
    {"title": "Git rebasing",       "tag": "git",    "views": 180},
    {"title": "Git aliases",        "tag": "git",    "views": 95},
    {"title": "Docker basics",      "tag": "devops", "views": 260},
]

# Count articles per tag
tag_counts = Counter(a["tag"] for a in articles)
print(tag_counts.most_common())
# [('python', 2), ('git', 2), ('devops', 1)]

# Group titles by tag
by_tag = defaultdict(list)
for a in articles:
    by_tag[a["tag"]].append(a["title"])

print(dict(by_tag))
# {'python': ['Python collections', 'Python decorators'],
#  'git':    ['Git rebasing', 'Git aliases'],
#  'devops': ['Docker basics']}

# Total views per tag using Counter arithmetic
views_by_tag = Counter()
for a in articles:
    views_by_tag[a["tag"]] += a["views"]

print(views_by_tag.most_common(1))
# [('python', 730)]

No extra boilerplate, no KeyErrors, no O(n) list operations — just the right tool for each job.

If you want to see these patterns applied to an actual AI publishing pipeline — including scripts that auto-post to Dev.to, track stats, and generate content — check out the full pipeline guide at germy5.gumroad.com/l/xhxkzz.

Python Sets: Fast Lookups, Deduplication, and Set Operations

German Yamil — Thu, 14 May 2026 14:16:29 +0000

If you have ever written a loop to check whether a value already exists in a list, or used a list just to collect unique items, you have been solving a problem that Python sets handle natively — and far more efficiently. Sets give you O(1) membership testing, automatic deduplication, and a suite of mathematical operations that make filtering and comparing collections of data almost trivial.

Why Sets? O(1) Lookup vs O(n) List

The core reason to reach for a set instead of a list is performance. When you write value in my_list, Python scans every element from the beginning until it finds a match — that is O(n) time. When you write value in my_set, Python hashes the value and checks a single bucket — that is O(1) time, regardless of whether the set has 10 items or 10 million.

import time

data_list = list(range(1_000_000))
data_set = set(data_list)

# List: scans up to 1,000,000 elements
start = time.perf_counter()
999_999 in data_list
print(f"List lookup: {time.perf_counter() - start:.6f}s")

# Set: one hash lookup
start = time.perf_counter()
999_999 in data_set
print(f"Set lookup:  {time.perf_counter() - start:.6f}s")

On a typical machine the set lookup is hundreds of times faster. This gap grows as the collection grows.

Sets also deduplicate automatically. Converting a list to a set is the idiomatic one-liner for removing duplicates:

tags = ["python", "tutorial", "python", "beginners", "tutorial"]
unique_tags = list(set(tags))
# ['beginners', 'python', 'tutorial']  — order not guaranteed

Creating Sets

There are three ways to create a set:

# 1. Literal syntax — curly braces with values
languages = {"python", "rust", "go"}

# 2. set() constructor — from any iterable
from_list = set([1, 2, 3, 2, 1])   # {1, 2, 3}
from_str  = set("hello")            # {'h', 'e', 'l', 'o'}

# 3. Empty set — MUST use set(), not {}
empty = set()    # {} creates an empty dict, not a set

One trap beginners hit: {} creates an empty dict. Always use set() for an empty set.

Set Operations: Union, Intersection, Difference, Symmetric Difference

Python sets support the full algebra of mathematical sets through readable operators:

published = {"intro-python", "python-lists", "python-dicts", "python-sets"}
drafted   = {"python-sets", "python-generators", "python-typing"}

# Union — all slugs in either collection
all_slugs = published | drafted
# {'intro-python', 'python-lists', 'python-dicts',
#  'python-sets', 'python-generators', 'python-typing'}

# Intersection — slugs in both (already published AND drafted)
overlap = published & drafted
# {'python-sets'}

# Difference — drafted but NOT yet published
needs_review = drafted - published
# {'python-generators', 'python-typing'}

# Symmetric difference — in one but not both
unique_to_each = published ^ drafted
# {'intro-python', 'python-lists', 'python-dicts',
#  'python-generators', 'python-typing'}

Each operator has a method equivalent (union(), intersection(), difference(), symmetric_difference()) that accepts any iterable, not just other sets, which is useful when your source data hasn't been converted yet.

Membership Testing with `in`

The in operator is where sets shine in everyday code:

STOP_WORDS = {"the", "a", "an", "is", "it", "in", "on", "at"}

def extract_keywords(text: str) -> list[str]:
    words = text.lower().split()
    return [w for w in words if w not in STOP_WORDS]

print(extract_keywords("Python is a great language for automation"))
# ['python', 'great', 'language', 'for', 'automation']

Making STOP_WORDS a set instead of a list costs nothing in readability but makes each in check instant rather than linear.

Set Methods: add, remove, discard, pop, clear

Sets are mutable and come with a practical set of mutation methods:

seen = {"slug-a", "slug-b"}

seen.add("slug-c")        # adds slug-c; no-op if already present

seen.remove("slug-a")     # removes slug-a; raises KeyError if missing
seen.discard("slug-z")    # removes if present; silent no-op if missing

item = seen.pop()         # removes and returns an arbitrary element
seen.clear()              # empties the set in place

The distinction between remove() and discard() matters in pipelines: use discard() when absence is not an error, remove() when you want to catch unexpected state.

Frozen Sets: Immutable and Hashable

A frozenset is an immutable version of a set. Once created it cannot be modified. This unlocks two capabilities that regular sets lack: it can be used as a dictionary key and as an element inside another set.

# frozenset as a dict key — useful for caching combination results
cache: dict[frozenset, str] = {}

def get_combined_tag_slug(tags: set[str]) -> str:
    key = frozenset(tags)
    if key not in cache:
        cache[key] = "-".join(sorted(tags))
    return cache[key]

print(get_combined_tag_slug({"python", "tutorial"}))
# 'python-tutorial'
print(get_combined_tag_slug({"tutorial", "python"}))   # cache hit
# 'python-tutorial'

# frozenset as an element of another set
pipelines = {
    frozenset({"fetch", "parse", "publish"}),
    frozenset({"fetch", "validate", "upload"}),
}

frozenset supports all the read-only set operations (union, intersection, in, etc.) but none of the mutation methods.

Set Comprehensions

Just like list and dict comprehensions, Python has set comprehensions:

slugs = [
    "python-lists",
    "Python-Lists",   # duplicate after normalization
    "python-sets",
    "PYTHON-SETS",
]

normalized = {slug.lower() for slug in slugs}
# {'python-lists', 'python-sets'}

Set comprehensions are a clean way to transform and deduplicate in one expression. You can also add a filter condition:

# Only slugs that start with "python"
python_slugs = {s.lower() for s in slugs if s.lower().startswith("python")}

Real Pattern: Deduplicating Published Article Slugs

Here is a realistic use case from an automated publishing pipeline. You have a JSON file of published slugs and an incoming batch of candidate articles. You want to skip anything already published without scanning a list on every check:

import json
from pathlib import Path

def load_published_slugs(path: Path) -> set[str]:
    if not path.exists():
        return set()
    data = json.loads(path.read_text())
    return set(data.get("published", []))

def filter_new_articles(
    candidates: list[dict],
    published: set[str],
) -> list[dict]:
    return [a for a in candidates if a["slug"] not in published]

def mark_published(slug: str, path: Path) -> None:
    data = json.loads(path.read_text()) if path.exists() else {"published": []}
    published = set(data["published"])
    published.add(slug)
    data["published"] = sorted(published)
    path.write_text(json.dumps(data, indent=2))

# Usage
queue_path = Path("publish_queue.json")
published_slugs = load_published_slugs(queue_path)

candidates = [
    {"slug": "python-sets",        "title": "Python Sets"},
    {"slug": "python-generators",  "title": "Python Generators"},
    {"slug": "intro-python",       "title": "Intro to Python"},  # already published
]

to_publish = filter_new_articles(candidates, published_slugs)
for article in to_publish:
    # ... publish article ...
    mark_published(article["slug"], queue_path)

The published_slugs set makes not in published a single hash lookup for every candidate, keeping the pipeline fast even with hundreds of tracked slugs.

When NOT to Use Sets

Sets are powerful but not universal. Avoid them when:

Order matters. Sets are unordered; you cannot rely on iteration order or index into a set with my_set[0]. Use a list or collections.OrderedDict if sequence is meaningful.
You need duplicates. Sets silently discard repeated values. If multiplicity matters (a shopping cart with qty > 1, a word-frequency count), use a list or collections.Counter.
Elements are unhashable. Sets can only hold hashable objects. Lists and dicts cannot be set elements; frozenset and tuple can.
You need the nth item. There is no slicing on sets. Convert to a sorted list first if you need positional access.

A quick decision rule: if your primary operations are "does this item exist?" and "give me the unique items", reach for a set. If you need order, indexing, or duplicates, reach for a list.

Automating a content pipeline — tracking published slugs, filtering already-processed URLs, deduplicating API responses — is exactly where sets pay off most. If you want to see how these patterns fit together in a full Python publishing workflow, the pipeline guide linked above walks through the complete system.

Passive Income for Python Developers: The Automation-First Approach

German Yamil — Wed, 13 May 2026 18:35:38 +0000

Most passive income advice is written by people selling courses about passive income.

This is written by someone running a Python automation pipeline that publishes daily content and sells a technical ebook. Here's what actually works — and what the numbers look like after 45 published articles.

The Core Idea

Passive income from technical content works like this:

Content → Audience → Trust → Product → Sale

The problem: each step takes time. Writing content, building audience, maintaining trust — these are ongoing jobs. Without automation, you're trading hours for dollars.

With automation, you front-load the work (building the system) and amortize it across everything the system produces.

What Python Can Automate

Here's what my pipeline handles without manual intervention:

# Daily 10am — runs automatically via launchd
def daily_publish():
    article = queue.get_next()          # Get queued article
    url = devto.publish(article)        # Publish to Dev.to
    hashnode.crosspost(url, article)    # Mirror to Hashnode (2M+ readers)
    cover.attach(article, url)          # Add cover image
    links.patch_stale(url, article)     # Fix links in older articles
    queue.mark_published(article, url)  # Update state

Every day, 10am, whether I'm working or not. If the Mac sleeps, launchd fires the job on wake. No manual steps.

The Real Numbers

After 45 published articles across 3 months:

Metric	Value
Articles published	45
Total views	473
Daily publishing cost	~$0.40 (Claude API)
Monthly infra cost	$0 (runs on laptop)
Time spent daily	~0 minutes

The pipeline generates content; I tune the strategy.

Where the Income Comes From

Ebooks (Gumroad/KDP): Package your knowledge as a PDF or epub. One-time write, indefinite sales. My pipeline sells for $9.99. The content that drives sales: articles that show developers how to build automation systems.

The key insight: content for beginners builds SEO and audience. Content for intermediate developers converts to sales. You need both.

Building the System vs. Buying It

You can build this from scratch. Here's what that takes:

Component	Time to Build
Dev.to publisher	2 hours
Queue system	3 hours
Hashnode cross-poster	2 hours
Cover image generator	2 hours
Link patcher	1 hour
launchd scheduler	30 min

Total: ~10 hours of focused Python work.

Or you can get the complete source code with documentation and skip the build phase.

The Compounding Effect

Here's what 45 articles looks like on a view graph:

Week 1:  12 views  (3 articles)
Week 2:  28 views  (3 articles, older ones still getting views)
Week 3:  47 views  (3 articles, 6 older still active)
Week 4:  61 views  (accumulating...)
Month 2: 180 views
Month 3: 345 views

Each article is a permanent asset. Month 3 articles still drive traffic from Month 1 content.

The Mistakes to Avoid

Don't target only beginners. Beginners read and leave. Intermediate developers buy solutions to real problems.

Don't use cron on a laptop. Cron skips when the machine sleeps. Use launchd (Mac) or systemd (Linux).

Don't skip cross-posting. Publishing only to Dev.to leaves Hashnode's 2M+ readers on the table. The canonical URL keeps SEO on Dev.to while Hashnode provides reach.

Don't write without a product. Content without something to sell builds audience for someone else's product.

Starting Points

If you're building from scratch:

Pick a topic you know deeply
Generate 30 article outlines (Claude does this in 10 minutes)
Write/generate articles for weeks 1-4
Build the publisher while week 1 publishes
Add the product before week 3

The system compounds. The earlier you start, the earlier it pays.

Get the Full Pipeline

The complete Python AI Publishing Pipeline — all scripts, architecture docs, and setup guide.

📋 Free: AI Publishing Checklist (7 steps) — PDF, no email required.

🚀 Full pipeline + source code: germy5.gumroad.com/l/xhxkzz — $9.99, 30-day money-back guarantee.

If this was useful, the ❤️ button helps other developers find it.

Python os Module: File System Operations Every Script Needs

German Yamil — Tue, 12 May 2026 18:02:31 +0000

Python os Module: File System Operations Every Script Needs

Every automation script eventually needs to touch the file system — create a directory, check if a file exists, read a config value from the environment.

The os module is Python's interface to the operating system. No shell required.

What os gives you

import os

# Where am I right now?
print(os.getcwd())          # /Users/yamil/projects/pipeline

# What's in this directory?
print(os.listdir("."))      # ['main.py', 'state.json', 'chapters']

# Does this path exist?
print(os.path.exists("state.json"))   # True or False

Three things you can do without touching the shell, without subprocess, and without any external packages. os ships with Python and works the same on macOS, Linux, and Windows.

os.path — the building blocks

os.path handles path string manipulation. You give it strings, it gives you strings back.

import os

path = "/Users/yamil/projects/pipeline/state.json"

os.path.dirname(path)       # '/Users/yamil/projects/pipeline'
os.path.basename(path)      # 'state.json'
os.path.splitext(path)      # ('/Users/yamil/projects/pipeline/state', '.json')
os.path.abspath("state.json")  # resolves relative path to absolute

# Join paths safely (handles slashes correctly on all platforms)
os.path.join("output", "2026", "chapters", "ch01.md")
# 'output/2026/chapters/ch01.md'

# Check what kind of path it is
os.path.exists(path)    # True if file or directory exists
os.path.isfile(path)    # True only for files
os.path.isdir(path)     # True only for directories

The most common gotcha: never build paths with string concatenation ("dir" + "/" + "file"). Use os.path.join() — it handles the separator correctly regardless of operating system.

Working with directories

import os

# Where are you?
print(os.getcwd())

# Change working directory
os.chdir("/Users/yamil/projects")
print(os.getcwd())  # /Users/yamil/projects

# List directory contents (returns filenames as strings)
entries = os.listdir(".")
# ['pipeline', 'notes.md', 'env']

# Filter to just files
files = [e for e in os.listdir(".") if os.path.isfile(e)]

# Create one directory (raises FileExistsError if it already exists)
os.mkdir("output")

# Create nested directories — equivalent to mkdir -p
os.makedirs("output/2026/chapters", exist_ok=True)
# exist_ok=True: no error if the directory already exists

os.makedirs() with exist_ok=True is the pattern you want in almost every script. It's idempotent — safe to call every time the script runs.

Creating, removing, and renaming

import os

# Create a file (via open, not os — but os handles the rest)
with open("draft.txt", "w") as f:
    f.write("content")

# Rename a file (fails if destination exists on some systems)
os.rename("draft.txt", "final.txt")

# Remove a file
os.remove("final.txt")

# Remove an empty directory
os.rmdir("empty_dir")

# For non-empty directories, use shutil
import shutil
shutil.rmtree("output_dir")  # deletes everything inside

Before vs after renaming with a safety check:

# Before: risky — what if the destination already exists?
os.rename("temp_output.json", "state.json")

# After: explicit check first
if os.path.exists("state.json"):
    os.remove("state.json")
os.rename("temp_output.json", "state.json")

There's a better way to do this — covered in the next section.

os.replace() — atomic file replacement

This is the most important os function most beginners don't know about.

import os, json

def save_state(state: dict, state_file: str) -> None:
    """Write state atomically — no partial writes on failure."""
    tmp = state_file + ".tmp"

    # Write to temp file first
    with open(tmp, "w", encoding="utf-8") as f:
        json.dump(state, f, indent=2)

    # Replace atomically — on POSIX systems, this is a single syscall
    os.replace(tmp, state_file)
    # If the process crashes here, state_file still has the old data
    # If os.replace() succeeds, state_file has the new data — no in-between state

Why this matters:

# Naive approach — dangerous
with open("state.json", "w") as f:
    json.dump(state, f)   # if Python crashes mid-write, file is corrupted

# Safe approach — atomic
with open("state.json.tmp", "w") as f:
    json.dump(state, f)
os.replace("state.json.tmp", "state.json")  # all-or-nothing

Unlike os.rename(), os.replace() silently overwrites the destination if it exists. On POSIX systems (macOS, Linux), the replacement is atomic at the OS level — readers either see the old file or the new one, never a partial write.

Use this pattern any time you're writing state files, config files, or any output that another process might be reading concurrently.

Environment variables

import os

# Read a variable (KeyError if not set)
api_key = os.environ["OPENAI_API_KEY"]

# Read with a default (no error if not set)
model = os.getenv("MODEL", "gpt-4o")
debug = os.getenv("DEBUG", "false").lower() == "true"

# Check if a variable exists
if "DATABASE_URL" in os.environ:
    print("Database configured")

# Set a variable for the current process (and subprocesses)
os.environ["LOG_LEVEL"] = "INFO"

# Get all environment variables as a dict
env_copy = os.environ.copy()
env_copy["EXTRA_VAR"] = "injected"
# Pass env_copy to subprocess.run() to give subprocess a modified environment

The os.getenv() pattern with a default is the right approach for optional config. Never hardcode API keys or paths — read them from the environment.

# Pipeline config pattern
import os

API_KEY   = os.environ["OPENAI_API_KEY"]          # required — fail fast if missing
MODEL     = os.getenv("PIPELINE_MODEL", "gpt-4o") # optional with default
MAX_RETRY = int(os.getenv("MAX_RETRY", "3"))       # type conversion after getenv
OUTPUT    = os.getenv("OUTPUT_DIR", "chapters")    # directory override

Walking directory trees: os.walk()

os.listdir() gives you one directory level. os.walk() gives you everything recursively.

import os

# Walk the entire directory tree
for dirpath, dirnames, filenames in os.walk("chapters"):
    print(f"Dir:   {dirpath}")
    print(f"Dirs:  {dirnames}")
    print(f"Files: {filenames}")
    print()

Dir:   chapters
Dirs:  ['en', 'es']
Files: []

Dir:   chapters/en
Dirs:  []
Files: ['ch01.md', 'ch02.md', 'ch03.md']

Dir:   chapters/es
Dirs:  []
Files: ['ch01.md', 'ch02.md']

Practical pattern — collect all Markdown files:

import os

def find_markdown_files(root: str) -> list[str]:
    """Return absolute paths of all .md files under root."""
    results = []
    for dirpath, _, filenames in os.walk(root):
        for filename in filenames:
            if filename.endswith(".md"):
                results.append(os.path.join(dirpath, filename))
    return results

md_files = find_markdown_files("chapters")
print(f"Found {len(md_files)} markdown files")

os.path vs pathlib — when to use each

pathlib is the modern alternative. For new code, it's usually the better choice.

# os.path — verbose but explicit
import os
base = os.path.dirname(os.path.abspath(__file__))
config = os.path.join(base, "config", "settings.json")
os.makedirs(os.path.dirname(config), exist_ok=True)

# pathlib — cleaner, same result
from pathlib import Path
base = Path(__file__).parent
config = base / "config" / "settings.json"
config.parent.mkdir(parents=True, exist_ok=True)

Use os when you need:

os.replace() for atomic writes — pathlib has no equivalent
os.environ / os.getenv() for environment variables
os.walk() (pathlib has rglob() which is usually cleaner, but os.walk() gives you more control)
Compatibility with code or libraries that only accept strings

Use pathlib when you need:

Path manipulation in new code
Reading/writing files (path.read_text(), path.write_text())
Glob patterns (path.glob("*.json"))
Cleaner API in general

In practice, most real automation scripts use both.

Real pipeline example: organizing generated chapter files

This is the pattern from the ebook publishing pipeline — generating chapter files and organizing them into per-language directories.

import os
import json

def organize_chapters(
    source_dir: str,
    output_base: str,
    languages: list[str],
) -> dict[str, list[str]]:
    """
    Move generated chapter files from a flat source directory
    into output_base/lang/ subdirectories.

    Expects filenames like: ch01_en.md, ch01_es.md, ch02_en.md
    """
    organized = {lang: [] for lang in languages}

    # Create output directories for each language
    for lang in languages:
        lang_dir = os.path.join(output_base, lang)
        os.makedirs(lang_dir, exist_ok=True)

    # Walk source directory and sort by language suffix
    for filename in os.listdir(source_dir):
        if not filename.endswith(".md"):
            continue

        stem, _ = os.path.splitext(filename)  # 'ch01_en'
        parts = stem.rsplit("_", 1)            # ['ch01', 'en']
        if len(parts) != 2:
            continue

        chapter_id, lang = parts
        if lang not in languages:
            continue

        src = os.path.join(source_dir, filename)
        dst = os.path.join(output_base, lang, filename)

        # Atomic replace — safe even if destination exists from a previous run
        os.replace(src, dst)
        organized[lang].append(dst)
        print(f"  {filename} → {lang}/")

    return organized


def save_manifest(organized: dict, manifest_path: str) -> None:
    """Write the chapter manifest atomically."""
    tmp = manifest_path + ".tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        json.dump(organized, f, indent=2)
    os.replace(tmp, manifest_path)


# Usage
result = organize_chapters(
    source_dir="generated",
    output_base="chapters",
    languages=["en", "es"],
)

save_manifest(result, "chapters/manifest.json")
print(f"Organized {sum(len(v) for v in result.values())} files")

This script is safe to re-run. os.makedirs(..., exist_ok=True) won't fail if directories already exist. os.replace() won't fail if the destination already exists from a previous partial run.

The pipeline uses os and pathlib together for every file operation — organizing chapters, managing state files, and cleaning up temp dirs: germy5.gumroad.com/l/xhxkzz — pay what you want, min $9.99.

Python Error Handling: try, except, finally, and raise Done Right

German Yamil — Mon, 11 May 2026 22:00:04 +0000

Python Error Handling: try, except, finally, and raise Done Right

You wrote a script that calls an API, parses the response, and saves a file. It runs fine for a week. Then one day the API is down — and your script crashes with a cryptic traceback, or worse, it swallows the error entirely and writes an empty file like nothing happened.

That's what bad exception handling looks like. This article fixes it.

The Problem: Bare `except` Catches Everything

The most dangerous pattern in beginner Python code:

# BAD — do not do this
try:
    data = fetch_data(url)
    save(data)
except:
    pass

What does except: pass actually catch? Everything. Including:

KeyboardInterrupt — your Ctrl+C now does nothing
SystemExit — sys.exit() is silently swallowed
MemoryError — you ran out of RAM, script keeps running
Your actual bug, hidden forever

The script "succeeds" and writes nothing, and you have no idea why.

try/except Basics: Catch the Right Exception

Name the exception you expect. Every stdlib function documents what it raises — use that.

# BAD
try:
    with open("config.json") as f:
        config = json.load(f)
except:
    config = {}

# GOOD
try:
    with open("config.json") as f:
        config = json.load(f)
except FileNotFoundError:
    config = {}   # file doesn't exist yet — that's fine, use defaults
except json.JSONDecodeError as e:
    raise RuntimeError(f"config.json is malformed: {e}") from e

The as e binding gives you the exception object. Use it — log it, include it in a message, or re-raise it. Never throw away the information.

Multiple except Clauses and Exception Hierarchy

You can stack multiple except blocks. Python checks them top to bottom and runs the first match.

import urllib.request
import json

def fetch_json(url: str) -> dict:
    try:
        with urllib.request.urlopen(url, timeout=10) as resp:
            return json.loads(resp.read().decode())
    except urllib.error.HTTPError as e:
        print(f"HTTP {e.code}: {e.reason}")
        return {}
    except urllib.error.URLError as e:
        print(f"Network error: {e.reason}")
        return {}
    except json.JSONDecodeError as e:
        print(f"Bad JSON at char {e.pos}: {e.msg}")
        return {}

Order matters because of inheritance. URLError is the parent of HTTPError — if you put URLError first, HTTPError will never be reached. Most specific exceptions go first.

You can also group unrelated exceptions that share the same handler:

except (TimeoutError, ConnectionResetError) as e:
    print(f"Connection problem: {e}")

The else Clause: Rarely Taught, Very Useful

else runs when the try block completes with no exception. This lets you separate "the thing that might fail" from "what to do when it succeeds" — without wrapping the success code inside try where it might accidentally catch the wrong error.

# Without else — risky: process() errors are caught by except
try:
    data = fetch_json(url)
except urllib.error.URLError:
    data = {}
else:
    # only runs if fetch_json() succeeded
    result = process(data)   # errors here propagate normally
    save(result)

Use else whenever the code after a successful operation could itself raise — you don't want those errors mixed up with the errors from the operation you were protecting.

finally: Guaranteed Cleanup, Even on Crash

finally always runs. Exception or no exception, return statement or not.

def write_report(path: str, content: str) -> None:
    f = open(path, "w")
    try:
        f.write(content)
    except OSError as e:
        print(f"Write failed: {e}")
        raise
    finally:
        f.close()   # always runs — file is never left open

This is exactly what with open(...) does for you under the hood. Use finally for resources that don't have a context manager: database cursors, raw sockets, temp directories you manage manually, locks.

import threading

lock = threading.Lock()

def update_shared_state(value):
    lock.acquire()
    try:
        shared_state["count"] += value
    finally:
        lock.release()   # always released, even if += raises

raise: Re-Raising and Raising New Exceptions

Three forms you need to know:

# 1. Raise a new exception
raise ValueError("price must be positive")

# 2. Re-raise the current exception (inside an except block)
try:
    connect()
except ConnectionError:
    log.error("Connection failed")
    raise   # propagates the original exception unchanged

# 3. Raise a new exception while preserving the original cause
try:
    raw = json.loads(text)
except json.JSONDecodeError as e:
    raise ValueError(f"Invalid pipeline config") from e

Form 3 is exception chaining. The from e attaches the original error as __cause__, so the full traceback shows both errors. Python will print "The above exception was the direct cause of the following exception" — invaluable for debugging.

Custom Exception Classes

When your library or pipeline has its own failure modes, define your own exceptions. Callers can then catch yours specifically without catching everything.

# Define once in exceptions.py
class PipelineError(Exception):
    """Base class for all pipeline errors."""

class APIRateLimitError(PipelineError):
    """Raised when the LLM API returns 429."""
    def __init__(self, retry_after: int = 60):
        self.retry_after = retry_after
        super().__init__(f"Rate limited — retry after {retry_after}s")

class ChapterValidationError(PipelineError):
    """Raised when generated content fails quality checks."""
    def __init__(self, chapter_id: str, reason: str):
        self.chapter_id = chapter_id
        super().__init__(f"Chapter {chapter_id} failed validation: {reason}")

Now callers can be precise:

try:
    content = generate_chapter("ch03")
except APIRateLimitError as e:
    time.sleep(e.retry_after)
    content = generate_chapter("ch03")   # retry after waiting
except ChapterValidationError as e:
    log.warning(f"Skipping {e.chapter_id}, will regenerate later")
except PipelineError as e:
    log.error(f"Unhandled pipeline error: {e}")
    raise

The hierarchy APIRateLimitError → PipelineError → Exception means you can catch broadly (all pipeline errors) or narrowly (only rate limits), depending on where you are in the code.

Exception Chaining in Practice

Here is the pattern the pipeline uses when calling external services:

import anthropic

def call_llm(prompt: str, model: str = "claude-sonnet-4-6") -> str:
    client = anthropic.Anthropic()
    try:
        message = client.messages.create(
            model=model,
            max_tokens=4096,
            messages=[{"role": "user", "content": prompt}],
        )
        return message.content[0].text
    except anthropic.RateLimitError as e:
        raise APIRateLimitError(retry_after=60) from e
    except anthropic.APIConnectionError as e:
        raise PipelineError("LLM unreachable — check network") from e
    except anthropic.APIStatusError as e:
        raise PipelineError(f"LLM API error {e.status_code}: {e.message}") from e

The original SDK exception is preserved as __cause__. When this crashes in production, you see both the pipeline-level message and the raw SDK error in the traceback.

Real Pipeline Pattern: LLM + File + Subprocess Errors

Putting it all together — a realistic generation loop with proper error handling at every layer:

import time
import logging
from pathlib import Path

log = logging.getLogger(__name__)

def generate_and_save(chapter_id: str, prompt: str, output_dir: Path) -> bool:
    """
    Generate one chapter and save it. Returns True on success.
    Raises PipelineError on unrecoverable failure.
    """
    max_retries = 3
    delay = 5.0

    for attempt in range(1, max_retries + 1):
        try:
            log.info(f"[{chapter_id}] Attempt {attempt}/{max_retries}")
            content = call_llm(prompt)
        except APIRateLimitError as e:
            log.warning(f"[{chapter_id}] Rate limited, waiting {e.retry_after}s")
            time.sleep(e.retry_after)
            continue
        except PipelineError:
            if attempt == max_retries:
                raise
            log.warning(f"[{chapter_id}] LLM call failed, retrying in {delay}s")
            time.sleep(delay)
            delay *= 2
            continue
        else:
            # LLM call succeeded — now handle file errors separately
            try:
                out_path = output_dir / f"{chapter_id}.md"
                out_path.write_text(content, encoding="utf-8")
                log.info(f"[{chapter_id}] Saved to {out_path}")
                return True
            except OSError as e:
                raise PipelineError(f"Cannot write {chapter_id}") from e

    return False   # exhausted retries without raising

Notice the structure:

except APIRateLimitError — handle specifically, with the retry_after data
except PipelineError — handle the general case, re-raise on final attempt
else — file write is outside the LLM retry logic entirely
File errors raise immediately (no retry), chained to the original OSError

What NOT to Do: Swallowing Exceptions Silently

A quick checklist of patterns that destroy debugging:

# NEVER — hides all errors
except:
    pass

# NEVER — hides the error message, logs nothing useful
except Exception:
    pass

# BAD — logs something but still swallows it;
# the caller thinks the function succeeded
except Exception as e:
    print(f"Error: {e}")
    return None   # caller gets None, has no idea why

# BAD — catching broad Exception when you mean a specific error
try:
    value = d["key"]
except Exception:   # should be KeyError
    value = "default"

# GOOD — specific, logged, re-raised or handled with intent
except KeyError:
    log.warning("Key missing from config, using default")
    value = "default"

If you catch an exception and don't re-raise it, you are making a deliberate decision that the error is recoverable and the caller doesn't need to know. Make that decision consciously, not by habit.

The full pipeline wraps every LLM call and file write in try/except/finally — here's how the retry logic works end-to-end: germy5.gumroad.com/l/xhxkzz — pay what you want, min $9.99.

DEV Community: German Yamil

Python enumerate, zip, map, and filter: Loop Upgrades Every Beginner Needs

Why These Builtins Matter

enumerate() — Never Write i = 0; i += 1 Again

zip() — Pair Two Lists Cleanly

map() — Apply a Function to Every Item

filter() — Keep Only Matching Items

Combining Them: Real Chains

sorted() with key= — The Most Useful Builtin You Are Underusing

any() and all() — Short-Circuit Evaluation for Collections

Real Pipeline Pattern

Quick Decision Reference

Further Reading

Python String Methods: The Complete Reference You'll Actually Use

🎁 Free: AI Publishing Checklist — 7 steps in Python · Full pipeline: germy5.gumroad.com/l/xhxkzz (pay what you want, min $9.99)

Strings Are Immutable — What That Means in Practice

Cleaning Input: strip(), lstrip(), rstrip()

Case Conversion: upper(), lower(), title(), capitalize(), swapcase()

Searching: find(), index(), startswith(), endswith(), in

Splitting and Joining: split(), rsplit(), splitlines(), join()

Replacing Text: replace() and translate()

Testing String Content: isdigit(), isalpha(), isalnum(), isspace()

Padding and Alignment: zfill(), ljust(), rjust(), center()

Real Pipeline Patterns

Further Reading

Python datetime: Dates, Times, and Timezones Without the Confusion

The Three Types: date, time, and datetime

Creating Datetimes: now(), today(), and date.today()

strftime() — Format a Datetime to String

strptime() — Parse a String into datetime

timedelta — Date Arithmetic

Comparing Datetimes — Sorting and Filtering

Timezone-Aware Datetimes with zoneinfo (Python 3.9+)

Unix Timestamps: timestamp() and fromtimestamp()

Real Pipeline Pattern: Logging, Scheduling, and Time-Since-Last-Run

Further Reading

Python Regex: re Module Patterns That Actually Make Sense

🎁 Free: AI Publishing Checklist — 7 steps in Python · Full pipeline: germy5.gumroad.com/l/xhxkzz (pay what you want, min $9.99)

Why Regex Exists (and When to Reach for It)

re.search() vs re.match() vs re.fullmatch()

re.findall() and re.finditer()

Basic Patterns: The Building Blocks

Character Classes [] and Negation [^]

Groups () and Named Groups (?P...)

re.sub() — Substitution and Replacement

Compiled Patterns with re.compile()

Flags: IGNORECASE, MULTILINE, DOTALL

Real Pipeline Patterns

Further Reading

How to Turn Your Python Scripts Into a Side Income Stream

What Makes a Python Script Worth Selling

Three Packaging Models

1. Sell the Script (One-time, $10-50)

2. Sell Documentation (One-time, $15-30)

3. Sell Access (Recurring, $5-20/month)

Finding Buyers

What Actually Converts

A Python Script Worth Selling: Real Example

The Math

Starting This Week

Further Reading

Get the Complete System

Python functools: lru_cache, partial, reduce, and wraps

lru_cache — Stop Paying Twice for the Same Result

partial — Freeze Some Arguments Up Front

wraps — Every Decorator Must Use This

reduce — Fold a Sequence to a Single Value

total_ordering — Define Two Comparisons, Get Six

cached_property — Compute Once Per Instance

Which Ones You'll Use Every Day

Further Reading

Python itertools: Efficient Loops Without the Boilerplate

What itertools Gives You

chain() — Concatenate Iterables Without Building a List

islice() — Take the First N Items from Any Iterable

batched() — Process in Chunks of N

groupby() — Group Sorted Data by a Key

product() — Cartesian Product Replaces Nested Loops

combinations() and permutations()

repeat() and cycle() — Generating Test Data

enumerate() — Never Write `i = 0; i += 1` Again

Why the `collections` module exists

`defaultdict` — no more "if key not in dict"

`Counter` — count anything in one line

`deque` — efficient append/pop from both ends

`namedtuple` — lightweight records with field names

`OrderedDict` — insertion order with extras

`ChainMap` — layered configuration

Membership Testing with `in`

The Problem: Bare `except` Catches Everything