DEV Community: Saurav Jha

Python Generator

Saurav Jha — Sat, 21 Feb 2026 02:49:39 +0000

Generator functions are a special kind of function that return a lazy iterator. These are objects that you can loop over like a list. However, unlike lists, lazy iterators do not store their contents in memory.

A generator expression (also called a generator comprehension) looks almost identical to a list comprehension - but instead of creating a full list in memory, it creates a generator object that produces values lazily (one at a time).

List Comprehension vs Generator Expression
List comprehension (creates full list in memory)

squares = [x*x for x in range(5)]

print(squares)

Output:

[0, 1, 4, 9, 16]

All values are computed immediately
Stored in memory

Generator expression (lazy evaluation)

squares = (x*x for x in range(5))

print(squares)

***Output: *
Nothing is computed yet
Values are generated only when needed

How to use a generator expression?
You must iterate over it
for num in squares:
print(num)

Or convert it:
print(list(squares))

Feature	List Comprehension	Generator Expression
Syntax	`[x for x in ...]`	`(x for x in ...)`
Memory	High	Low
Execution	Immediate	Lazy
Type	list	generator

Memory Example (Important)

import sys

lst = [x for x in range(1000000)]
gen = (x for x in range(1000000))

print(sys.getsizeof(lst))  # large
print(sys.getsizeof(gen))  # small

Generator uses much less memory.

Why "without calling a function"?
Normally, to create a generator, you'd write:

def my_generator():
    for x in range(5):
        yield x*x

But with a generator expression:

gen = (x*x for x in range(5))

No function definition needed.

Very Common Use Case
Passing directly into functions like sum():

total = sum(x*x for x in range(1000000))

Notice:
No brackets
Memory efficient
Clean syntax

Python Yield Statement

On the whole, yield is a fairly simple statement. Its primary job is to control the flow of a generator function in a way that’s similar to return statements.
When you call a generator function or use a generator expression, you return a special iterator called a generator. You can assign this generator to a variable in order to use it. When you call special methods on the generator, such as next(), the code within the function is executed up to yield.

When the Python yield statement is hit, the program suspends function execution and returns the yielded value to the caller. (In contrast, return stops function execution completely.) When a function is suspended, the state of that function is saved. This includes any variable bindings local to the generator, the instruction pointer, the internal stack, and any exception handling.

yield can be used in many ways to control your generator’s execution flow. The use of multiple Python yield statements can be leveraged as far as your creativity allows.

When to use generator expressions?
✔ Large datasets
✔ Streaming data
✔ When you only iterate once
✔ Memory-sensitive applications
❌ When you need indexing or multiple passes

How lazy evaluation works internally?
Lazy evaluation means values are computed only when they are actually needed, not in advance.
In Python, this is implemented mainly through iterators and generators.

Eager vs Lazy (mental model)
Eager evaluation

data = [x * 2 for x in range(5)]

What happens internally:

Loop runs immediately
All values computed
Stored in memory as a list

Lazy evaluation

What happens:

Nothing is computed
Only a generator object is created
Values are computed one at a time, on demand

What a generator really is internally
A generator is:

A state machine
With an instruction pointer
And saved local variables When Python sees yield, it:

Pauses execution
Saves local state
Returns a value
Resumes later from the same spot

Step-by-step execution

def squares():
    for i in range(3):
        yield i * i

Internal flow
g = squares() # generator created (no execution)
next(g) # runs until first yield → returns 0
next(g) # resumes → returns 1
next(g) # resumes → returns 4
next(g) # StopIteration raised

At each next():
Python resumes from the last saved instruction
Executes until next yield
Saves state again

Why generators are memory-efficient?

range(1_000_000)

Stores start, stop, step only
No list of numbers

(x*x for x in range(1_000_000))

Stores:
Reference to range
Current index
Execution state

Memory usage is constant, not proportional to size.

Lazy evaluation in built-ins
Many Python functions are lazy:

Function	Lazy?
`range()`	✅
`map()`	✅
`filter()`	✅
`zip()`	✅
`sum()`	❌ (consumes lazily)

Example:
map(lambda x: x*x, range(10))

No computation until iterated.

How StopIteration ends lazy evaluation?
When generator finishes:

Python raises StopIteration
Iteration protocol catches it
Loop stops This is how for loops work internally.

Why lazy evaluation is single-pass?

Once consumed:

gen = (x for x in range(3))
list(gen)   # [0, 1, 2]
list(gen)   # []

Why?
State machine has reached the end
No reset unless recreated

“Lazy evaluation in Python works by using iterators and generators, which compute values only when requested. Internally, a generator is a state machine that pauses execution at each yield, saves its local state, and resumes later. This allows Python to process large or infinite data streams efficiently with constant memory usage.”

Python generators support advanced control methods that let you send data into, raise exceptions inside, and terminate a generator from the outside.

These are:

.send(value)
.throw(exception)
.close()

.send(value) — send data into a generator

Normally, generators only yield values out.
.send() allows you to send a value back into the generator, which becomes the result of the last yield expression.

def counter():
    value = yield 0
    while True:
        value = yield value + 1

gen = counter()

print(next(gen))       # start generator → yields 0
print(gen.send(10))    # sends 10 into generator → yields 11
print(gen.send(20))    # yields 21

Key rules
First call must be next(gen) or gen.send(None)
send(x) assigns x to the last yield expression

.throw() — raise an exception inside the generator
.throw() injects an exception at the point where the generator is paused.

def generator():
    try:
        while True:
            yield "running"
    except ValueError:
        yield "ValueError handled"

gen = generator()
print(next(gen))                 # running
print(gen.throw(ValueError))     # ValueError handled

Use cases

Cancel work
Signal error conditions
Interrupt long-running generators
If the generator does not catch the exception → it propagates outward.

.close() — stop the generator gracefully
.close() raises a GeneratorExit inside the generator.

def generator():
    try:
        while True:
            yield "working"
    finally:
        print("Cleaning up resources")

gen = generator()
print(next(gen))
gen.close()

Output
working
Cleaning up resources


**FastAPI dependency that yields a session. Automatically commits if no exception occurs,or rolls back if an exception is raised.**

async def get_db()-> AsyncGenerator[AsyncSession]:
    async with SessionLocal() as session:
       try:
           yield session
           await session.commit()
       except Exception:
           await session.rollback()  # Central rollback
           raise

Lifecycle Summary
Method Purpose
next() Resume generator
send(x) Resume + send value
throw(e) Resume + raise exception
close() Terminate generator

To Summarize

A generator in Python is a function that:
Uses the yield keyword
Produces values one at a time
Remembers its state between executions
Unlike a normal function, it does not return all results at once.

When Python sees yield:
It returns the value
Pauses execution
Saves local state
Resumes from that point on next iteration

Dual write problem in distributed systems

Saurav Jha — Mon, 29 Dec 2025 12:05:30 +0000

The dual write problem is a classic issue that arises in distributed systems when a single logical operation needs to update two (or more) separate systems or data stores — for example, writing to a database and sending an event/message to a message broker (like Kafka).

Because these systems are independent, ensuring atomicity (all-or-nothing behavior) across them is extremely difficult without a distributed transaction protocol. Let’s break it down clearly:

Distributed transaction protocol:
A distributed transaction protocol is a mechanism that ensures atomicity (all-or-nothing behavior) for a transaction that spans multiple independent systems, such as multiple databases, services, or message brokers.
In other words, It makes several different systems behave as if they were performing a single, unified transaction.
Consensus-based (Paxos/Raft) guarantees strong consistency per state machine widely used in distributed DBs, config stores.

TrueTime + Paxos guarantees global ACID widely used in Google Spanner.

Microservices use:
Saga Pattern for business-level distributed workflows
Transactional Outbox Pattern for local atomicity
Idempotency + retries
These approaches trade strict ACID for eventual consistency + resilience.

Example Scenario

Suppose you have a user service that:
Stores user data in PostgreSQL.
Publishes a “UserCreated” event to Kafka.

The naive (and common) approach:

BEGIN
INSERT INTO users (id, name) VALUES (...);
SEND "UserCreated" EVENT TO Kafka;
COMMIT

If the database insert succeeds but the Kafka send fails (or vice versa), your systems become inconsistent — one reflects the change, the other doesn’t.

This is the dual write problem — trying to atomically update two systems that don’t share a transaction coordinator.
There’s no global transaction manager ensuring both operations succeed or fail together.
Failures (network issues, process crashes, retries) can easily cause partial updates.
Retrying can lead to duplicates or out-of-order events.

The Core Problem

This is the dual write problem — trying to atomically update two systems that don’t share a transaction coordinator.

There’s no global transaction manager ensuring both operations succeed or fail together.

Failures (network issues, process crashes, retries) can easily cause partial updates.

Retrying can lead to duplicates or out-of-order events.

Consequences
Failure Scenario
DB write succeeds, message send fails results ->State exists in DB but no event emitted — downstream systems never learn of it.
Message send succeeds, DB write fails results ->Event emitted for data that doesn’t exist — consumers act on invalid state.
Retry logic applied incorrectly results ->Duplicate events or multiple DB inserts.

Common Solutions / Patterns

Transactional Outbox Pattern

Write the event into the same database as the business data, in the same transaction.

A background process (or CDC tool like Debezium) later reads the “outbox” table and publishes to Kafka.

Guarantees consistency between the DB and the emitted events.

✅ Pros:

Strong consistency between DB and message.

Simple to implement if you control both DB and messaging.

🚫 Cons:

Adds complexity and operational overhead.

Requires deduplication on consumer side.

Change Data Capture (CDC)

Instead of manually writing to Kafka, rely on a CDC tool (e.g., Debezium, Oracle GoldenGate).

It monitors database changes and automatically publishes events when rows change.

✅ Pros:

No dual write logic in app.

Strong consistency if CDC is reliable.

🚫 Cons:

Possible event lag.

Requires reliable CDC infra and schema stability.

Idempotent & Retry-Safe Design

Make operations idempotent (safe to retry).

Use unique request IDs and deduplication to avoid inconsistent states even if writes are repeated.

✅ Pros:

Works across many systems.

🚫 Cons:

Still requires careful design.

Doesn’t solve ordering issues.

The Transactional Outbox Solution

Instead of writing directly to Kafka, you:

Write both the business data and the event to the same database transaction.

A background Outbox Processor (or CDC tool) later reads the outbox table and safely publishes events to Kafka.

CREATE TABLE orders (
    id UUID PRIMARY KEY,
    customer_id UUID NOT NULL,
    total NUMERIC NOT NULL,
    created_at TIMESTAMP DEFAULT now()
);

CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    event_type TEXT NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP DEFAULT now(),
    published BOOLEAN DEFAULT FALSE
);

Application Transaction (Atomic Write)

import uuid
import json
from psycopg2 import connect
conn = connect(...)
cur = conn.cursor()
order_id = uuid.uuid4()

try:
    # Both writes happen in one transaction
    cur.execute(
        INSERT INTO orders (id, customer_id, total) 
        VALUES (%s, %s, %s)
    , (order_id, "c123", 100.0))

    event = {
        "event_id": str(uuid.uuid4()),
        "type": "OrderCreated",
        "order_id": str(order_id)
    }

    cur.execute(
        INSERT INTO outbox (id, event_type, payload)
        VALUES (%s, %s, %s),(event["event_id"], event["type"], json.dumps(event)))

    conn.commit()  # ✅ both are saved atomically
except Exception as e:
    conn.rollback()
    raise e

At this point:
The order exists in the DB.
The event is stored, but not yet published to Kafka.
No dual write risk, because both were done in one atomic transaction.

Outbox Processor (Async Publisher)

A small background worker or CDC tool continuously scans for new events. If Kafka or your app crashes mid-publish, the event remains in the outbox and will be retried safely.

Goal: Prevent inconsistency when writing to a database and publishing events to a message broker (fixes the dual write problem).

Design ecommerce platform like Amazon, Flipkart

Saurav Jha — Fri, 05 Dec 2025 18:52:39 +0000

Start with functional requirements with constraint
User login/registration/manage profiles

user should be able to search the product based on product name
user should be able to view the details of products
user should be able to add product to cart
user should be able to do the checkout and payment
System should notify user once order is placed
user should be able to track the order status
manage purchase of item having limited stock

Non functional requirements

Per day order processing 100K
Platform should be highly available during increased workload
Platform should ensure consistency and prevent duplicate order/ payment.

Make security a first-class requirement

Prevent bot/ DDoS attack on the platform
Only registered/ logged in user can place order

Calculate transaction per second(TPS) roughly
Its’s critical metric in system design, impacting decisions about server capacity, database design, network infrastructure, and more.
A purchase flow typically triggers many more reads (catalog read, inventory check, pricing, user profile, coupon validation) needs to be considered for TPS calculations but can be descoped during interview discussion.

Non functional requirement
10M monthly active user
system should be highly available with respect to searching and viewing the products
and highly consistent with respect to placing the order and payment
latency ~200ms

Calculating TPS
10M daily active users
1 order per user/ daily
Transactions per Purchase: Each purchase involves 3 transactions (item selection, payment processing, order confirmation).
1,000,0000 users × 1 purchases/user/day × 3 transactions/purchase = 30 million transactions/day.

Converting to TPS:
There are 86,400 seconds in a day.
TPS = Total daily transactions / Seconds in a day.
TPS= 30 * 10 power 6 / 10 power 5 = 300 TPS

I'm approximating 86,400 seconds/day as 100k for mental calculation.

Design the architecture to satisfy these constraints

Alembic basics to advance

Saurav Jha — Fri, 08 Aug 2025 10:33:28 +0000

Alembic is a Python tool that integrates with the SQLAlchemy ORM to apply model changes to relational databases such as PostgreSQL, MySQL, and Oracle. It supports both online migrations (recommended for development and UAT environments) and offline migrations (recommended for production). However, Alembic does not support NoSQL databases like MongoDB or DynamoDB.

For example you have a db model that you would like to migrate

run the below command in the terminal
pip install alembic
alembic init migrations
This command creates a migrations directory and generates an alembic.ini file

You will find the migrations folder and the alembic.ini file created in the project directory.
Next, open the env.py file inside the migrations folder and add the DB_CONNECTION and target_metadata variables as shown in the code snippet below.

Please note: In this example, I've used SQLModel, which is a wrapper around SQLAlchemy. If you're working with SQLAlchemy directly, make sure to use SQLAlchemy's own metadata.

Once done run the below command to generate the initial revision file.
alembic revision --autogenerate -m "initial migration"

Go to migrations > version > revision file and verify the auto generated code. Resolve the error if any.

Run the below command and verify the generated tables in database
alembic upgrade head

In my case table were created successfully.

Suppose you want to remove a column from a table.
To do this, go to the database model and delete the corresponding field.
In the code snippet below, I’ve removed the gender field as an example.

Execute the following command. It will generate a revision file, as illustrated below.

alembic revision --autogenerate -m "remove gender"

Run the below command to apply the changes.
alembic upgrade head

def upgrade() -> None:
def downgrade() -> None:(to remove it on rollback)
Both the methods are crucial to perform DB upgrade and downgrade.

Scenarios Where You Use upgrade()
Adding New Tables
Example: Introducing a new projects table.
upgrade() → creates the table.
Adding a New Column
Example: Adding a salary column to the employee table.
upgrade() → uses op.add_column.
Changing Column Type or Constraints
Example: Increasing phone column size.
upgrade() → uses op.alter_column.
Adding Foreign Keys / Indexes
Example: Linking salary.employee_id to employee.id.
upgrade() → uses op.create_foreign_key.
Renaming Tables or Columns
Example: Renaming job_title to position.
upgrade() → uses op.alter_column(new_column_name=...).

Scenarios Where You Use downgrade()
downgrade() is the exact reverse of upgrade(), used when:
Rolling Back a Failed Deployment
You deployed a migration that caused an issue.
downgrade() → safely reverts to the previous state.
Reverting Experimental Features
Example: You added a gender column but later decide to remove it.
downgrade() → drops the column.
Undoing Schema Changes During Development
Example: You added an index but want to test performance without it.
downgrade() → drops the index.
Synchronizing Database With Older Code Versions
You need to roll back to an earlier release of your application.
downgrade() → brings DB schema back in sync with that release.

You add a gender column (upgrade).
Then business decides not to store gender.
You write another migration where:
upgrade() → drops gender column.
downgrade() → re-adds it (in case you need to revert).

Run the command to re-add the column.
alembic downgrade 678f33609bf3
Make sure you use down revision number

To revert one step:
alembic downgrade -1
To revert to the base (empty schema):
alembic downgrade base
Best Practice for Downgrades

Always reverse changes in the opposite order of upgrade().
If your upgrade() creates something, downgrade() should drop it.
If your upgrade() drops something, downgrade() should recreate it.
Always include foreign key handling when applicable.

Quorum and Consensus in Distributed Systems:

Saurav Jha — Sun, 13 Jul 2025 11:17:05 +0000

In distributed systems, quorum and consensus are key to ensuring consistency, availability, and fault tolerance.

Quorum: A quorum refers to the minimum number of nodes required to perform operations reliably. Typically, it is defined as the majority of nodes in a peer group. For a system with N nodes, a quorum is achieved when at least (N/2) + 1 nodes agree.

For example, in a 5-node cluster, at least 3 nodes must participate to form a quorum. If this number is not met due to node failures or network issues, the system becomes unavailable, and no new operations—such as log commits can proceed.

Consensus: Consensus is the process through which distributed nodes agree on a single, consistent state of data. It serves as the backbone for synchronization and coordination in distributed systems, ensuring that all nodes maintain the same view of shared data.

Common Use Cases of Quorum:
Distributed Databases Systems like Cassandra and DynamoDB rely on quorum-based mechanisms for consistency.

Cassandra uses the Paxos consensus algorithm to ensure that data is consistently replicated and agreed upon.

DynamoDB implements quorum-based replication to balance availability and consistency.
Learn more about Cassandra's Paxos protocol:
https://lnkd.in/gh4bnFfw

Distributed File Systems: Quorum is also crucial in file systems for ensuring consistent file operations across nodes.
Google’s Chubby lock service, which supports the Google File System (GFS), uses the Paxos consensus algorithm to maintain high availability and consistency.
Worth reading the paper on Google’s File System
https://lnkd.in/gdHwFN9r

*Popular Consensus Algorithms*
Raft
Paxos
PBFT (Practical Byzantine Fault Tolerance)
In-depth overview of the Raft algorithm
https://lnkd.in/gB4u43mi