Building an Event-Driven User Signup Flow with Redpanda and Python

This tutorial builds an event-driven signup flow with Python and Redpanda: creating, updating, and deleting a user become events on a Redpanda Cloud topic instead of direct database writes. You'll write a producer that turns CLI commands into events, a consumer that folds those events into a SQLite table, and a query script that reads that table directly — never touching Redpanda at all. That last piece is where the event-driven mental model actually clicks into place.

In a typical signup flow, one function does everything:

• Validates the input.
• Writes a row to the database.
• Maybe queues a welcome email.
• Maybe updates an analytics table.

Every time the app needs a new reaction to a signup, someone edits that same function. The database write and the side effects it triggers stay welded together.

Event-driven design breaks that weld. Instead of reacting to a signup inline, the signup handler publishes one fact: "this happened." Anything downstream — a database writer, an email service, an analytics job — subscribes to that fact on its own schedule, in its own process, without knowing the handler exists.

What you need before starting

This tutorial assumes Python — reading a CLI script and a SQLite call should feel familiar — but not event streaming. If you've never used a message broker before, that's the gap this tutorial fills:

• What an event is.
• Why a broker sits between the code that causes an event and the code that reacts to it.
• How that separation looks in real Python.

A quick note on terms: Redpanda's client library calls the object traveling through a topic a message. This tutorial calls that same object an event once its payload holds a UserEvent. The two words describe the same thing — message is the broker's vocabulary, event is this tutorial's.

Create a Redpanda Cloud cluster

Redpanda Cloud's Serverless tier removes the infrastructure step entirely — no Docker, no cluster sizing, no networking to configure.

1. Sign up for Redpanda Cloud and create a Serverless cluster. It's provisioned in a few minutes and ready to use immediately.
2. On the Topics page, create a topic named user-lifecycle — this is where every signup, update, and delete event will land.
3. On the Security page, create a SASL user with the SCRAM-SHA-256 mechanism, and save its password somewhere safe.
4. On the cluster's Overview page, copy the bootstrap server address. Every producer and consumer you write will connect through this one address.

You now have four values: the bootstrap server, a username, a password, and a mechanism name. Everything else in this tutorial builds on those four.

Configure the project

Install the single third-party dependency:

pip install -r requirements.txt

requirements.txt lists exactly one package: kafka-python. It speaks Kafka's wire protocol, which Redpanda implements, so the same client library that talks to Kafka talks to Redpanda without modification.

Copy .env.example to .env and fill in the four values from your cluster:

REDPANDA_BOOTSTRAP_SERVERS=<bootstrap-server-address>
REDPANDA_SASL_USERNAME=<your-username>
REDPANDA_SASL_PASSWORD=<your-password>
REDPANDA_SASL_MECHANISM=SCRAM-SHA-256

config.py is the only file that reads these variables. It loads them once and hands every other module a ready-made connection dict:

def kafka_common_config() -> dict:
    """Connection kwargs shared by both KafkaProducer and KafkaConsumer."""
    return {
        "bootstrap_servers": BOOTSTRAP_SERVERS,
        "security_protocol": SECURITY_PROTOCOL,
        "sasl_mechanism": SASL_MECHANISM,
        "sasl_plain_username": SASL_USERNAME,
        "sasl_plain_password": SASL_PASSWORD,
    }

Producer and consumer both call this function, so the connection settings live in exactly one place.

Define the event contract

Before writing a producer or a consumer, define the shape of the event they'll exchange. models.py holds the UserEvent dataclass, and it imports neither kafka nor sqlite3 — the event schema is a plain contract, independent of how it's transported or stored.

@dataclass
class UserEvent:
    event_type: str
    user_id: str
    payload: dict
    event_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    timestamp: str = field(default_factory=_now)

    def to_json(self) -> bytes:
        return json.dumps(asdict(self)).encode("utf-8")

    @staticmethod
    def from_json(raw: bytes) -> "UserEvent":
        data = json.loads(raw.decode("utf-8"))
        return UserEvent(**data)

event_type is one of SIGNUP, UPDATE, or DELETE. payload carries whatever data that event type needs:

• An email and name for a signup.
• A status change for an update.
• Nothing at all for a delete.

Keeping the envelope generic like this means adding a new event type later costs a new constant and a new branch in the consumer's dispatch — not a new topic or a schema migration.

Write the producer: turn a CLI command into an event

producer.py is the only place a "signup" becomes an event. It builds a KafkaProducer, then serializes and sends a UserEvent:

def build_producer() -> KafkaProducer:
    return KafkaProducer(
        **config.kafka_common_config(),
        key_serializer=lambda k: k.encode("utf-8"),
        value_serializer=lambda v: v,
        acks="all",
    )


def publish(producer: KafkaProducer, event: UserEvent) -> None:
    future = producer.send(config.TOPIC_NAME, key=event.user_id, value=event.to_json())
    record_metadata = future.get(timeout=10)
    print(f"published {event.event_type} for {event.user_id} "
          f"-> partition {record_metadata.partition}, offset {record_metadata.offset}")

Two choices in this code matter more than the rest of the file:

• Keying by user_id. Redpanda only guarantees message order within a single partition, and messages sharing a key always land on the same partition. Keying by user_id — not by event_id — keeps one user's SIGNUP → UPDATE → DELETE sequence from arriving out of order.
• acks="all". The producer waits for Redpanda's full in-sync replica set to confirm the write, not just the partition leader, before it considers the event published.

The rest of producer.py is CLI plumbing: argparse subcommands for signup, update, and delete that build a UserEvent from command-line flags and hand it to publish(). None of it touches SQLite, and none of it knows a consumer exists. That's the point — the producer's only responsibility is turning a command into an event and putting it on the topic.

Write the consumer: fold events into a read-model

consumer.py is the other half of the contract. It doesn't expose any CRUD API of its own — it subscribes to the topic and replays whatever the producer published, in order, into a SQLite table:

def apply_event(repo: UserRepository, event: UserEvent) -> None:
    if event.event_type == EventType.SIGNUP:
        user = User(user_id=event.user_id, email=event.payload["email"], name=event.payload["name"])
        repo.create(user)
    elif event.event_type == EventType.UPDATE:
        repo.update(event.user_id, event.payload, event.timestamp)
    elif event.event_type == EventType.DELETE:
        repo.delete(event.user_id)

The main loop wires a KafkaConsumer to this dispatch function:

consumer = KafkaConsumer(
    config.TOPIC_NAME,
    **config.kafka_common_config(),
    group_id=config.CONSUMER_GROUP_ID,
    auto_offset_reset="earliest",
    enable_auto_commit=False,
    key_deserializer=lambda k: k.decode("utf-8") if k else None,
    value_deserializer=lambda v: v,
)

for message in consumer:
    event = UserEvent.from_json(message.value)
    apply_event(repo, event)
    consumer.commit()

enable_auto_commit=False is the detail worth pausing on. The consumer commits its offset by hand, only after apply_event() finishes — after the row lands in SQLite. If the process crashes between reading a message and finishing the write, Redpanda still considers that message unread, and the consumer replays it on restart. This guarantee is called at-least-once delivery: it never loses an event, but it can redeliver one.

Redelivery only stays safe because repository.py writes with INSERT OR REPLACE:

def create(self, user: User) -> None:
    self.conn.execute(
        """
        INSERT OR REPLACE INTO users
            (user_id, email, name, status, created_at, updated_at)
        VALUES (?, ?, ?, ?, ?, ?)
        """,
        (user.user_id, user.email, user.name, user.status, user.created_at, user.updated_at),
    )
    self.conn.commit()

Replaying the same SIGNUP event twice overwrites a row with identical values instead of failing on a duplicate primary key. At-least-once delivery and idempotent writes are a pair: one guarantees you won't miss an event, the other guarantees seeing it twice doesn't corrupt anything.

Write the query script: read state without touching Redpanda

Everything so far has been the write path. query.py is the read path, and it's short enough to show in full:

if args.command == "list":
    users = repo.list_all()
    for user in users:
        print(f"{user.user_id}  {user.email:<30} {user.name:<20} {user.status}")
else:  # get
    user = repo.get(args.user_id)
    print(user if user else f"no user found with id {args.user_id}")

No kafka import. No producer, no consumer, no topic. query.py reads straight from the users table that consumer.py already built. This is the mental-model shift the whole tutorial has been building toward: writes travel through events so that any number of independent consumers could react to them, but reads don't need Redpanda at all. Reads just hit the materialized view the consumer already produced.

That separation is also why producer.py never imports repository.py, and why repository.py never imports kafka. The write path and the read path are two independent code paths that agree on exactly one thing — the UserEvent schema in models.py — and nothing else. A second consumer, like an email sender or an audit log, could subscribe to the same topic tomorrow without producer.py or query.py changing by a single line.

Run the flow end-to-end

Start the consumer first, in its own terminal, so it's listening before any events arrive:

python consumer.py

In a second terminal, publish a signup:

python producer.py signup --email [e-mail] --name "Ada Lovelace"

The producer's terminal prints the partition and offset the event landed on. The consumer's terminal prints confirmation that the user now exists in the SQLite table. Update and delete work the same way:

python producer.py update --user-id <user_id> --status inactive
python producer.py delete --user-id <user_id>

Then check the current state, reading straight from SQLite and not through Redpanda:

python query.py list
python query.py get --user-id <user_id>

What you built

The CRUD operations here are deliberately simple — a signup flow was never the hard part. What's worth taking away is the shape underneath this signup flow: a producer that only knows how to turn a command into an event, a consumer that only knows how to fold events into state, and a read path that skips Redpanda entirely, because reads don't depend on it. That shape scales past this tutorial's four commands — any process that needs to react to a user signing up subscribes to the same topic, on its own schedule, without ever touching producer.py.