Event-driven architecture: events, outbox and idempotency

Part 3 of 8 · week 3 of the learning path

In an event-driven system a service records a fact that has happened and others react to it on their own time, which absorbs spikes and lets you add consumers without touching the producer.

Learning objectives

• Tell events, commands and queries apart and name them correctly.
• Choose between a queue and a log, and design for duplicates and ordering.
• Apply the outbox, idempotent consumer, dead-letter and CQRS patterns.

Concepts

Concept	Meaning	Example name
Event	A fact that already happened, named in the past tense	OrderPlaced
Command	A request to a specific service to do something	ChargeCard
Query	A request for data	GetOrder
Queue	Each message goes to one consumer and is removed	Work distribution
Log	Ordered, retained stream that many consumers read at their own pace	Event streaming, replay
Consumer group	Instances that share the partitions of a topic	Scale out a consumer

Delivery and ordering

• Most systems deliver at least once, so a message can arrive twice. Make handlers idempotent, for example by recording processed event IDs, so duplicates do no harm.
• Ordering is only guaranteed within a partition. Use a key such as the order ID so all events for one order land in the same partition.
• Move messages that keep failing to a dead-letter queue after limited retries, and alert on it.
• Version event schemas, keep changes backward compatible and register them so consumers can evolve safely.

Patterns

• Outbox: write the data change and the event in one database transaction, then a relay publishes the event, so you never update data without publishing or the reverse.
• Saga: coordinate multi-step workflows through events, with compensating events on failure.
• CQRS: keep the write model separate from one or more read-optimized models that are updated from events.
• Event sourcing: store the sequence of events as the source of truth and rebuild state from it; powerful for audit, but adds complexity.

Worked example, order flow. The Order service saves the order and writes OrderPlaced to its outbox. Inventory reacts and emits StockReserved. Payment reacts and emits PaymentCaptured. Shipping starts only after that. Later the team adds an email service and an analytics pipeline as new consumers of the same events, with no change to Order.

Pitfalls: treating events as remote calls that expect a reply, event chains nobody can follow without tracing, breaking schema changes, assuming global ordering, and putting huge payloads in events instead of references.

Real-world examples

• LinkedIn and Kafka: LinkedIn built Kafka around 2010 to move activity and operational data between systems at scale, and open sourced it; it became the standard log for event streaming.
• Stripe webhooks: Stripe tells merchants about payments by sending events to their endpoint, retrying on failure, so merchants must handle duplicates and out-of-order delivery. It is a public example of at-least-once delivery.
• Netflix: it streams very large volumes of operational and viewing events through a central pipeline for monitoring and analytics, decoupling the producers from the many teams consuming them.

Design exercise: design the events for a food delivery order (placed, accepted, picked up, delivered, refunded). Choose partition keys, say where you need an outbox, and describe what a consumer does on a duplicate and on an out-of-order event.

Quiz 3 (5 questions, pass mark 4)

1. Which of these is a command rather than an event?
   • A) OrderPlaced
   • B) PaymentCaptured
   • C) ChargeCard
   • D) UserRegistered
2. What does at-least-once delivery mean for a consumer?
   • A) A message may arrive more than once, so handlers must be idempotent
   • B) Duplicates never happen
   • C) Messages may be lost
   • D) Exactly one copy is always delivered
3. What does the outbox pattern guarantee?
   • A) Messages are encrypted
   • B) Ordering across all topics
   • C) Faster consumers
   • D) The data change and the event are committed together and the event is then published reliably
4. How do you keep all events for one order in order on a partitioned log?
   • A) Use a random key
   • B) Use the order ID as the key so they share a partition
   • C) Create a topic per event
   • D) Make messages bigger
5. What is CQRS?
   • A) A type of queue
   • B) A database index
   • C) Separating the write model from one or more read-optimized models, often updated by events
   • D) A caching protocol

Quiz 3 answers

1. C
2. A
3. D the outbox writes the data change and the event in one transaction, so neither can be lost without the other.
4. B
5. C

Sources and further reading

• Apache Kafka documentation, starting with the introduction to event streaming.
• The Log: what every software engineer should know about real-time data's unifying abstraction by Jay Kreps of LinkedIn, on why a log underlies messaging and replication.
• Transactional outbox pattern on microservices.io.
• Stripe webhooks documentation, a public example of at-least-once event delivery with retries and signatures.
• Amazon Builders' Library: timeouts, retries, and backoff with jitter, for safe retries and idempotent APIs.

---

Part 3 of 8 in the System Design Learning Path series. Next: part 4.