Event-driven architecture looks simple at first.
Your application performs a business operation, publishes an event to Kafka or SQS, and another service consumes it.
@Transactional
public void createOrder(CreateOrderCommand command) {
Order order = orderRepository.save(...);
kafkaTemplate.send(
"orders",
new OrderCreatedEvent(order.getId())
);
}
Looks reasonable.
The order is saved, an OrderCreatedEvent is published, and other services can react to it.
But there is a problem hiding in those few lines:
What happens if the database transaction succeeds, but publishing the event fails?
And on the consumer side:
What happens if the same event is delivered twice?
These questions lead to several patterns that become essential once event-driven systems move beyond simple demos:
- Transactional Outbox
- Inbox Pattern
- Idempotent Consumers
- Durable Retries
- Multi-instance-safe processing
- Observable failure handling
Let's build the architecture step by step.
The Dual-Write Problem
Imagine an order service that needs to:
- Save an order to PostgreSQL.
- Publish
OrderCreatedto Kafka.
Conceptually:
Database ──────> COMMIT ✓
|
Kafka ─────────> PUBLISH ✗
The database and Kafka are independent systems.
A successful database commit does not guarantee a successful Kafka publish.
Consider:
1. INSERT order
2. COMMIT
3. Publish OrderCreated
4. Application crashes
If the application crashes between steps 2 and 3, the order exists but the event doesn't.
Other services may never know that the order was created.
Reversing the operations doesn't fix it:
1. Publish OrderCreated
2. INSERT order
3. Database transaction fails
Now consumers may receive an event for an order that doesn't exist.
This is the classic dual-write problem.
The Transactional Outbox Pattern
Instead of trying to atomically update the database and message broker, persist the event as part of the same database transaction as the business operation.
Database Transaction
┌─────────────────────────────┐
│ │
Request ──> Business Data │
│ + │
│ Outbox Event │
│ │
└────────── COMMIT ───────────┘
|
v
Outbox Dispatcher
|
v
Kafka / SQS
When an order is created, we persist:
ORDER
+
OUTBOX EVENT
inside the same transaction.
Either both commit or neither commits.
A separate dispatcher finds pending outbox records and publishes them to the broker.
A simplified lifecycle could look like:
PENDING
|
v
PROCESSING
|
+──── success ────> PUBLISHED
|
└──── failure ────> RETRY / FAILED
This removes the dangerous database-and-broker dual write from the business transaction.
It also gives us something extremely useful:
persistent delivery state.
Make the Outbox Debuggable
Reliability isn't only about retrying failed operations.
Eventually, someone will need to answer:
What happened to this event?
A useful outbox should contain enough information to answer that question:
eventId
eventType
source
correlationId
payload
status
attempts
createdAt
availableAt
publishedAt
lastError
Instead of reconstructing everything from distributed logs, an engineer can inspect the actual state:
SELECT *
FROM event_outbox
WHERE status = 'FAILED';
This leads to an important principle:
Reliability mechanisms should also improve debuggability.
If an event cannot be delivered, that failure should be visible and inspectable.
Reliable Publishing Is Only Half the Problem
Suppose our outbox works perfectly.
Every event eventually reaches Kafka.
We're still not finished.
Most event-driven systems operate with at-least-once delivery.
That means the same event may arrive more than once.
Producer
|
v
Kafka
|
+──── OrderCreated #123 ────> Consumer
|
+──── OrderCreated #123 ────> Consumer
A consumer could successfully process an event and then crash before acknowledging it.
The broker delivers the event again.
If the handler sends an email, perhaps the customer receives two emails.
If it performs a payment operation, the consequences can be considerably worse.
Consumers therefore need to assume:
Every event can arrive more than once.
The Inbox Pattern
The Inbox Pattern provides a durable record of received events.
Before processing an event, the consumer registers its unique event ID.
Broker
|
v
Inbox Registration
|
+── event already exists ──> DUPLICATE
|
└── new event
|
v
RECEIVED
|
v
PROCESSING
/ \
v v
PROCESSED FAILED
If the same eventId arrives again, the consumer knows that it has already seen it.
The event ID becomes an idempotency boundary.
Instead of depending on exactly-once delivery, we make duplicate delivery safe.
The Inbox Is More Than a Deduplication Table
A minimal inbox could contain nothing more than processed event IDs.
In a production system, however, it can become a durable history of event processing.
Consider storing:
eventId
eventType
source
correlationId
payload
status
attempts
receivedAt
processedAt
availableAt
lastError
Now imagine an incident:
Order 123 was created, but the downstream action never happened.
You can inspect the inbox.
Was the event received?
Did processing start?
Did the handler fail?
How many attempts were made?
When is the next retry?
What was the last error?
Those questions become much easier to answer when processing state is explicit.
Retries Should Survive Application Restarts
Spring provides excellent retry mechanisms.
For example:
@Retryable
public void handle(OrderCreatedEvent event) {
...
}
This can be perfectly appropriate for short-lived transient failures.
But durable event processing introduces another question:
What happens if the JVM dies?
Event processing fails
|
v
Retry scheduled in memory
|
v
Application restarts
If retry state exists only in memory, it disappears with the process.
For critical event processing, retry state can instead be persisted:
FAILED
|
| availableAt <= now
v
PROCESSING
|
+──── success ────> PROCESSED
|
└──── failure ────> FAILED
|
+ attempts++
+ availableAt = next retry
A scheduler periodically finds events whose retry time has arrived.
Because the state lives in the database, restarting the application doesn't destroy the retry information.
Back Off Instead of Hammering a Failing Dependency
Retrying continuously can make an outage worse.
If a downstream service is unavailable, thousands of failed events retrying as quickly as possible only add pressure.
A better strategy is exponential backoff:
Attempt 1 → immediate
Attempt 2 → +1 second
Attempt 3 → +2 seconds
Attempt 4 → +4 seconds
Attempt 5 → +8 seconds
Eventually, the configured retry limit is exhausted.
At that point, the event can remain explicitly failed:
status = FAILED
availableAt = null
Automatic retries stop.
But importantly, the failure doesn't disappear.
The event remains available for investigation and operational recovery.
What About Dead-Letter Queues?
Dead-letter queues are useful, particularly for broker-level failures.
But the broker's DLQ doesn't necessarily need to become the application's primary record of processing failure.
There is a useful distinction:
Broker concern Application concern
Delivery failure Processing failure
Malformed message Business handler failure
Transport problem Retry exhaustion
| |
v v
DLQ INBOX
The two mechanisms can coexist.
A database-backed inbox gives the application direct visibility into its own processing state, while a DLQ remains available for appropriate broker- and transport-level failures.
Then You Deploy Multiple Pods
Everything becomes more interesting once the application runs more than one instance.
OUTBOX
|
pending event
|
┌──────────┴──────────┐
v v
Pod A Pod B
Dispatcher Dispatcher
Both instances may discover the same pending event.
Without concurrency control, both may attempt to process it.
Production implementations therefore need a concept of claiming or locking.
For example:
status
lockOwner
lockedAt
An instance claims records transactionally before processing them.
Other instances can then determine that those records are already being handled.
But this creates another question:
What happens if a pod claims an event and then dies?
The system needs a deterministic mechanism for recovering stale claims after an appropriate timeout.
At this point, the outbox is no longer just a database table plus a scheduled query.
It has become infrastructure.
Even the Scheduler Can Fail
There is another failure mode that is surprisingly easy to overlook.
Suppose the dispatcher completes successfully and schedules its next execution:
Dispatcher completes
|
v
schedule(nextRun)
|
X
TaskScheduler rejects the task
If the scheduler continues reporting itself as running, the application has entered a dangerous state.
Everything appears healthy.
But no future dispatch will happen.
Events can quietly accumulate in the outbox.
A reliable scheduler therefore benefits from an explicit lifecycle:
STOPPED
|
v
STARTING
|
v
RUNNING
|
+──── scheduling failure ────> FAILED
A useful invariant is:
A scheduler must not report itself as running if no task is scheduled and no work is currently executing.
Scheduling infrastructure itself needs observable failure semantics.
Observability Is Part of Reliability
Imagine receiving a production incident at 2 AM:
We created the order, but the downstream system didn't process it.
Ideally, you should be able to follow the event:
Order
|
v
Outbox Event
|
+── created
+── claimed
+── publish attempts
+── published
|
v
Broker
|
v
Inbox Event
|
+── received
+── processing attempts
+── failure reason
+── retry schedule
+── processed
Correlation metadata should connect the pieces.
Payloads should be readable.
State transitions should be explicit.
Failures should remain inspectable.
Logs should explain what the infrastructure is doing without becoming the only source of truth.
A reliable system isn't only one that recovers from failures. It is one that helps engineers understand those failures.
Putting It All Together
Once these pieces are combined, the architecture starts looking like this:
Business Transaction
|
v
Transactional Outbox
|
v
Outbox Dispatcher
|
v
Kafka / SQS
|
v
Inbox
|
v
Idempotency Check
|
v
Event Handler
|
+── Success
|
└── Durable Retry
Underneath it all is persistent state:
Persistence
├── Outbox delivery state
├── Inbox processing state
├── Attempts
├── Retry scheduling
├── Lock ownership
└── Failure information
That's quite a bit of infrastructure around what originally looked like:
kafkaTemplate.send(...);
From Architecture to Implementation: NERV Event
These are the problems I wanted to solve consistently across Spring Boot applications.
None of the individual patterns are new.
Transactional outbox is well understood.
Idempotent consumers are well understood.
Retries, locking, and message brokers are well understood.
The difficult part is making them work together consistently in a production application.
That's what led me to build NERV Event.
NERV Event is an open-source event infrastructure library for Spring Boot that implements the architecture described in this article.
It brings together:
- Transactional outbox persistence
- Durable inbox processing
- Idempotent event consumption
- Persistent retries
- Exponential retry policies
- Multi-instance-safe processing
- Kafka integration
- AWS SQS integration
- Scheduler lifecycle and failure visibility
- Event retention
- Operational inspection
- Correlation metadata
- Human-readable persisted payloads
The goal isn't to hide event-driven architecture behind magic.
The goal is to make its behavior predictable, observable, and easy to debug.
NERV Event
The project is open source:
The repository contains the source code, documentation, configuration, and examples.
Even if you don't use the library directly, I hope the architecture and implementation can be useful as a reference when designing reliable event-driven Spring Boot applications.
I'll cover individual parts of NERV Event in future articles, including transactional publishing, inbox processing, retries, Kafka and SQS integration, multi-pod deployments, and operational tooling.
Final Thoughts
Adding Kafka or SQS to a Spring Boot application doesn't automatically make the application reliably event-driven.
The difficult parts exist around the broker:
Business Transaction
|
v
Transactional Outbox
|
v
Reliable Delivery
|
v
At-Least-Once Messaging
|
v
Inbox + Idempotency
|
v
Durable Processing
|
v
Retries + Recovery
|
v
Observability
Each layer addresses a different failure mode.
And in distributed systems, those failures aren't theoretical.
Processes restart. Networks fail. Messages are redelivered. Dependencies become unavailable. Schedulers fail. Multiple instances compete for the same work.
The objective isn't to pretend these failures won't happen.
It's to design the system so that when they do:
State is preserved. Recovery is predictable. And engineers can understand exactly what happened.
That's the philosophy behind NERV Event.
Top comments (1)
The outbox pattern is essential, but the piece teams often underestimate in production is polling table contention under sustained load.
If multiple worker instances poll the outbox table with naive "SELECT ... FOR UPDATE", row lock escalation can quickly degrade the primary write throughput of the application. Using "FOR UPDATE SKIP LOCKED" (or streaming outbox changes via database CDC instead of periodic polling) keeps event publishing completely decoupled from transactional latency.