DEV Community

Cover image for Transactional Outbox Pattern: Memory Queue for Speed, PostgreSQL for Recovery
Valeriy
Valeriy

Posted on

Transactional Outbox Pattern: Memory Queue for Speed, PostgreSQL for Recovery

#springboot #java #distributedsystems #systemdesign

In distributed systems, the Transactional Outbox Pattern is a common
way to reliably publish events after a database transaction.

The idea is simple: business data and an outbox event are saved in the
same database transaction. If the transaction is committed, the event is
safely stored and can be published to Kafka.

But saving the event is only half of the problem.

How do we move it from PostgreSQL to Kafka quickly and reliably?

A common answer is database polling. A worker checks the outbox table
again and again, finds new events, and publishes them.

It works well. But there is a trade-off:

the lower delivery latency we want, the more often we need to poll the
database.

So I started experimenting with another approach:

What if PostgreSQL keeps the durable event, but memory becomes the
fast path for delivery?

This leads to a simple model:

Memory Queue for speed. PostgreSQL for durability and recovery.

That is the idea I want to explore in this article.

The usual approach: poll the outbox table

A simple Transactional Outbox flow can look like this:

Transactional Outbox Pattern

The application saves business data and the outbox event in one
transaction.

Then a worker periodically asks PostgreSQL:

Do we have new events?

For example, it can run every second.

This is a good solution, but there is a trade-off:

Transactional Outbox Pattern

If we poll every 10 seconds, the database load is smaller, but an event
may wait several seconds before publishing.

If we poll much more often, latency gets better, but PostgreSQL receives
more queries and uses more database connections.

Of course, polling can be optimized with batching, indexes,
SKIP LOCKED, partitioning, and longer polling intervals.

There are also other solutions, such as PostgreSQL LISTEN / NOTIFY or
Debezium CDC.

For many systems, Debezium is probably the right choice. But it also
adds Kafka Connect, Debezium, and more infrastructure to operate.

I wanted to test a simpler application-level approach.

Memory Queue as the fast path

The normal flow in my implementation looks like this:

Transactional Outbox Pattern

The important part starts after the commit.

The application does not wait for the next polling cycle.

After the transaction is committed, only the event ID is added to the Memory Queue — not the full event payload. The Batch Publisher takes these IDs, loads the corresponding events from PostgreSQL in batches, and sends them to Kafka.

PostgreSQL keeps the event. Memory keeps only the work to be processed.

Memory is the fast path. PostgreSQL is the durable source of
truth.

Important: polling is not gone

One important detail: I did not remove polling completely.

I moved it from the normal path to the recovery path.

During normal operation, committed event IDs go directly to the Memory Queue. There is no need to scan PostgreSQL again and again just to discover new events.

Recovery works differently. A background worker periodically scans PostgreSQL for events that were not successfully processed by the fast path.

But the Recovery Worker does not publish these events directly to Kafka. It only puts their IDs back into the same Memory Queue.

From that point, both flows use the same pipeline:

Memory Queue → Batch Publisher → Kafka

This is an important part of the design. Recovery is not a second delivery system. It only finds unfinished work and returns it to the normal publishing flow.

Different entry points. Same publishing pipeline.

Transactional Outbox Pattern
There is a price for moving polling to recovery.

The recovery interval becomes part of the worst-case delivery latency when the fast path fails.

Very roughly:

Normal latency
= queue wait + batch publishing

Recovery latency
= recovery interval + queue wait + batch publishing
Enter fullscreen mode Exit fullscreen mode

For example, if normal polling would run every second and recovery runs every 10 seconds, PostgreSQL receives much fewer polling queries during normal operation.

But an event that needs recovery may also wait longer.

So both happy-path latency and recovery latency should be measured.

Memory is not durable — and that is intentional

At first, using an in-memory queue for reliable event delivery sounds dangerous.

And it would be dangerous if memory were the only place where the event existed.

But it is not.

Consider this case:

1. Order saved
2. Outbox event saved
3. Transaction committed
4. Application crashes
5. Memory Queue is lost
Enter fullscreen mode Exit fullscreen mode

Did we lose the event?

No.

We lost the fast path, but not the event.

The complete event is still stored in PostgreSQL. After restart, the Recovery Worker finds the unpublished event and puts its ID back into the Memory Queue.

This is also why I keep only the eventId in memory. The durable state always stays in PostgreSQL.

The Memory Queue can disappear. The event cannot.

What if Recovery finds an event already in memory?

This is an important race to think about.

Imagine this situation:

  1. The transaction is committed.
  2. eventId = 1421 is already in the Memory Queue.
  3. The event has not been published yet.
  4. At the same time, the Recovery Worker sees the same unpublished event in PostgreSQL.

Can it publish the event twice?

The important detail is that the Recovery Worker does not publish
directly.

It can only put the event ID back into the queue.

Normal Flow ---- eventId 1421 ----+
                                  |
                                  v
                            Memory Queue
                                  |
                                  v
                            DB claim/lock
                                  |
                                  v
                         Batch Publisher
                                  |
                                  v
                                Kafka
                                  ^
                                  |
Recovery ------- eventId 1421 ----+
Enter fullscreen mode Exit fullscreen mode

The queue deduplicates IDs and tracks in-flight events. After that, the
database lease protects the actual publish claim.

So the race is handled before it can become two independent publishing
flows.

Still, I do not treat this as exactly-once delivery.

Duplicate delivery is always something we should expect in an
event-driven system.

The Kafka consumer should be idempotent too.

In this project, the Notification Service uses eventId for
consumer-side idempotency, so replaying the same event does not repeat
the business operation.

Recovery can rediscover work. It should not create a second
publishing path.

Keeping recovery scans small

There is another problem with recovery.

Imagine the outbox after a long time:

10,000,000 events
Enter fullscreen mode Exit fullscreen mode

Maybe only 20 or 50 events are currently unfinished.

Recovery should not care about millions of already published events.

For this reason, the project separates active and archived data.

Conceptually:

ACTIVE

NEW
PROCESSING
FAILED

    |
    | successful publish
    v

ARCHIVE

SENT
Enter fullscreen mode Exit fullscreen mode

The Recovery Worker works only with unpublished events in the ACTIVE
partition.

Successfully published events move to ARCHIVE and are no longer part of
recovery scans.

The idea is simple:

Recovery cost should depend on unfinished work, not on the complete
history of the outbox.

This becomes more important as the outbox grows.

Trade-offs

This approach has its own trade-offs.

The main one is recovery latency. The normal path is fast because it does not wait for database polling. But if the fast path is interrupted, delivery depends on the Recovery Worker and its scan interval.

The second trade-off is that the Memory Queue is local to each application instance. PostgreSQL remains the shared source of truth, while database claiming and locking coordinate recovery when multiple instances are running.

And this is still an at-least-once delivery model. An event may be delivered more than once, so idempotency is required. In this project, idempotency is implemented on both sides: for incoming requests and for Kafka event processing.

Finally, this approach is not intended to replace Debezium in every system. If CDC and Kafka Connect are already part of your platform, Debezium may be a better choice.

The goal here is different: keep the normal delivery path fast and simple, while PostgreSQL provides durability and recovery when something goes wrong.

I wanted to test it, not only draw it

Architecture diagrams are useful.

But I wanted to see what actually happens with the queue, PostgreSQL,
publishing latency, recovery, and failures.

So the project grew beyond the first Memory Queue experiment.

It now includes:

  • Spring Boot;
  • PostgreSQL;
  • Kafka;
  • batch publishing;
  • crash recovery;
  • producer and consumer idempotency;
  • rate limiting;
  • Gatling load tests;
  • Prometheus metrics;
  • Grafana dashboards;
  • OpenTelemetry tracing;
  • Grafana Tempo;
  • structured logging with OpenSearch;
  • Docker Compose environments.

I also created three implementations of the same main flow:

Implementation Stack
Servlet Spring MVC + JDBC
Reactive WebFlux + R2DBC
Virtual Threads Spring MVC + JDBC + Virtual Threads

This gives me a way to run similar load tests against different Spring
models and compare their behavior.

That experiment deserves a separate article.

Some parts became reusable Spring Boot starters

While working on the project, some infrastructure code became useful
outside this example too.

So I moved it into separate Spring Boot starters.

Transactional Outbox Starter

https://github.com/KHolodilin/spring-boot-outbox-starter

Idempotency Starter

https://github.com/KHolodilin/spring-boot-idempotency-starter

The main project uses these starters in the Servlet and Virtual Threads
implementations.

The Reactive implementation keeps its own R2DBC-based flow.

I will not go deep into the starters here because this article is mainly
about the Memory Queue + Recovery idea.

Try the project

Transactional Outbox Pattern Try the project

If you want to see how this works in code, including recovery, batching,
observability and load testing, here is the project:

https://github.com/KHolodilin/spring-transactional-outbox-kafka

You can run the stack with Docker Compose and experiment with the flow
yourself.

If you find the idea useful, feel free to ⭐ the repo or fork it.

There are also a few open issues for contributors if you want to try
something yourself.

Any feedback is welcome. I'm still experimenting with the approach and
improving the project. 🚀

What would you do differently?

I am especially interested in the trade-offs of this approach.

Would you use normal database polling, Debezium, PostgreSQL
LISTEN / NOTIFY, or something similar to this Memory Queue + Recovery
model?

And more interesting:

What failure scenario would you test first?

If you find a weak point in the implementation, feel free to open an
issue. I would be interested to test it.

Top comments (0)