Never call Kafka inside a payment transaction (the transactional outbox pattern)

When building a core financial ledger, the single most dangerous line of code you can write looks completely harmless:

@Transactional
public TransferResponse postTransfer(TransferRequest request) {
    // 1. Update balances & create ledger entries in Postgres
    updateBalances(request);
    createLedgerEntries(request);

    // 2. Publish event to Kafka
    kafkaTemplate.send("transfers.posted", new TransferEvent(request)); // ❌ DISASTER

    return response;
}

It looks clean. It passes unit tests. And in production, it will inevitably corrupt your financial system.

Here is the story of how I encountered the classic Dual-Write Problem while designing LedgerX - a double-entry ledger engine - and how implementing the Transactional Outbox Pattern gave me verifiable, crash-resilient exactly-once semantics.

---

The Dual-Write Trap

In a distributed financial system, you need two things to happen when money moves:

1. Mutate state in PostgreSQL (balances, immutable journal entries, audit log).
2. Broadcast an event to Apache Kafka so downstream notification services, fraud analysis, and data warehouses react.

The fatal trap is that PostgreSQL transactions and Kafka publishes cannot share an atomic ACID transaction.

Consider the failure modes:

• Scenario A: The Kafka broker hangs or throws a network timeout. The @Transactional block catches the exception and rolls back the database. But what if Kafka actually received the bytes right before the network dropped? Your event was published, but the money never moved.
• Scenario B: You publish to Kafka at the end of the method. The Kafka send succeeds, but when Hibernate issues the final COMMIT to PostgreSQL, a database constraint fails or the connection drops. You just broadcasted a payment that does not exist in the ledger.
• Scenario C: Kafka is temporarily down for maintenance. Your entire payment API returns HTTP 500, rejecting transfers even though your database is completely healthy.

Kafka is never called inside the transfer transaction.

---

The Solution: The Transactional Outbox Pattern

The Transactional Outbox Pattern decouples your database commit from message delivery by storing the message directly inside PostgreSQL as an append-only table.

Because the outbox row is written inside the same database transaction as the financial transfers and ledger entries, it is protected by the database's ACID guarantees. Either everything commits together, or nothing does.

sequenceDiagram
    autonumber
    actor Client
    participant API as LedgerX API
    participant DB as PostgreSQL (ACID)
    participant Relay as Outbox Relay Worker
    participant Kafka as Apache Kafka Broker
    participant Consumer as Audit Consumer

    Client->>API: POST /api/v1/transfers (Idempotency-Key)
    activate API
    API->>DB: BEGIN TRANSACTION
    API->>DB: Lock accounts (ORDER BY id FOR UPDATE)
    API->>DB: Insert Transfer & Balanced Entries
    API->>DB: Insert outbox row (published_at = NULL)
    API->>DB: COMMIT TRANSACTION
    API-->>Client: 201 Created (Transfer Posted)
    deactivate API

    loop Every 500ms
        activate Relay
        Relay->>DB: SELECT * FROM outbox FOR UPDATE SKIP LOCKED
        Relay->>Kafka: Produce event (TransferPosted.v1)
        Kafka-->>Relay: Broker ACK
        Relay->>DB: UPDATE outbox SET published_at = NOW()
        deactivate Relay
    end

    Kafka->>Consumer: Consume event
    activate Consumer
    Consumer->>DB: INSERT INTO processed_events ON CONFLICT DO NOTHING
    Consumer->>DB: Append immutable audit record
    deactivate Consumer

---

3 Critical Engineering Decisions in LedgerX

Implementing the pattern in production requires more than a naive table and polling loop. Here are three architectural choices that made LedgerX bulletproof:

1. Concurrency with FOR UPDATE SKIP LOCKED

If you run multiple instances of your application, multiple outbox pollers will fight over the same pending rows.

Instead of external distributed locks (like Redis/Redlock), LedgerX uses PostgreSQL's native row locking:

SELECT id, aggregate_type, aggregate_id, event_type, payload, headers, created_at, published_at, attempts, last_error, next_attempt_at
FROM outbox
WHERE published_at IS NULL
  AND next_attempt_at <= NOW()
ORDER BY created_at ASC
LIMIT 50
FOR UPDATE SKIP LOCKED;

SKIP LOCKED allows instance A to lock rows 1 to 50 while instance B concurrently processes rows 51 to 100 without blocking or waiting.

2. Durable Exponential Backoff in SQL

If Kafka goes down, an unconstrained relay will spin in a tight CPU loop, flooding logs with connection errors.

LedgerX stores retry metadata directly in the outbox table (V8__outbox_next_attempt_at.sql):

ALTER TABLE outbox 
  ADD COLUMN attempts INT NOT NULL DEFAULT 0,
  ADD COLUMN last_error TEXT,
  ADD COLUMN next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW();

When a publish fails, the delay expands exponentially: $\text{delay} = \min(300, \text{attempts}^2)$ seconds. The poller query simply excludes rows where next_attempt_at > NOW(), letting failing rows rest while fresh transactions continue flowing.

3. Delivery Guarantees: At-Least-Once Outbox and Exactly-Once Effect

The outbox pattern guarantees at-least-once publishing. Network partitions or restarts during broker acknowledgment can cause a message to be published more than once. Kafka is never called inside the transfer transaction.

To achieve an end-to-end exactly-once processing effect, consumers track processed message IDs using an atomic deduplication table:

INSERT INTO processed_events (event_id, consumer_group, processed_at)
VALUES (?, 'audit-consumer', NOW())
ON CONFLICT (event_id, consumer_group) DO NOTHING;

If the row already exists, the consumer immediately ACKs Kafka and skips side effects.

---

Conclusion

By moving event publication out of the HTTP transaction and into a durable database-backed outbox:

• Payment requests succeed even when Kafka is completely offline.
• Data never goes out of sync between PostgreSQL and event streams.
• The system achieves ironclad consistency without complex distributed transaction protocols.

You can inspect the full Java 21 implementation, Flyway migrations, and Testcontainers test suite on GitHub: github.com/Akinmoldun/LedgerX.