How to scale Kafka maintaining Order of events

Intro: The Sequential Processing Paradox

Some of the business processes requires events to be processed in order e.g. In e-commerce, the order of events is important. You cannot process OrderFulfilled before OrderCreated or PaymentAuthorized.

If we have single partition, Kafka does ensure chronological order. However, this significantly affects the performance, since we cannot configure multiple consumers to do processing in parallel. On the other hand, if we have multiple partitions, then we would lose the processing order, since Kafka does not guarantee order across partitions.

Standard Kafka advice is to use multiple partitions and use a Message Key (like order_id) to ensure all related events land in the same partition in chronological order. With this approach we can ensure that events related to same key (like order_id) are processed in order.

However, if we use this approach, we still need to follow some best practices to make this solution robust. Also, quite often we use the Transactional Outbox (For Data Consistency) and Inbox pattern (Idempotent processing) along with Kafka, we need to make sure that Outbox and Inbox retain the order.

---

The Concepts

To maintain order at scale, we must move beyond a simple "Kafka-only" view to a three-stage architectural safety chain.

1. The Transactional Outbox

The Outbox pattern works by saving your business data (the Order) and the event message (the Intent) into the same database in a single, atomic step. A separate "Relay" process then polls this table and publishes the messages to Kafka, ensuring that if your database update succeeds, your message eventually follows.

How it Can Mess Up the Order
If your Relay process picks up messages and tries to send them in parallel without a strategy, OrderUpdated might accidentally reach Kafka before OrderCreated. Similarly, if a message fails and you move to the next one before retrying the first, you’ve broken the chronological chain. Without using a Message Key (like order_id) during this relay step, Kafka cannot guarantee that related events stay in the same "line" for the consumer.

What to Ensure for Success

Sequential Polling: Ensure your Relay reads from the Outbox table using a strict ORDER BY (usually a sequence ID or timestamp) so events are sent exactly as they occurred.

Deterministic Partitioning: Always attach a consistent Business Key to every message so Kafka routes them to the same partition every time.

Max In-Flight Requests: Limit the producer to only one "unconfirmed" request at a time (or use Kafka’s idempotent producer settings) to prevent a network retry from flipping the order of two messages.

Clean-up Logic: Implement a "Janitor" task to delete processed rows from the Outbox table; a bloated table will slow down your Relay and eventually delay your entire event stream.

2. Keyed Partitioning

By using order_id as the Kafka Key, we ensure the "Log" for that specific order is physically sequential. Kafka handles the heavy lifting of keeping v1 ahead of v2 within the partition.

3. The Sub-Inbox

• Why we use an Inbox first
  The Inbox acts as a "buffer" between Kafka and your business logic. By saving the message to a database table immediately and acknowledging Kafka, you ensure the message is never lost if your service crashes, and you keep the Kafka partition moving at high speed without waiting for slow external APIs.

• How a "Sub-Inbox" differs
  In a standard Inbox, you often only need a simple status and received_at timestamp. In a Sub-Inbox, the table must be "Aware" of the specific business entity it is processing.

Also, in Standard Inbox, you have one worker (or a very small number) that pulls the oldest message, processes it, and moves to the next. If the first message is slow, the worker waits, and the whole table sits idle. In Sub-Inbox you have a pool of many workers. The "Order" is preserved because we use database locking as the gatekeeper.

• How the Worker Pool Operates

Imagine a team of 10 workers standing in front of a giant digital task board (the Inbox Table). Instead of the first worker taking the top 10 tasks, each worker acts independently but follows a shared logic:

The Selective Search: A worker looks at the table and asks: "Give me the oldest pending message, but ignore any message whose order_id is already being processed by another worker."

The Instant Lock: Once the worker finds an available order_id (e.g., Order-101), it immediately places a "Lock" on that ID in the database.

The Parallel Leap: While Worker 1 is busy with Order-101, Worker 2 looks at the board. It sees Order-101 is locked, so it instantly "skips" over it and grabs Order-102.

The Release: Only when Worker 1 completely finishes the task and deletes (or marks "Complete") the row for Order-101 is that ID "unlocked." Now, if there is a second message for Order-101 (like a shipping update), the next available worker can safely grab it.

• Things to Ensure

The "Skip Locked" Command: In your code, you must use a specific database command (like SELECT ... FOR UPDATE SKIP LOCKED). Without the "SKIP LOCKED" part, Worker 2 would sit and wait for Worker 1 to finish, which defeats the whole purpose of having a pool.

Transaction Boundaries: The "Lock" must live inside a database transaction. The worker should: Start Transaction → Lock & Get Task → Process Task → Update Status → Commit Transaction. If anything fails in the middle, the transaction rolls back and the lock disappears, keeping the data safe.

---

Important Decisions & Future-Proofing

1. Partition Strategy: Planning for Growth

Partitions are your unit of parallelism. You cannot easily increase partition counts later without changing the hash result of your keys, which breaks the sequence.

• Guidance: Over-partition from Day 1. If you need 10 now, create 60. This allows you to scale your consumer group up to 60 instances without a complex data migration.

2. Decision Matrix: Choosing Your Strategy

Requirement	Raw Kafka Key	Standard Inbox	Sub-Inbox (Parallel)
Ordering Scope	Per Partition	Per Key	Per Key
Concurrency	1 Thread/Partition	1 Thread/Service	N Threads/Service
Failure Impact	Blocks Partition	Blocks Service	Blocks ONE Order Only
Complexity	Low	Medium	High

---

Technical Guide: Spring Boot Implementation

Producer: Transactional Outbox

@Transactional
public void placeOrder(OrderDTO dto) {
    // 1. Persist Business State
    Order order = orderRepo.save(new Order(dto));

    // 2. Persist Event to Outbox in the same transaction
    outboxRepo.save(new OutboxEvent(
        order.getId(), 
        "ORDER_CREATED", 
        json(order)
    ));
}

Consumer: Inbox + Manual Ack

Set ack-mode: manual_immediate in your application.yml.

@KafkaListener(topics = "orders", groupId = "fulfillment-service")
@Transactional
public void onMessage(ConsumerRecord<String, String> record, Acknowledgment ack) {
    // Unique constraint on (order_id + version) handles idempotency
    inboxRepository.save(new InboxEntry(
        record.key(), 
        record.value(), 
        record.offset()
    ));

    // Acknowledge ONLY after the DB transaction commits
    ack.acknowledge(); 
}

Retries and the "Problem Bin" (DLQ)

In an ordered flow, moving a failed PaymentAuthorized message to a DLQ while letting ShipmentRequested proceed results in corrupted state.

• Blocking Retries: Use Exponential Backoff. Because of the Sub-Inbox, a retry for Order-101 only blocks Order-101.
• The Order Lock: If a message reaches the retry limit and moves to a DLQ, you must flag that order_id as "Blocked" in the database. No subsequent events for that ID should be processed until the DLQ item is resolved by a human.

Specifying and Testing the Contract

• Specification: Use a Schema Registry. Define order_id as a required partitioning key.
• Verification: Use Pact (Contract Testing). The Consumer defines a "Pact" (e.g., "I require a non-null order_id"), and the Producer verifies this in their CI/CD pipeline.

Production Checklist

• Partitions: Created with enough headroom for 3 years of growth.
• Database Index: INBOX table has a composite index on (status, order_id).
• Idempotency: Unique constraints prevent duplicate processing on retries.
• Monitoring: Alerting set for "Inbox Age" (time an event waits in the DB).
• Cleanup: A background job (TTL) purges processed rows every 24 hours.

Takeaway

Strict ordering is often viewed as a performance tax. By shifting the logic from the rigid Kafka partition to a Database-backed Sub-Inbox, you get the Sequence Integrity along with parallel processing