Amit Kamble

Posted on Mar 24

Kafka Ordering in the Real World: How to Scale Without Killing Performance

#kafka #partition #performance #robust

Intro: The Sequential Processing Paradox

In e-commerce, the order of events is non-negotiable. You cannot process OrderFulfilled before OrderCreated or PaymentAuthorized. Standard Kafka advice is to use a Message Key (like order_id) to ensure all related events land in the same partition in chronological order.

The Performance Killer: This leads to Head-of-Line (HOL) Blocking. If Partition 1 contains 1,000 orders and one "hot" order triggers a slow external fraud check or a database timeout, every other order in that same partition—even healthy ones—is stalled. This article provides the blueprint to achieve strict ordering with horizontal scalability.

The Concepts: The Distributed Integrity Chain

To maintain order at scale, we must move beyond a simple "Kafka-only" view to a three-stage architectural safety chain.

1. The Transactional Outbox (Producer Safety)

Never call Kafka directly from your business logic. If your database commits but Kafka is down, you have a "dual-write" failure and a lost event.

The Pattern: Save the Order update and a Message entry into a local OUTBOX table in the same database transaction.
The Relay: A separate process (the "Relay") polls this table and publishes to Kafka. This guarantees At-Least-Once delivery.

2. Keyed Partitioning (The Sequence Pipe)

By using order_id as the Kafka Key, we ensure the "Log" for that specific order is physically sequential. Kafka handles the heavy lifting of keeping v1 ahead of v2 within the partition.

3. The Sub-Inbox (Consumer Parallelism)

The Kafka listener doesn't process logic; it simply writes the message into an INBOX table and acknowledges the Kafka offset immediately.

Why it works: Kafka is now "clear" to keep ingestion high.
The "Sub" Logic: Multiple background worker threads query the INBOX table using Database Locking (e.g., SELECT ... FOR UPDATE SKIP LOCKED). While Worker A processes Order-101, Worker B can simultaneously process Order-102 from the same table/partition.

Important Decisions & Future-Proofing

1. Partition Strategy: Planning for Growth

Partitions are your unit of parallelism. You cannot easily increase partition counts later without changing the hash result of your keys, which breaks the sequence.

Guidance: Over-partition from Day 1. If you need 10 now, create 60. This allows you to scale your consumer group up to 60 instances without a complex data migration.

2. Decision Matrix: Choosing Your Strategy

Requirement	Raw Kafka Key	Standard Inbox	Sub-Inbox (Parallel)
Ordering Scope	Per Partition	Per Key	Per Key
Concurrency	1 Thread/Partition	1 Thread/Service	N Threads/Service
Failure Impact	Blocks Partition	Blocks Service	Blocks ONE Order Only
Complexity	Low	Medium	High

Technical Guide: Spring Boot Implementation

Producer: Transactional Outbox

@Transactional
public void placeOrder(OrderDTO dto) {
    // 1. Persist Business State
    Order order = orderRepo.save(new Order(dto));

    // 2. Persist Event to Outbox in the same transaction
    outboxRepo.save(new OutboxEvent(
        order.getId(), 
        "ORDER_CREATED", 
        json(order)
    ));
}

Consumer: Inbox + Manual Ack

Set ack-mode: manual_immediate in your application.yml.

@KafkaListener(topics = "orders", groupId = "fulfillment-service")
@Transactional
public void onMessage(ConsumerRecord<String, String> record, Acknowledgment ack) {
    // Unique constraint on (order_id + version) handles idempotency
    inboxRepository.save(new InboxEntry(
        record.key(), 
        record.value(), 
        record.offset()
    ));

    // Acknowledge ONLY after the DB transaction commits
    ack.acknowledge(); 
}

Retries and the "Problem Bin" (DLQ)

In an ordered flow, moving a failed PaymentAuthorized message to a DLQ while letting ShipmentRequested proceed results in corrupted state.

Blocking Retries: Use Exponential Backoff. Because of the Sub-Inbox, a retry for Order-101 only blocks Order-101.
The Order Lock: If a message reaches the retry limit and moves to a DLQ, you must flag that order_id as "Blocked" in the database. No subsequent events for that ID should be processed until the DLQ item is resolved by a human.

Specifying and Testing the Contract

Specification: Use a Schema Registry. Define order_id as a required partitioning key.
Verification: Use Pact (Contract Testing). The Consumer defines a "Pact" (e.g., "I require a non-null order_id"), and the Producer verifies this in their CI/CD pipeline.

Production Checklist

Partitions: Created with enough headroom for 3 years of growth.
Database Index: INBOX table has a composite index on (status, order_id).
Idempotency: Unique constraints prevent duplicate processing on retries.
Monitoring: Alerting set for "Inbox Age" (time an event waits in the DB).
Cleanup: A background job (TTL) purges processed rows every 24 hours.

Takeaway

Strict ordering is often viewed as a performance tax. By shifting the logic from the rigid Kafka partition to a Database-backed Sub-Inbox, you get the best of both worlds: Absolute Sequence Integrity at Blazing Parallel Speed.

DEV Community