A senior engineer's deep dive into the guarantees, tradeoffs, and production gotchas of implementing strictly ordered Kafka consumers — and the honest conversation about when not to.
The Why
In my previous post, I built a zero-loss Kafka consumer using @RetryableTopic. But I deliberately called out one tradeoff that deserves its own conversation:
"Moving a message to a retry topic breaks ordering guarantees for that partition."
That sentence sent me down a rabbit hole. If retryable topics — one of Spring Kafka's most powerful resilience tools — inherently break ordering, what does it actually take to guarantee strict message order in a production Kafka system?
This post is everything I found: the real guarantees Kafka provides, the configuration decisions that break ordering silently, and the honest tradeoffs you need to make before choosing this path.
What Kafka Actually Guarantees (And What It Doesn't)
Let's start with what Kafka promises, because most ordering bugs come from misreading this.
Kafka guarantees ordering within a partition. It makes no guarantees across partitions.
That's it. Full stop. Not per topic. Not per consumer group. Per partition.
Before going deeper, here is where each ordering guarantee actually lives in the pipeline — because each layer can independently break the guarantee above it:
PRODUCER BROKER CONSUMER
───────────────────────── ───────────────────────── ─────────────────────────
Stable partition key Single partition concurrency=1 (or 1 per partition)
→ same key always → messages stored enable.auto.commit=false
routes to same partition in append order AckMode.RECORD
enable.idempotence=true Replication (acks=all) No @Async hand-offs
max.in.flight=1 → no gaps from CooperativeStickyAssignor
→ no broker reordering leader failover → rebalance-safe handoff
from retried batches
← Ordering guarantee lives across ALL three layers →
Breaking any one layer silently breaks end-to-end order
This is also the map for debugging ordering incidents in production: work left to right. A gap in your offset sequence? Check the producer's max.in.flight setting. Out-of-order messages after a deployment? Check the rebalance assignor. Duplicates? Check idempotence and ack mode.
This has a direct consequence: if messages that must be processed in sequence land on different partitions, you have no ordering guarantee regardless of how well you configure your consumer. The ordering problem starts at the producer, not the consumer.
Consider an e-commerce order lifecycle:
Event 1: ORDER_PLACED (orderId: 1001)
Event 2: PAYMENT_CAPTURED (orderId: 1001)
Event 3: INVENTORY_RESERVED (orderId: 1001)
Event 4: ORDER_SHIPPED (orderId: 1001)
All four events must be processed in sequence. If EVENT_3 is processed before EVENT_2, inventory is reserved against an unpaid order. If EVENT_4 arrives before EVENT_3, the shipment goes out against unreserved stock.
This is exactly the class of problem where Kafka ordering matters — and where getting it wrong causes real business damage.
Or consider a financial context (as a reference, not a recommendation to use Kafka for core ledger operations): a multi-leg settlement where debit must precede credit. Kafka can work here, but only with explicit ordering discipline. We will come back to where Kafka ordering fits and where it does not.
The Foundation: Partition Key Strategy
Since ordering is partition-scoped, the partition key is the most important architectural decision you will make for ordered consumers.
The rule: all messages that must be processed in sequence must share the same partition key. Kafka routes messages with the same key to the same partition using a consistent hash.
// Producer: always key by the entity whose events must be ordered
ProducerRecord<String, String> record = new ProducerRecord<>(
"order-events",
order.getOrderId(), // ← This is the partition key
eventPayload
);
What goes wrong here in practice:
- Null keys — Kafka round-robins null-key messages across all partitions. Order is immediately lost.
- Random UUIDs as keys — same problem. Each message lands on a different partition.
- Using producer instance ID or timestamp as the key — technically unique per message, which defeats the purpose entirely.
- Changing partition count after go-live — Kafka's hash function maps keys to partitions based on partition count. Increasing partitions remaps keys to different partitions, breaking the ordering guarantee for in-flight and future messages.
Production rule: treat your partition count as immutable once consumers are live. Over-provision partitions at topic creation time rather than resizing later.
Consumer Concurrency: The Silent Ordering Killer
This is where the majority of ordering bugs I have seen are introduced — not at the architecture level, but in a configuration property that is easy to overlook.
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(3); // ← This is the problem
return factory;
}
Setting concurrency > 1 means Spring Kafka spins up multiple consumer threads, and under normal operation each thread is assigned its own partition — a single partition is never served by two threads simultaneously. The real ordering risk is subtler: during a rebalance, a partition can be moved from one thread to another. The new thread can start consuming from that partition before the previous thread has finished processing its current batch. That window — old thread still running, new thread already polling — is where out-of-order processing occurs. It is easy to overlook precisely because it only surfaces under rebalance conditions, not in steady-state load testing.
For strict ordering, the rule is:
factory.setConcurrency(1); // One thread per listener container
Or, if you want to scale with partition count:
// Set concurrency = number of partitions
// Each thread owns exactly one partition
factory.setConcurrency(partitionCount);
The second option is correct but requires discipline: if you set concurrency=6 against a topic with 6 partitions, each thread owns one partition and ordering within each partition is preserved. The moment concurrency exceeds partition count, some partitions get no dedicated thread and ordering breaks.
Equally important: never use @Async inside a @KafkaListener method. The moment you hand processing off to another thread pool, the calling thread moves to the next message while the previous one is still being processed. Ordering is gone.
// This silently breaks ordering
@KafkaListener(topics = "order-events")
public void consume(ConsumerRecord<String, String> record) {
asyncService.processAsync(record.value()); // returns immediately
}
// Processing must complete before the next message is touched
@KafkaListener(topics = "order-events")
public void consume(ConsumerRecord<String, String> record) {
orderProcessingService.process(record.value()); // blocking, synchronous
}
Offset Commit Strategy: Where Zero-Loss and Ordering Intersect
If you read my previous post on zero-loss consumers, you already know that enable.auto.commit=true is dangerous for reliability. For ordered consumers, it is doubly dangerous.
Auto-commit uses a time-based interval. It has no awareness of whether your processing logic has completed. In a high-throughput system, you can auto-commit offsets for messages your application is still processing — and if the application crashes, those messages are never reprocessed, silently breaking both your delivery guarantee and your ordering assumptions.
The correct configuration:
// application.yml
spring:
kafka:
consumer:
enable-auto-commit: false
auto-offset-reset: earliest
# In your ContainerFactory
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.RECORD);
AckMode.RECORD commits the offset after each record is fully processed. For strict ordering this is the safest mode — you never advance past a message until it has been handled.
If you need manual control (for example, batching acks across a transaction boundary):
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE);
With MANUAL_IMMEDIATE, you must call acknowledgment.acknowledge() explicitly. The critical rule: acknowledge in order. Out-of-order acknowledgment creates offset gaps. When the consumer restarts, it will reprocess from the earliest unacknowledged offset, which may cause you to re-process messages you already handled.
Error Handling: The Hardest Tradeoff in Ordered Systems
This is where the real tension lives, and I want to be direct about it because most blog posts gloss over it.
The fundamental conflict: in an ordered consumer, a failed message blocks the partition. Every message behind it waits. But the standard tool for Kafka resilience — @RetryableTopic — resolves this by moving the failed message to a separate retry topic, which immediately breaks ordering for that partition.
You have to choose. There is no configuration that gives you both non-blocking retries and guaranteed ordering.
Option 1: Blocking Retry (Preserves Order, Risks Throughput)
@Bean
public DefaultErrorHandler errorHandler() {
// Retry up to 3 times with exponential backoff
ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(3);
backOff.setInitialInterval(1_000L);
backOff.setMultiplier(2.0);
backOff.setMaxInterval(10_000L);
DefaultErrorHandler handler = new DefaultErrorHandler(
new DeadLetterPublishingRecoverer(kafkaTemplate), backOff
);
// Never retry these — they will always fail
handler.addNotRetryableExceptions(
JsonParseException.class,
ValidationException.class
);
return handler;
}
With DefaultErrorHandler and no retry topic, retries happen in-place. The partition is blocked while retries are attempted. This is the correct choice when:
- Your messages have business-level dependencies on each other (true ordering requirement)
- A transient failure retrying for a few seconds is acceptable
- Your downstream can recover quickly
When retries are exhausted, the message goes to the DLT via DeadLetterPublishingRecoverer. At that point, you have a decision: stop the consumer (strict ordering, no skipping) or skip to the next message (lose strict ordering, maintain throughput).
Option 2: DLT With Consumer Pause (Best of Both Under Prolonged Failure)
For scenarios where the downstream is fully down (not just a transient blip), blocking retries will pin your partition for a long time. A better pattern is to pause the consumer and let messages accumulate safely in the broker — they stay in their original partition order, untouched, ready to be processed in sequence once the dependency recovers.
The trigger for a pause is your error handler detecting consecutive exhausted retries within a time window — not a single failure. A single transient failure should exhaust the backoff and either recover or go to DLT. It is only when failures are sustained (indicating downstream unavailability rather than a bad message) that pausing the entire consumer is warranted.
@Component
public class OrderedConsumerController {
private final KafkaListenerEndpointRegistry registry;
// Called by your error handler after N consecutive DLT publishes within a window,
// indicating the downstream service is unavailable — not just a bad message
public void pauseConsumer(String listenerId) {
MessageListenerContainer container = registry.getListenerContainer(listenerId);
if (container != null && container.isRunning()) {
container.pause();
log.warn("Consumer [{}] paused due to sustained downstream failure", listenerId);
}
}
public void resumeConsumer(String listenerId) {
MessageListenerContainer container = registry.getListenerContainer(listenerId);
if (container != null) {
container.resume();
log.info("Consumer [{}] resumed after downstream recovery", listenerId);
}
}
}
Recovery monitoring is what completes this pattern. In my previous post on zero-loss consumers, I used a HeartbeatScheduler that periodically probed the downstream on a fixed interval and published a recovery event once the health check passed. The same approach applies here: a scheduled task that calls your downstream's health endpoint every 10–30 seconds, and calls resumeConsumer() on success. The key difference in an ordered consumer is that you must not resume until you are confident the dependency is stable — a premature resume that fails immediately will cause another pause cycle and risk committing a partial offset.
Producer-Side Guarantees: The Part Most Ordering Guides Skip
You can configure your consumer perfectly and still lose ordering at the broker level if your producer is misconfigured. Three settings matter:
spring:
kafka:
producer:
acks: all # Wait for all in-sync replicas to acknowledge
retries: 3
properties:
enable.idempotence: true
max.in.flight.requests.per.connection: 1
acks=all — without this, a message can be acknowledged by the leader but not yet replicated. If the leader fails before replication, the message is lost. Your consumer never sees it, and the sequence has a permanent gap.
enable.idempotence=true — producer retries can cause duplicate messages at the broker if the original request succeeded but the acknowledgment was lost in transit. With idempotence enabled, Kafka deduplicates using a producer ID and sequence number. This is a prerequisite for correct ordered delivery.
max.in.flight.requests.per.connection=1 — this one is subtle. When a producer has multiple in-flight requests, a retry of request N can overtake request N+1 if N+1 was acknowledged first. The result: messages arrive at the broker out of order. Setting this to 1 ensures only one batch is in-flight at a time. Note: with enable.idempotence=true, Kafka allows up to 5 in-flight requests without reordering risk, but setting it to 1 is the conservative and safest choice for strictly ordered workloads.
Rebalance Handling: The Ordering Disruption You Cannot Ignore
A consumer group rebalance — triggered by a consumer joining, leaving, or timing out — temporarily pauses all consumers in the group while partition ownership is reassigned. Any in-flight processing during a rebalance can complete out of order relative to the messages that follow.
Use the Cooperative Sticky Assignor:
spring:
kafka:
consumer:
properties:
partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
The default RangeAssignor is a stop-the-world rebalance: all consumers stop, all partitions are unassigned, then reassigned. CooperativeStickyAssignor only reassigns partitions that actually need to move. Consumers that keep their partition assignments continue processing uninterrupted. For ordered consumers on high-partition topics, this is a significant improvement.
Use Static Group Membership to reduce rebalance frequency:
spring:
kafka:
consumer:
properties:
group.instance.id: order-consumer-instance-1 # Unique per instance
session.timeout.ms: 60000
Without static membership, every application restart triggers a rebalance. With group.instance.id, Kafka recognises the consumer as the same instance rejoining and avoids a full rebalance if it comes back within the session timeout. For Kubernetes deployments where pods restart frequently, this is a material difference.
Implement ConsumerRebalanceListener for clean handoffs:
@Component
public class OrderedRebalanceListener implements ConsumerRebalanceListener {
private final Set<TopicPartition> currentAssignment = new HashSet<>();
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// Complete or checkpoint in-flight work before partitions are handed off
log.info("Partitions revoked: {}. Completing in-flight processing.", partitions);
// Flush any buffered state, commit offsets manually if using MANUAL ack mode
currentAssignment.removeAll(partitions);
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
log.info("Partitions assigned: {}", partitions);
currentAssignment.addAll(partitions);
}
}
Idempotent Processing: Non-Negotiable
Rebalances cause redelivery. Network failures cause redelivery. Consumer restarts cause redelivery. Your consumer will process the same message more than once — the question is whether your business logic handles it correctly.
Every handler in an ordered consumer must be idempotent. Processing the same message twice must produce the same result as processing it once.
A simple deduplication pattern using the original offset and partition as a correlation key:
@Service
public class IdempotentOrderProcessor {
private final OrderRepository orderRepository;
private final ProcessedMessageRepository processedMessages;
public void process(ConsumerRecord<String, String> record) {
String deduplicationKey = record.topic() + "-" + record.partition() + "-" + record.offset();
if (processedMessages.existsByKey(deduplicationKey)) {
log.info("Duplicate message detected, skipping. Key: {}", deduplicationKey);
return;
}
// Process the message
OrderEvent event = deserialize(record.value());
orderRepository.applyEvent(event);
// Mark as processed (ideally in the same transaction as the business operation)
processedMessages.save(new ProcessedMessage(deduplicationKey));
}
}
Important note on this pattern: if you are using a relational database, saving the processed message key and applying the business operation should be in the same database transaction. If the application crashes between the business operation and saving the key, you get a reprocessed message on restart. Transactional outbox or a unique constraint on the key column enforced by the database are both valid approaches.
Sequence Numbers: Defense in Depth
Kafka offsets tell you where a message sits within a partition. But they do not tell you the intended business sequence — which can differ if messages are produced out of order by the upstream system, or if a producer retry creates a gap.
For critical ordered consumers, embedding a sequence number in the message payload gives you an additional layer of defense:
public class OrderEvent {
private String orderId;
private String eventType;
private long sequenceNumber; // Monotonically increasing per orderId
private Instant occurredAt;
// ...
}
In your consumer, validate before processing:
public void process(OrderEvent event) {
long expectedSeq = orderSequenceTracker.getNext(event.getOrderId());
if (event.getSequenceNumber() < expectedSeq) {
log.warn("Duplicate or old event detected. orderId={}, seq={}, expected={}",
event.getOrderId(), event.getSequenceNumber(), expectedSeq);
return; // Idempotency: already processed
}
if (event.getSequenceNumber() > expectedSeq) {
log.error("Gap in sequence detected. orderId={}, seq={}, expected={}",
event.getOrderId(), event.getSequenceNumber(), expectedSeq);
throw new SequenceGapException("Missing event in sequence for order: " + event.getOrderId());
}
applyEvent(event);
orderSequenceTracker.advance(event.getOrderId());
}
A SequenceGapException thrown here will trigger your error handler. Whether you block-retry, pause the consumer, or route to a DLT depends on your business tolerance — but at least you have detected the gap rather than processing out-of-order silently.
Observability: You Cannot Debug What You Cannot See
Ordered consumers fail in subtle ways. A rebalance redelivery processed twice. A gap in the offset sequence. A message processed 200ms before its predecessor completed. None of these show up as errors in your application logs without deliberate instrumentation.
Log partition and offset on every message:
@KafkaListener(topics = "order-events", groupId = "order-processor")
public void consume(
ConsumerRecord<String, String> record,
@Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
@Header(KafkaHeaders.OFFSET) long offset) {
log.info("Processing message. topic={}, partition={}, offset={}, key={}",
record.topic(), partition, offset, record.key());
orderProcessingService.process(record);
log.info("Completed message. topic={}, partition={}, offset={}",
record.topic(), partition, offset);
}
Expose consumer lag as a metric. Consumer lag — the gap between the latest offset produced and the latest offset committed by your consumer — is the single most important health signal for an ordered consumer. A growing lag means messages are accumulating unprocessed, which eventually translates to ordering delays even if no failures have occurred.
With Micrometer and Spring Boot Actuator, consumer lag is exposed automatically if you include spring-kafka metrics. Ensure your Prometheus scrape config picks it up:
management:
metrics:
tags:
application: order-consumer
endpoints:
web:
exposure:
include: health,metrics,prometheus
Alert on: lag growing beyond your SLA threshold, DLT publish rate above zero, offset gap between consecutive commits, and rebalance frequency.
Where Kafka Ordering Fits — And Where It Does Not
After all of the above, the honest answer: Kafka ordering is appropriate for event-driven workflows where sequence matters but the domain can tolerate the complexity overhead.
It fits well for:
- Order lifecycle events (placed → paid → shipped → delivered)
- User action streams that must be replayed in order
- Audit trails where chronological sequence is required
- CDC (Change Data Capture) streams that must apply database changes in order
It is the wrong tool for:
- Core financial ledger operations requiring ACID guarantees across multiple legs
- Workflows where any ordering failure must immediately halt the entire pipeline — a relational database with proper transaction management is simpler and safer
- Low-volume, low-latency scenarios where the operational complexity of Kafka outweighs its throughput benefits
If your requirement is "these four operations must all succeed or all fail, in order, atomically" — that is a distributed transaction problem. Look at the Saga pattern with orchestration, or a transactional outbox, before reaching for Kafka ordering.
The Full Picture: Configuration Reference
Every property below has a reason. The two most commonly misconfigured ones in ordered consumers are max.poll.interval.ms and max.poll.records — and getting either wrong causes silent ordering failures rather than hard errors.
spring:
kafka:
consumer:
enable-auto-commit: false
auto-offset-reset: earliest
properties:
partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor
group.instance.id: ${HOSTNAME}-consumer # Static membership — prevents rebalance on pod restart
session.timeout.ms: 60000
# Conservative ceiling: limits how many records a single poll() fetches.
# If your processing logic takes 200ms per record, 50 records = 10s of work per poll.
# This directly informs max.poll.interval.ms below — they must be sized together.
max.poll.records: 50
# Must exceed your worst-case processing time for a full poll batch.
# Formula: max.poll.interval.ms > max.poll.records × max_processing_time_per_record_ms
# With 50 records at 200ms each = 10,000ms. 300,000ms gives a 30× safety margin.
# If this deadline is breached, Kafka treats the consumer as dead, triggers a rebalance,
# and the in-flight batch may be reprocessed by another thread — breaking ordering.
max.poll.interval.ms: 300000
producer:
acks: all
retries: 3
properties:
enable.idempotence: true
max.in.flight.requests.per.connection: 1
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(1); // Or = partition count with sticky assignment
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.RECORD);
ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(3);
backOff.setInitialInterval(1_000L);
backOff.setMultiplier(2.0);
backOff.setMaxInterval(10_000L);
factory.setCommonErrorHandler(new DefaultErrorHandler(
new DeadLetterPublishingRecoverer(kafkaTemplate), backOff
));
return factory;
}
Final Thoughts
Guaranteed message ordering in Kafka is achievable, but it is not a feature you turn on. It is a property that emerges from a set of consistent decisions made across the producer, the broker, and the consumer — and broken by any one decision made without understanding its implications.
The combination that holds in production:
- Stable, business-meaningful partition keys set at the producer
-
enable.idempotence=trueandmax.in.flight.requests.per.connection=1on the producer -
enable.auto.commit=falsewithAckMode.RECORDon the consumer -
concurrency=1(or one thread per partition) with no async hand-offs - Blocking retry with
DefaultErrorHandler— not@RetryableTopic, which trades ordering for throughput - Cooperative sticky assignor and static group membership for rebalance stability
- Idempotent processing with deduplication keyed on topic + partition + offset
- Sequence numbers in the payload for gap detection
- Consumer lag and offset gap alerting as the primary health signals
I have not deployed this exact pattern to production — but I have run the zero-loss consumer from my previous post in production, and the foundation is the same: respect Kafka's actual guarantees, design for redelivery, and make your tradeoffs explicit rather than discovering them in an incident.
If you have implemented ordered consumers in production and found gaps in this — or made different tradeoff decisions — I would genuinely like to hear about it in the comments.
This is part of a series on production-grade Kafka with Spring Boot. The previous post covers zero-loss delivery with retryable topics.
Tags: #kafka #springboot #java #distributedsystems #eventdrivenarchitecture #microservices
Top comments (0)