- Book: Event-Driven Architecture Pocket Guide
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You have a topic with 12 partitions and a backlog of image-resize jobs. Each job takes a few seconds. You scale the worker deployment from 12 pods to 40, watch the dashboard, and nothing moves. Twelve pods are busy. Twenty-eight are connected, healthy and idle, because a Kafka consumer group hands each partition to exactly one member. Your parallelism was decided the day someone picked the partition count.
That rule is one reason teams run RabbitMQ or Amazon SQS next to Apache Kafka for job queues. KIP-932 describes it bluntly: consumer groups "introduce coupling between the number of consumers in a consumer group and the number of partitions", so users "often have to over-partition" to handle peak load.
Kafka 4.2 changes that. The 4.2.0 release announcement (February 17, 2026) says "Kafka Queues (Share Groups) is now production-ready", after a preview in 4.1 that was explicitly "still not ready for production".
A share group reads one partition with many hands
A share group is a new group type that sits next to classic consumer groups. Consumers join it with a group.id like always, but the broker no longer pins a partition to one member. Any number of consumers can fetch from the same partition at the same time, and the broker tracks every record individually instead of keeping a single committed offset.
The KIP calls that per-partition view a share-partition. The broker keeps a window of in-flight records between two markers: the share-partition start offset (SPSO) and end offset (SPEO). Every record in the window is in one of four states, per KIP-932:
- Available: can be handed to the next consumer that fetches.
- Acquired: handed to one consumer, under a time-limited lock.
- Acknowledged: done.
- Archived: given up on, never delivered again.
When a consumer fetches, the share-partition leader walks forward from the SPSO, finds Available records, moves them to Acquired and bumps each record's delivery count. While a record is acquired, no other consumer receives it.
One partition, many consumers: the broker locks each record to whoever acquired it.
The lock has a deadline. The Kafka 4.2 group configs list share.record.lock.duration.ms with a default of 30,000 ms. If the consumer neither acknowledges nor renews in that time, the lock is released and the record goes back to Available, ready for another consumer. A consumer that dies mid-job just stops holding locks; its records come back after the timeout.
The window also has a size limit. The KIP says that when the in-flight limit is reached, "fetching records will temporarily yield no further records" until acquired records clear. In 4.2 the cap is a broker setting, group.share.partition.max.record.locks, default 2,000 per share-partition, per the 4.2 broker configs. The 4.3 group configs add a per-group override, share.partition.max.record.locks.
All this state has to survive broker restarts. A new share coordinator persists it to an internal topic, __share_group_state, which the KIP describes as highly partitioned (50 partitions by default).
ACCEPT, RELEASE, REJECT, and the new RENEW
By default a share consumer uses implicit acknowledgement: the next poll() accepts everything from the previous batch. For a job queue you want explicit mode, where you decide per record. The KafkaShareConsumer Javadoc requires you to acknowledge every record from a poll before the next poll, and the acknowledgements go to the broker on the next commitSync(), commitAsync() or poll().
String sd = StringDeserializer.class.getName();
Properties p = new Properties();
p.setProperty("bootstrap.servers",
System.getenv("KAFKA_BOOTSTRAP"));
p.setProperty("group.id", "resize-workers");
p.setProperty("key.deserializer", sd);
p.setProperty("value.deserializer", sd);
p.setProperty("share.acknowledgement.mode", "explicit");
var consumer = new KafkaShareConsumer<String, String>(p);
consumer.subscribe(List.of("resize-jobs"));
while (true) {
var records = consumer.poll(Duration.ofMillis(500));
for (var rec : records) {
try {
resize(rec.value());
consumer.acknowledge(rec, AcknowledgeType.ACCEPT);
} catch (TransientFailure e) {
consumer.acknowledge(rec, AcknowledgeType.RELEASE);
} catch (Exception e) {
consumer.acknowledge(rec, AcknowledgeType.REJECT);
}
}
consumer.commitSync();
}
resize and TransientFailure are yours. The three original acknowledgement types come straight from the KIP. ACCEPT marks the record done. RELEASE "makes the record available for another delivery attempt", which is right for a timeout against a flaky downstream. REJECT says the record is unprocessable and it is not delivered again.
That leaves the job that takes longer than the lock. Before 4.2 you had two bad options: raise the lock duration for the whole group (capped by the broker's group.share.max.record.lock.duration.ms, 60 seconds by default per the broker configs), or let a slow job's lock expire and watch a second worker start the same job. 4.2 adds a fourth type from KIP-1222, RENEW, which the release announcement says lets "applications extend acquisition lock timeouts on records requiring longer processing times."
Per the Javadoc, RENEW only works in explicit mode, you call it on each iteration of the poll loop, and a successfully renewed record "is returned again by the poll(Duration) call". The pattern is to run the job on another thread and keep the poll thread renewing until the job finishes. The consumer is not thread-safe, so every acknowledge call stays on the poll thread. Here pool is an ExecutorService:
// running: map of key(rec) -> CompletableFuture<Void>
var job = running.get(key(rec));
if (job == null) {
running.put(key(rec), CompletableFuture
.runAsync(() -> resize(rec.value()), pool));
consumer.acknowledge(rec, AcknowledgeType.RENEW);
} else if (job.isCompletedExceptionally()) {
consumer.acknowledge(rec, AcknowledgeType.REJECT);
running.remove(key(rec));
} else if (job.isDone()) {
consumer.acknowledge(rec, AcknowledgeType.ACCEPT);
running.remove(key(rec));
} else {
consumer.acknowledge(rec, AcknowledgeType.RENEW);
}
The Javadoc example for this pattern also sets share.acquire.mode to record_limit and max.poll.records to 1, so a worker only acquires what it can actually hold. share.acquire.mode comes from KIP-1206, also in 4.2. The docs say RENEW is "intended only for situations where the processing times of the records exceeds the acquisition lock duration", so do not renew by reflex on fast jobs.
What you give up: ordering
A classic consumer group gives you strict order within a partition because one consumer reads it front to back. A share group gives you much less. The KIP guarantees that records returned in one batch for a share-partition come "in order of increasing offset". Across batches there is no guarantee, and redelivered records can arrive out of offset order.
Walk it through. Worker A acquires offset 41 and stalls. Worker B acquires 42 and finishes. A's lock expires, 41 goes back to Available, and worker C processes it after 42. If 41 was "create account" and 42 was "rename account", your rename just ran against a row that does not exist yet.
If the order of events for the same key matters (account ledgers, state machines, change-data-capture streams), keep using a consumer group, keyed partitioning and the one-consumer-per-partition rule that you were fighting. Share groups fit work where each record stands alone: resize this image, send this email, call this webhook, score this document.
The other half of the trade is the delivery guarantee. KIP-932 says "the delivery behavior is at-least-once." A lock can expire on a job that actually finished, or a consumer can crash after the side effect and before the commit. Either way, the same record can run twice. Your handler has to make that harmless.
The usual way is a claim row in the same transaction as the side effect. Here it is against PostgreSQL, with a job id carried in the payload so it also covers duplicates that a producer retry wrote at a different offset:
boolean claim(Connection db, String jobId)
throws SQLException {
String sql = "INSERT INTO processed_jobs (job_id) "
+ "VALUES (?) ON CONFLICT (job_id) DO NOTHING";
try (var ps = db.prepareStatement(sql)) {
ps.setString(1, jobId);
return ps.executeUpdate() == 1;
}
}
Call it inside the transaction that writes the result. If it returns false, the job already ran: ACCEPT the record and move on.
Poison messages: five strikes, then silence
A share group already has a retry limit built in. Every acquisition bumps the delivery count, and the KIP sets the default attempt limit at 5. When a record hits it, the record "moves into Archived state and is not eligible for additional delivery attempts." In 4.2 the limit is the broker setting group.share.delivery.count.limit (default 5, range 2 to 10, per the 4.2 broker configs), and the 4.3 group configs add a per-group override, share.delivery.count.limit.
After the last attempt, a poison record is archived, not parked somewhere you can inspect it.
Archived means gone from the queue's point of view. The record still sits in the topic log at its offset, but nothing will deliver it to your share group again, and nobody gets paged. A poison message no longer blocks the partition the way it can with a classic consumer, which is a real improvement. It also means a bad deploy can quietly archive thousands of records.
A dead-letter topic for share groups exists on paper: KIP-1191 is accepted and would write rejected and exhausted records to a configured DLQ topic. It does not appear in the 4.2 or 4.3 release announcements, so check your broker version before counting on it. Until then, park the record yourself when its last delivery fails, before the broker archives it. ConsumerRecord exposes the count as deliveryCount(). The REJECT branches above need the same treatment, since a rejected record is archived just as silently. The first loop's catch blocks become:
} catch (TransientFailure e) {
if (lastDelivery(rec)) park(rec);
else consumer.acknowledge(rec, AcknowledgeType.RELEASE);
} catch (Exception e) {
park(rec);
}
And the two helpers:
boolean lastDelivery(ConsumerRecord<String, String> rec) {
int tries = rec.deliveryCount()
.map(Short::intValue).orElse(1);
return tries >= MAX_ATTEMPTS;
}
void park(ConsumerRecord<String, String> rec) {
try {
deadLetters.send(new ProducerRecord<>(
"resize-jobs.dlq", rec.key(), rec.value())).get();
consumer.acknowledge(rec, AcknowledgeType.ACCEPT);
} catch (InterruptedException | ExecutionException e) {
consumer.acknowledge(rec, AcknowledgeType.RELEASE);
}
}
deadLetters is an ordinary KafkaProducer. MAX_ATTEMPTS has to match the delivery count limit in force for the group, which on 4.2 is the broker's group.share.delivery.count.limit. The .get() waits for the DLQ write before the ACCEPT, so a failed write releases the record instead of dropping it. On the last delivery a release still ends in the archive, so alert on producer errors as well as on the DLQ topic's growth.
Turning it on without surprises
The 4.2 upgrade notes carry three practical details. On 4.1, share groups were behind a feature flag you enabled with kafka-features.sh at share.version=1. In 4.2, clusters with fewer than three brokers have to lower the replication factor and min ISR of the share coordinator's state topic to 1, because __share_group_state defaults to three replicas (broker configs). And 4.2.1 "includes a fix for a critical deadlock in the KIP-932 Share Group path", so start at 4.2.1 or later, not 4.2.0.
Should you rip out the job queue that sits next to your Kafka cluster? Not on day one. If your workers are already idempotent, your jobs are independent, and running one less piece of infrastructure matters to you, share groups are worth a pilot on one non-critical queue. If your jobs depend on order, or your team has never had to think about at-least-once delivery, the partition-per-consumer rule was protecting you, and keeping it is a fine decision.
If this was useful
Share groups move two problems from the broker back into your code: idempotent consumers and retry handling. Those are the kind of traps the Event-Driven Architecture Pocket Guide covers, along with sagas, CQRS and the outbox pattern. If you are about to put a job queue on Kafka, it is the checklist I would want next to the design doc.
Top comments (0)