Fernando Azevedo

Posted on Nov 3 Edited on Nov 4 Originally published at fernando.moretes.com

Lambda with Kafka queues: parallelism past partitions, and what it costs

#awscloud #kafka #awslambda #eventdriven

For years the answer to "my Kafka consumer can't keep up with the spike" was the same: more partitions. The problem is that a partition is not a throughput knob, it is a contract with the producer, with rebalancing and with whoever operates the cluster. On November 2, 2026, AWS added a second consumption mode, Queue, to the self-managed Kafka event source mapping (ESM), built on Apache Kafka 4.2 share groups. The question that matters is not "does this scale further?". It does. The question is: who guarantees that record 4,812 on partition 3 was processed once, and what happens to it when it fails for the fifth time at 2 AM?

The problem: the partition became a concurrency ceiling

In Stream mode (the default), the ESM uses the consumer group protocol: each partition belongs to exactly one consumer. The Lambda documentation is explicit about the consequence: in provisioned mode, to keep ordering within a partition, MaximumPollers is capped at the topic's partition count, and each poller handles up to 5 MB/s or up to 5 concurrent invocations.

Do the math for a 12-partition topic: 12 pollers times 5 invocations is 60 simultaneous executions, at most. If the handler calls a fraud model that takes 800 ms, the ceiling is close to 75 records per second with single-record batches, and no ReservedConcurrentExecutions setting changes that.

The two traditional ways out carry a maintenance cost, not a build cost:

Over-partitioning: 12 becomes 96. The partition key distribution changes, rebalancing gets slower, the broker carries more segment files and replicas, and you almost never go back, because shrinking partitions does not exist in Kafka.
Copying into another queue: a consumer that republishes to SQS. Now you operate two queues, two retention policies, two duplication points and a bridge that someone has to keep current.

For work where each event is an independent unit (media encoding, notifications, telemetry ingestion, per-transaction fraud scoring), paying the price of ordering without needing it is pure waste.

Anatomy: share group, lock and acknowledgement

KIP-932 swaps partition ownership for record ownership. The broker keeps a window of in-flight records per partition, between the share-partition start offset (SPSO) and end offset (SPEO). Each consumer that fetches data acquires a temporary lock on records, not on the partition. When done, the consumer acknowledges each record with one of three types: ACCEPT (processed), RELEASE (return for another attempt) or REJECT (unprocessable, never comes back).

Three group parameters govern the behavior, all settable per group with kafka-configs.sh --entity-type groups and clamped by broker minimums and maximums:

share.record.lock.duration.ms: how long the lock lasts. The default is 30 s; once it expires, the record becomes available to another consumer.
share.delivery.count.limit: how many deliveries a record gets before it is archived. The default is 5.
share.partition.max.record.locks: how many records can be locked at once per partition. This is the real parallelism ceiling within a partition.

Kafka 4.2, released on 02/17/2026, declared share groups production-ready and added the RENEW type, to extend the lock during long processing.

In Lambda, the Queue mode ESM uses the Kafka Share Consumer: each event poller reads from all partitions, with no per-partition ordering, and supports up to 10 concurrent invocations, double the Stream mode figure. The event payload stays identical to traditional Kafka, with topic, partition and offset, so the function code does not change.

Where parallelism comes from, and where the failing record goes

The diagram answers one question: in Queue mode, who controls concurrency and who controls failure. Concurrency comes from the number of pollers, not partitions. Failure has two paths: with an on-failure destination configured, Lambda sends the record to SQS; without one, the broker redelivers up to share.delivery.count.limit and then archives it. The dashed arrow is the asynchronous failure path. Simplification: VPC networking and cluster authentication were left out.

Kafka 4.2 self-managed

Producers unchanged (external)
Topic, 12 partitions share group (messaging)

Lambda ESM Queue mode

Event pollers 10 invocations each (compute)
Lambda function idempotent handler (compute)

Side effects

DynamoDB idempotency keys (data)
SQS on-failure destination (messaging)

Flows

producers -> topic: publishes
pollers -> topic: acquires lock / ack
pollers -> fn: invokes in batches
fn -> ddb: conditional write
pollers -> dlq: retries exhausted

Stream and Queue on the same ESM

Criterion	Stream (default)	Queue
Ordering	Guaranteed per partition	None
Invocations per poller	Up to 5	Up to 10
Poller ceiling (provisioned)	Partition count	`MaximumPollers` up to 2,000
On-demand	Scales on offset lag	5 pollers, 5 MB/s or 50 invocations
Starting position	`StartingPosition` accepted	Not supported; uses the share group position
Batching window	Supported	Validation error
Amazon MSK	Yes	Not at launch

The configuration that decides whether it works

Turning it on is one line: SelfManagedKafkaEventSourceConfig.ConsumptionMode: Queue. Operating it is a different conversation. Four decisions make the difference.

Scale: Queue mode on-demand stops at 5 pollers at 1 MB/s each, meaning 5 MB/s or 50 invocations, whichever comes first. For real spikes, use provisioned mode: MinimumPollers from 1 to 200, MaximumPollers from 1 to 2,000, each poller with up to 5 MB/s. In practice the function's concurrency is the limit: 40 pollers times 10 is 400 executions, and the account's default concurrency quota has to absorb that without starving everything else.

Lock versus timeout: the default lock is 30 s. If a 100-record batch takes 45 s in the function, the broker treats the lock as expired and hands the same records to another poller. The documentation I read does not say whether the poller renews the lock, so I treat share.record.lock.duration.ms greater than the function timeout as a rule, and BatchSize small enough to fit inside it.

Cost: provisioned mode bills per EPU-hour, US$ 0.185 in us-east-1 in the documentation's own example, and one EPU holds 10 pollers by default. Queue ESMs only share a PollerGroupName with other Queue ESMs; mixing with Stream returns a validation error.

Backlog: there is no StartingPosition. A new share group starts at the latest offset, so an ESM created over a topic with an accumulated backlog ignores that backlog until someone adjusts share.auto.offset.reset on the group.

The record that disappears without a warning: In Queue mode, MaximumRetryAttempts defaults to 0 and, with no on-failure destination configured, the broker is the one redelivering. After share.delivery.count.limit attempts (5 by default), the record is archived: it does not come back, it does not land in any queue, it raises no error in the function. For billing notifications or fraud alerts, that is a lost event with no trail. Configure the on-failure destination on SQS, SNS or a Kafka topic before the first deploy, not after the first incident.

When to use it, and when not to

Use Queue mode when three things are true at once: each event is an independent unit of work, the handler is idempotent on a business key (not on the offset) and the measured bottleneck is function concurrency, not broker throughput. Per-transaction fraud scoring, push and email delivery, thumbnails and transcoding, telemetry enrichment: all of them pass that test.

Do not use it when state depends on sequence. Per-account ledger postings, order state machines, balance updates, CDC that applies an UPDATE after an INSERT: anything where "event 2 before event 1" produces a wrong number. In a financial system that is not a performance bug, it is a ledger divergence that someone reconciles by hand at month-end close.

Do not use it as a shortcut to avoid understanding your own lag either. If the consumer is behind because the handler makes a synchronous 3 s call to a legacy system with a 20-connection pool, doubling concurrency only moves the queue from Kafka to the legacy system, and now it goes down too. The new parallelism needs an explicit limit at the far end: ReservedConcurrentExecutions on the function or a MaximumPollers value derived from the capacity of whatever receives the call.

And note the launch scope: only self-managed Kafka and Confluent Cloud, on version 4.2 or later with share groups enabled. Amazon MSK waits until it offers that version.

Anti-patterns

Idempotency on the offset: topic-partition-offset only deduplicates broker redelivery. A producer retry creates a new offset for the same payment; the key has to be the transactionId, with attribute_not_exists(pk) in DynamoDB and a TTL aligned with topic retention.
Switching an ordered consumer's mode in production: moving a balance flow from Stream to Queue because lag went up. Throughput improves the same day and reconciliation breaks at month-end close.
Lock shorter than the timeout: a function with a 120 s timeout and share.record.lock.duration.ms at the 30 s default. Every slow batch is delivered twice, and invocation cost doubles exactly at the peak.
Parallelism with no ceiling at the far end: MaximumPollers: 2000 with a downstream that accepts 50 connections. Kafka stops piling up and the database starts refusing.
Mixed poller group: planning shared EPUs across Stream and Queue ESMs. The API refuses it, and the provisioned-mode budget calculated with grouping savings comes out wrong.

Migration without a big bang: Share groups and consumer groups are independent and read the same topic without interfering. Create the Queue ESM pointing at a shadow function that only writes the idempotency key and measures Duration and ConcurrentExecutions, adjust share.auto.offset.reset if you need the backlog, compare for a week and only then disable the Stream ESM. No producer needs to know.

Curator's note: I have seen a topic go from 24 to 192 partitions because of one slow consumer, and that decision outlived the consumer, the team and the cluster version. What I would do today: Queue mode for the notification and scoring flow, an SQS on-failure destination from day one, a lock longer than the timeout and a MaximumPollers value derived from downstream capacity, not from the Kafka peak. The balance flow stays on Stream, with the partitions it already has. The hard-won lesson: the cost of over-partitioning was never creating the partitions, it was living with them for years.

Verdict

Queue mode solves a real problem: it removes the partition from its role as concurrency ceiling without touching producers, topics or function code, and it doubles invocations per poller from 5 to 10. That said, it shifts responsibility: ordering goes away, duplication becomes the rule and, with no on-failure destination, the broker silently archives the record after 5 deliveries. Adopt it for independent work that is idempotent on a business key, on self-managed Kafka 4.2 or Confluent Cloud, with the on-failure destination and lock sized before deploy. Do not adopt it for any flow where sequence defines state; there, Stream with a well-chosen partition count is still the right answer.

Rating: Adotar com condições / Adopt with condit

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

DEV Community