DEV Community: Charan Reddy Rachumalla

Kafka Consumers on EKS: Designing for a Platform That Restarts Your Consumer Group

Charan Reddy Rachumalla — Wed, 28 Oct 2026 14:56:22 +0000

The failure modes that actually page you — and the boring fixes that stop them.

A Kafka consumer is a stateful client built on a membership protocol. It joins a group, accepts a partition assignment, processes records in order, and proves it is still alive on a fixed cadence of heartbeats and poll calls. Every guarantee the group offers — ordered processing, shared load, exactly one owner per partition — rests on that membership staying stable while the work flows through it.

Kubernetes starts from the opposite assumption. A pod is disposable. The platform restarts pods, throttles CPU, rolls deployments, and reclaims nodes on a schedule the consumer group never agreed to, and each of those routine events reaches the group protocol as a member leaving, arriving, or going silent. Most consumer incidents on EKS are not Kafka bugs. They are the collision between a stateful, membership-based client and an orchestrator doing exactly what it was designed to do.

In this post, I start with the baseline configuration that prevents half of these problems outright, then walk through the eight failure modes that account for nearly everything else — symptom, mechanism, and fix, which is usually a config line or a lifecycle hook rather than a redesign. I close with the design principle that ties the eight together and the debug routine that names a failure mode in minutes.

The core tension

A standalone consumer fails when its code fails or when its broker fails. A consumer on EKS carries a third failure source that never sleeps: the platform itself. Rescheduling a pod is normal platform behavior. Enforcing a CPU limit is normal platform behavior. Terminating a container during a rollout is normal platform behavior. The group protocol holds no category for these events except membership change, so it reacts to all of them with the tools it has — rebalances, session expirations, revoked partitions — whether the event was an accident or a planned deploy.

That framing changes how the whole problem gets solved. A restart stops being an exception to recover from and becomes a permanent condition to design around: stable identities, generous timing, and a shutdown path the group can see coming. The eight modes below are what happens when a design skips that framing and assumes the laptop's stability on the platform's terms.

Start with the baseline

Half of these issues never happen with a sane baseline:

session.timeout.ms=10000
heartbeat.interval.ms=3000
max.poll.interval.ms=300000
max.poll.records=500
partition.assignment.strategy=CooperativeStickyAssignor
enable.auto.commit=false
auto.offset.reset=earliest

Three decisions in that block do most of the work. Manual commits put offset progress under the application's control, so a restart replays records instead of silently skipping or duplicating them. Cooperative-sticky assignment keeps partitions with their current owners through a rebalance wherever it can, instead of revoking everything and starting the assignment from zero. A poll interval with real headroom gives slow processing room to finish before the group concludes the member has died. Everything below is what still goes wrong on top of that baseline — and what to do when it does.

The eight failure modes

1. Pod restart and CrashLoop

Symptoms: The pod keeps restarting; the consumer leaves and rejoins the group constantly.

Cause: An application error or an OOMKill; a long startup; uncaught exceptions inside the poll loop. The poll loop is the consumer's heartbeat in the broad sense — the thread that polls is the thread that processes, so an exception that escapes it takes the whole member down, and the group sees a departure to rebalance around.

Fix: Fix resource limits, handle exceptions in the loop, and commit sync or async with idempotent processing, so a restart replays safely instead of corrupting progress. Idempotency is the load-bearing half of that fix: a consumer that can replay a batch without harm treats a restart as a routine event, while a consumer that cannot turns every CrashLoop into a data problem on top of an availability problem.

2. Session timeout — heartbeats missed

Symptoms: Logs report "member has failed"; frequent rebalances; consumption stalls.

Cause: GC pauses or CPU throttling starve the background heartbeat thread. The consumer is alive — it just cannot prove it in time. The heartbeat runs on its own thread precisely so that processing cannot block it. That separation holds only while the scheduler gives the thread its share of CPU; a throttled container breaks it quietly, and the first sign is the group expelling a member that was never down.

Fix: Raise session.timeout.ms, fix CPU and memory limits so the heartbeat thread gets scheduled, and keep poll() being called regularly. Raising the timeout buys tolerance; fixing the limits removes the cause. The timeout increase without the limits change leaves a consumer that fails more slowly instead of one that stops failing.

3. max.poll.interval.ms exceeded

Symptoms: "Consumer is not polling" — the member is kicked out mid-processing.

Cause: Processing per batch takes longer than the poll interval allows, usually because of slow database or API calls inside the loop. The poll interval is the group's second liveness check: it measures not whether the member exists but whether the member is still making progress through its assignment, and slow downstream calls stretch a single batch past the limit without anything being wrong with the consumer itself.

Fix: Raise max.poll.interval.ms, shrink batches with max.poll.records, or hand work to worker threads with pause/resume so the poll loop keeps moving. The three options trade against each other — a longer interval tolerates slower batches, smaller batches bound the work per cycle, and worker threads decouple processing from polling at the cost of more moving parts.

4. Rebalance storms

Symptoms: The group spends more time reassigning partitions than processing records.

Cause: Pods scaling up and down, rolling restarts, short timeouts, and an eager assignor that revokes everything on every change. A rebalance is not free: assignments pause while partitions move, in-flight work gets discarded or replayed, and a group that rebalances in a loop spends its capacity on coordination instead of consumption while lag climbs behind it.

Fix: Use cooperative-sticky assignment, tune the HPA carefully, and stabilize the pod lifecycle — no random restarts. Cooperative assignment revokes only the partitions that actually need to move, and sticky ownership keeps the rest where they are, so a single pod event stops being a group-wide interruption. Autoscaling against a consumer group needs that softness underneath it, because every scale event is a membership event.

5. Offset commit failures

Symptoms: CommitFailedException in the logs; records reprocessed after a restart.

Cause: A stale generation after a rebalance, or the coordinator moved or is unreachable. A commit is a claim about progress made under a specific generation of the group, and the coordinator rejects claims from a generation that no longer exists — the consumer processed the records, and the group has already moved on without recording that fact.

Fix: Commit only inside the poll loop, handle CommitFailedException and retry, and use commitAsync with callbacks carefully. Committing inside the loop keeps the commit attached to the generation that did the work. The reprocessing that follows a rejected commit is the group working as designed; the goal is making replays safe and rare.

6. Resource limits — CPU and memory

Symptoms: High CPU throttling, long GC pauses, OOMKilled pods, missed heartbeats.

Cause: Limits set too low for the batch sizes in use; JVM heap not matched to container memory. This mode sits underneath several of the others — throttling starves the heartbeat thread of mode 2, stretches the batch processing of mode 3, and turns the memory pressure of mode 1 into an OOMKill. A consumer's batch size is also a memory decision, since every record in the batch lives in the heap until the batch finishes.

Fix: Set requests and limits thoughtfully, tune the JVM (-Xmx against container memory, not host memory), right-size message batches, and watch it all with Prometheus and Grafana. The host-memory mistake is the subtle one: a JVM sized for the node it lands on will be killed by a container limit it never knew existed, and the pod restart that follows presents as a consumer problem with a platform cause.

7. DNS and bootstrap servers

Symptoms: UnknownHostException; the consumer cannot connect at startup; intermittent failures after that.

Cause: Wrong bootstrap addresses, CoreDNS issues, or private DNS without the VPC setup to back it. This is the one mode on the list that fails before the group protocol is even involved — the consumer cannot reach a broker to join anything — and the intermittent version is worse than the hard failure, because it presents as flakiness instead of a name-resolution problem with a fixed address.

Fix: Use the correct bootstrap DNS names, verify CoreDNS in the EKS cluster, and check VPC DNS settings and private hosted zones. Every layer of that chain is platform surface, and all of them have to agree before the consumer's own configuration gets a chance to matter.

8. Ungraceful shutdown

Symptoms: Every deploy triggers a full rebalance; partitions get reassigned the slow way.

Cause: Kubernetes kills the pod with no chance to leave the group or commit offsets. From the group's side, a terminated member and a crashed member look identical — both simply stop responding — so the group waits out the session timeout and then reassigns everything, on every single deploy, even though the shutdown was planned in advance.

Fix: Add a PreStop hook that calls close(), commit offsets before shutdown, and set terminationGracePeriodSeconds generously enough for the close to finish. A clean close() is the consumer announcing its departure: the group rebalances immediately and narrowly, offsets land before the pod dies, and a deploy stops looking like an outage. The grace period is the budget that makes the announcement possible, and it has to be long enough for the close to actually complete.

Designing for the platform

None of these eight modes is exotic, and that is the point. Read together, they reduce to a short list of design commitments. Timing gets budgeted as a first-class resource, sized for a platform that throttles and pauses things rather than a quiet laptop. Shutdown becomes a feature with its own code path instead of an abrupt disappearance the group has to time out. Identity stays as stable as the platform allows, so sticky assignments have something to stick to. And processing is written to survive replay, because replays are the group's recovery mechanism and no configuration makes them fully disappear.

The tradeoffs stay honest. Generous timeouts tolerate platform noise and also delay detection of a genuinely dead member. Cooperative-sticky assignment calms rebalances and still cannot hold a partition whose owner is gone. Bigger resource limits end throttling and cost money on every pod, every hour. The design point is choosing these tensions deliberately, not eliminating them.

Debug fast

kubectl get pods -n kafka-consumers
kubectl logs <pod>
kubectl get events
kafka-consumer-groups.sh --describe --group <group>

Lag plus member state tells you which issue you are in. Name the issue first; the fix follows from the name. The pod list and events stream show the platform's half of the story — restarts, kills, throttling — and the group description shows the protocol's half: who owns what, how far behind each partition sits, and whether membership is stable or churning.

What changes in practice

With the baseline in place and the eight modes handled, the operational picture changes in specific ways. Deploys stop triggering full rebalances, because the shutdown path announces the departure and the group reassigns narrowly. Restarts replay records safely instead of corrupting progress, because commits are manual and processing is idempotent. Timing stops being the hidden cause behind pages that present as broker problems. And diagnosis gets faster: lag and member state from kafka-consumer-groups.sh --describe identify which of the eight modes is active in minutes, and the fix follows from the name.

Takeaways

On EKS, the platform is part of the consumer's failure model: restarts, throttling, and shutdowns are normal events, not exceptions.
Most consumer pages trace back to timing — heartbeats, poll intervals, and shutdown grace — not to broker problems.
Cooperative-sticky assignment and manual commits remove the two loudest sources of churn before tuning even starts.
A graceful shutdown path is a feature: PreStop hook, a final commit, and a grace period long enough to use them.
Debug by symptom against the eight modes above; lag and member state from kafka-consumer-groups.sh --describe identify the mode in minutes.

What's next

The same failure-model thinking extends to every stateful client the platform hosts. Any process that joins a group, holds an assignment, and proves liveness on a cadence will meet its own version of these eight modes, and the fixes generalize with it: budget the timing, announce the shutdown, keep the identity stable, and make the work replayable. The consumer is simply where the collision shows up first, because the group protocol writes its assumptions down in configuration — and configuration can be reviewed line by line.

The platform is part of the failure model. Design for it there, in the open, and the pages stop.

Tiered Storage in Apache Kafka — Hot Data on Disk, Old Data in S3 (and the Same Trick That Cuts a Databricks Bill)

Charan Reddy Rachumalla — Mon, 26 Oct 2026 12:32:23 +0000

Every Kafka cluster I've worked on had the same problem. Not throughput. Disk.

The business wants a year of history kept. The brokers can hold maybe a week. So retention gets cut, data gets deleted, and three months later somebody asks for it.

Tiered storage fixed this in open-source Kafka, and the idea is honestly simple.

Recent data stays on the broker's local disk. That's the hot data, the stuff consumers are actually reading all day. When a segment is done and gets old, Kafka ships it to S3 and removes the local copy.

Nobody notices. Same topic, same offsets. A consumer can still read six-month-old data — the broker just pulls it back from S3 when asked. It's slower, sure. Old data always is. But it works, and we didn't have to build some separate archiving pipeline on the side to make it happen.

On EKS this setup is almost unfair. Tiny fast disks on the brokers, S3 doing the heavy lifting behind them. Retention stops being a math problem you solve with a spreadsheet and a prayer.

On-prem works too. The storage layer is pluggable, so MinIO, Ceph, whatever S3-compatible box is already sitting in the data center — it all works the same way.

And the same thinking saves money outside Kafka. Databricks is where I've seen storage bills quietly explode. Delta keeps old versions of files around for time travel, old partitions sit on full-price storage, and nobody ever goes back to clean it up.

The workaround is the same idea. Leave the recent data in Delta where it's fast. Take the old partitions, write them out once as plain Parquet into an archive bucket, and let an S3 lifecycle rule push them down to Glacier pricing. Drop them from the table. If someone actually needs last year's data — and that happens maybe twice a year — read the Parquet directly. Ten minutes of work beats paying hot-storage prices forever.

One honest warning. Tiered storage is not a backup. Delete the topic and the S3 data goes with it. It solves cost. It doesn't solve accidents.

Hot data where it's read. Cold data where it's cheap. That's the whole trick.

The system design

Write path. Producers write to the leaders as always. Nothing about the hot path changes — records land in the active log segment on local disk and are acknowledged per your normal acks and replication settings.

Offload path. When a segment closes and ages past the local retention window, the broker uploads it to remote storage through the Remote Storage Manager (RSM), records where it went in the Remote Log Metadata Manager (RLMM — an internal Kafka topic in the production setup), and only then deletes the local copy. Uploads are chunked (4 MiB here) so a multi-GB segment is many small, resumable S3 writes, not one giant one.

Read path. A fetch for a recent offset is served from local disk, exactly as before. A fetch for an old offset is resolved through the RLMM to the segment's S3 location; the broker streams the needed chunks back (through a local chunk cache) and serves them as a normal fetch response. The consumer cannot tell the difference — only the latency changes.

The two retention knobs are the whole design:

Setting	Meaning
`local.retention.ms` / `local.retention.bytes`	How long data stays on broker disk — the hot window
`retention.ms` / `retention.bytes`	Total history kept, local + remote together

On EKS the brokers run as a StatefulSet with small gp3 volumes, credentials come from IRSA (no keys in the pod), and the S3 plugin arrives via an init container. On-prem, point the same RSM at MinIO or Ceph — the design doesn't change, only the endpoint.

The configs

1. Broker config — `server.properties`

Apache Kafka ships the tiered-storage framework (KIP-405) but no S3 implementation, so the Remote Storage Manager is Aiven's open-source plugin — the standard choice.

# Turn the tiered-storage system on for this broker
remote.log.storage.system.enable=true

# Remote Log Metadata Manager: topic-based (production choice)
remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
remote.log.metadata.manager.impl.prefix=rlmm.config.
rlmm.config.remote.log.metadata.topic.replication.factor=3
rlmm.config.remote.log.metadata.topic.num.partitions=50

# Remote Storage Manager: Aiven plugin, S3 backend
remote.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
remote.storage.manager.impl.prefix=rsm.config.
rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3StorageBackend
rsm.config.storage.s3.bucket.name=my-kafka-tiered-storage
rsm.config.storage.s3.region=us-east-1
# Credentials come from the environment (on EKS: the pod's IRSA role).
# Never put keys in this file.

# 4 MiB chunks keep S3 requests sane on big segments
rsm.config.chunk.size=4194304

2. Per-topic opt-in — `topic-config.sh`

kafka-configs.sh --bootstrap-server kafka-0.kafka-headless:9092,kafka-1.kafka-headless:9092,kafka-2.kafka-headless:9092 \
  --entity-type topics --entity-name events --alter \
  --add-config remote.storage.enable=true,\
local.retention.ms=86400000,\
retention.ms=15552000000
# local.retention.ms = 1 day on disk (hot window)
# retention.ms       = 180 days total history

3. The pod spec — `kafka-on-eks.yaml`

The number to look at: 50 Gi of disk. Without tiered storage this same topic needed terabytes per broker.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kafka
  namespace: kafka
  annotations:
    # IRSA: this role carries the S3 permissions (see iam-policy.json).
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/kafka-tiered-storage
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kafka-config
  namespace: kafka
data:
  server.properties: |
    process.roles=broker,controller
    node.id=0
    controller.quorum.voters=0@kafka-0.kafka-headless.kafka.svc:9093,1@kafka-1.kafka-headless.kafka.svc:9093,2@kafka-2.kafka-headless.kafka.svc:9093
    listeners=PLAINTEXT://:9092,CONTROLLER://:9093
    advertised.listeners=PLAINTEXT://kafka-0.kafka-headless.kafka.svc:9092
    log.dirs=/var/lib/kafka/data
    num.partitions=12
    remote.log.storage.system.enable=true
    remote.log.metadata.manager.class.name=org.apache.kafka.server.log.remote.metadata.storage.TopicBasedRemoteLogMetadataManager
    remote.storage.manager.class.name=io.aiven.kafka.tieredstorage.RemoteStorageManager
    remote.storage.manager.impl.prefix=rsm.config.
    rsm.config.storage.backend.class=io.aiven.kafka.tieredstorage.storage.s3.S3StorageBackend
    rsm.config.storage.s3.bucket.name=my-kafka-tiered-storage
    rsm.config.storage.s3.region=us-east-1
    rsm.config.chunk.size=4194304
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: kafka
spec:
  serviceName: kafka-headless
  replicas: 3
  selector:
    matchLabels: { app: kafka }
  template:
    metadata:
      labels: { app: kafka }
    spec:
      serviceAccountName: kafka
      initContainers:
        # Kafka ships no S3 RSM — fetch Aiven's open-source plugin jars.
        - name: fetch-tiered-storage-plugin
          image: curlimages/curl:8.10.1
          command: ["sh", "-c", "cd /plugin && curl -sSLO https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/releases/download/v0.10.0/tiered-storage-for-apache-kafka-0.10.0.tgz && tar xzf tiered-storage-for-apache-kafka-0.10.0.tgz --strip-components=1"]
          volumeMounts:
            - { name: plugin, mountPath: /plugin }
      containers:
        - name: kafka
          image: apache/kafka:3.9.0
          ports:
            - { containerPort: 9092, name: client }
            - { containerPort: 9093, name: controller }
          volumeMounts:
            - { name: data, mountPath: /var/lib/kafka/data }
            - { name: config, mountPath: /opt/kafka/config/server.properties, subPath: server.properties }
            - { name: plugin, mountPath: /opt/kafka/libs/tiered-storage }
          resources:
            requests: { cpu: "2", memory: 4Gi }
            limits:   { cpu: "4", memory: 8Gi }
      volumes:
        - name: config
          configMap: { name: kafka-config }
        - name: plugin
          emptyDir: {}
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp3
        resources:
          requests:
            storage: 50Gi   # plenty, when disk only holds the hot window

4. The only S3 permissions the brokers need — `iam-policy.json`

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "ListTieredStorageBucket", "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::my-kafka-tiered-storage" },
    { "Sid": "ReadWriteSegments", "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-kafka-tiered-storage/*" }
  ]
}

The Databricks half: storage workaround + EKS compute split

Storage. The archive job below runs weekly: old Delta partitions are written once as plain Parquet into an archive bucket, deleted from the table, and VACUUMed so the space is genuinely released. The bucket's lifecycle rule then does the pricing work.

# databricks_archive_job.py (essentials)
from pyspark.sql import functions as F

HOT_WINDOW_DAYS = 180
TABLE = "prod.events"
ARCHIVE = "s3://my-archive-bucket/archive/events"
DATE_COL = "event_date"

cutoff = F.date_sub(F.current_date(), HOT_WINDOW_DAYS)
old = spark.read.table(TABLE).filter(F.col(DATE_COL) < cutoff)

# Land the copy BEFORE touching the table
old.write.mode("overwrite").partitionBy(DATE_COL).parquet(ARCHIVE)

spark.sql(f"DELETE FROM {TABLE} WHERE {DATE_COL} < date_sub(current_date(), {HOT_WINDOW_DAYS})")
spark.sql(f"VACUUM {TABLE} RETAIN 168 HOURS")   # 7 days of time-travel grace

# Reading it back, the rare time anyone asks:
#   spark.read.parquet("s3://my-archive-bucket/archive/events")

// s3-lifecycle-archive.json
{ "Rules": [ { "ID": "archive-to-glacier", "Status": "Enabled",
  "Filter": { "Prefix": "archive/" },
  "Transitions": [
    { "Days": 30,  "StorageClass": "GLACIER_IR" },
    { "Days": 180, "StorageClass": "DEEP_ARCHIVE" } ] } ] }

Compute. The archive reads, backfills, and compaction don't need Databricks at all. You can't run Databricks on EKS — its compute is Databricks-managed — but you don't need to: Delta Lake is open source, and the same tables and Parquet archives on S3 are readable and writable by plain Apache Spark running as EKS pods, at raw EC2 prices, with zero DBUs. Keep Databricks for the interactive work where its speed earns the markup; let cheap pods do the scheduled heavy lifting against the cold tiers, scaling to zero in between.

The evidence

What the storage tiers actually cost (AWS list prices, us-east-1):

Tier	$ / GB-month	20 TB of history
S3 Standard (and broker-disk equivalent thinking)	$0.023	~$460 / month
S3 Glacier Instant Retrieval	$0.004	~$80 / month
S3 Glacier Deep Archive	$0.00099	~$20 / month

Worked example — 20 TB total history, 1-day hot window: ~1 TB stays hot (~$23), 19 TB sits in Glacier Instant Retrieval (~$76). ≈ $99/month instead of ≈ $460 — about 78% off the storage line, before counting the broker disks you no longer buy (50 Gi gp3 per broker above, versus terabytes).

What Databricks actually charges: every workload pays twice — the cloud VM cost plus a per-DBU charge (Flexera's 2026 Databricks pricing guide: Jobs compute $0.15/DBU Premium, serverless jobs $0.35/DBU, all-purpose compute $0.40/DBU). All-in, Databricks typically lands at roughly 1.5–2.5× the cost of the raw compute it runs on (Reintech, Apache Spark vs Databricks 2026 comparison). Moving a scheduled batch job to Spark-on-EKS removes the entire DBU half of that bill for that job.

Why the split is legal, technically: Delta Lake is an open-source Linux Foundation project — "not locked to Databricks and can be used anywhere" (Databricks' own Delta Lake overview). External engines — Apache Spark batch and streaming, Flink, DuckDB — can now create, read, and write even Unity Catalog–managed Delta tables with commits coordinated by Unity Catalog (Databricks Community technical blog, 2026).

The honest limits: open-source Delta lacks Databricks-only optimizations (Photon, predictive I/O); Glacier Deep Archive objects must be restored in S3 before they can be read, which is why the lifecycle stops at Instant Retrieval first; self-managed Spark on EKS trades engineer time for the DBU savings; and tiered storage — Kafka or Delta — is a cost tool, not a backup.

Sources: Flexera, "Databricks pricing guide (2026)"; Reintech, "Apache Spark vs Databricks 2026"; Dremio, "Keep Databricks for ETL. Cut Analytics Costs"; Databricks Community, "Performing DML Operations on Unity Catalog Managed Tables from External Engines"; Delta Lake documentation (docs.delta.io); AWS S3 pricing (list, us-east-1). Config files: companion bundle kafka-tiered-storage/.

State, Not Events: Keeping a Kafka Topic Forever With Log Compaction

Charan Reddy Rachumalla — Fri, 23 Oct 2026 13:29:33 +0000

Two different systems. The same two lines of Kafka topic configuration.

retention.ms = -1
log.cleanup.policy = compact

Most Kafka topics are streams of events with a shelf life. Data lands, consumers read it, and after a few days or weeks the retention window deletes it. These two settings build the opposite thing. Infinite retention says records never expire by time. Log compaction says the broker keeps only the latest record for each key. Together they turn a topic into a permanent table: one current record per key, kept forever, replayable from the beginning at any moment.

I have used this combination in two production systems that look nothing alike. The pattern behind both is the same, and it is the subject of this article.

What the configuration actually does

With log.cleanup.policy=compact, the log cleaner works through the topic's older segments and drops any record whose key has a newer record later in the log. The newest value for every key always survives. With retention.ms=-1, no time-based deletion ever removes what compaction keeps. The result is a topic whose size tracks the number of distinct keys, not the volume of writes. A million updates to one key still leave one record.

That gives the topic a property normal topics do not have: a consumer that reads it from the beginning reconstructs the complete current state. Not the history of how the state evolved — the state itself, exactly as it stands now. Any service can rebuild its view of the world by replaying the topic, and a brand-new consumer catches up to the same picture as everyone else.

Project one: connector configs as a compacted topic

The first system is a self-service tool for Kafka connectors. It is a Flask application. Users sign in with SSO and see cards for the connector types they can create — Salesforce, Google, MirrorMaker 2, and other Camel sources and sinks. Picking a card opens a form with the fields that connector needs. The backend is an open-source Kafka Connect distributed cluster running on EKS, loaded with the Camel connector JARs.

When a user submits the form, the tool produces one record to a Kafka topic. The key is the connector name. The value is the connector config. That topic runs with infinite retention and compaction.

This makes the topic the registry of every connector in the system. A connector's config can never age out and disappear, because retention is infinite. An updated config does not pile duplicates on top of the old one, because compaction keeps exactly one live record per connector name — the latest write wins. The full set of connector definitions lives in one durable place, in the same system that runs them, with no separate database to keep in sync.

Project two: user data for log enrichment

The second system starts with an identity tool that emits user data logs — records for new users and updated records for existing users. Every value carries one unique user with all of their details. This data has to be kept forever, and there must never be duplicates for the same user.

Same configuration. The key is the user. The value is the full user record. Infinite retention keeps every user's current record permanently. Compaction guarantees that no matter how many updates a user accumulates, exactly one record per user survives — the current one.

This topic feeds a bigger pipeline: a log aggregation system that processes server logs from across the platform. As server logs stream through, the pipeline enriches them with user data, joining each log against the latest user record from the topic. Enrichment is only trustworthy if the reference data is complete and current, and that is exactly what the compacted topic provides. It is the full user base, deduplicated by construction, and replaying it from the beginning rebuilds that user base exactly as it stands today.

The rule behind both

Configs and user profiles share one trait: they describe state, not history. The useful question they answer is "what does this look like right now." Nobody needs the fourteen previous versions of a connector config or a user's profile. They need the current one, guaranteed present, guaranteed unique.

That is the decision test I apply. Data that describes events — things that happened, in order, where the history itself is the value — belongs in a normal topic with time-based retention. Data that describes current state belongs in a compacted topic with infinite retention. Events get retention windows. State gets compaction.

Two operational notes complete the picture. Compaction is a background process, so a superseded record can sit in the log for a while before the cleaner removes it; the guarantee is about what survives, not about instant cleanup. And a compacted topic still grows with the number of keys, so it fits state with a bounded key space — connector names, user IDs — rather than unbounded event streams.

Used this way, Kafka stops being only a pipe between systems. A compacted, infinitely retained topic is a durable, replayable store of current state that any consumer can rebuild from at any time. Two lines of configuration, and the topic never forgets what things look like right now.

Every Step Runs Inside poll(): The Kafka Consumer Flow, End to End

Charan Reddy Rachumalla — Wed, 21 Oct 2026 17:43:33 +0000

Every Kafka consumer application calls poll(). Very few developers can list what happens inside it.

That gap shows up at the worst time: a consumer that hangs, a group that rebalances in a loop, a commit that keeps failing. Each of those failures lives in a specific step of the flow below. The flow is seven steps. The first is setup, the next four are a handshake, and the last two are the steady state the consumer spends its life in.

Step 1: Create consumer and subscribe

Application code builds a KafkaConsumer with bootstrap.servers and a group.id, then calls subscribe() with the topic list.

subscribe() only registers interest. Nothing is fetched, nothing is assigned, no broker knows the consumer exists yet. Every step that follows is driven by poll() — it is the engine of the entire client, and the handshake below unfolds across its first calls.

Step 2: FindCoordinator

The consumer asks any bootstrap broker a simple question: which broker coordinates this group.

The answer is the broker that leads the __consumer_offsets partition assigned to the group. In this walkthrough that is Broker 3, coordinating the group payments-group. From this point, all group traffic — joins, heartbeats, offset commits — goes to Broker 3. None of it goes to the brokers holding the actual data.

Step 3: JoinGroup

The consumer sends a JoinGroup request carrying its protocol preferences and timeouts:

group.id=payments-group
session.timeout.ms=45000
rebalance.timeout.ms=300000

assignment protocols offered: range, cooperative-sticky

The coordinator starts a rebalance and elects a group leader from the members. It issues a generation number — generation 17 in this walkthrough — plus a member ID for each consumer. The generation matters later: it is how the coordinator detects members acting on a stale view of the group.

Step 4: SyncGroup

The elected leader computes the partition assignment for every member and sends it back through the coordinator. SyncGroup returns this consumer's share: partitions payments-0 and payments-1.

Only now does the consumer know what it owns. Everything before this step was negotiation.

Step 5: OffsetFetch

The consumer asks the coordinator — not the partition leaders — where the group left off. The committed offsets come back: payments-0 → 100, payments-1 → 250.

Position starts from the committed offset. If a partition has no committed offset at all, auto.offset.reset decides where to start. That one configuration quietly determines whether a brand-new group reads a topic from the beginning or only sees new records.

Step 6: Fetch

Fetch requests go to each partition's leader broker — this is the data path, separate from the coordinator path. payments-0 is fetched from Broker 1, payments-1 from Broker 2.

Each fetch starts at the consumer's position, and record batches stream back per poll(). Fetch is a long poll: the broker holds the request, waiting up to fetch.max.wait.ms to accumulate at least fetch.min.bytes of data. An idle topic does not mean a stuck consumer. It usually means the broker is patiently waiting to send a batch worth sending.

Step 7: Process and commit

Application code processes the records, then commits. The commit carries the next offset: after processing through offset 109 on payments-0, the committed offset is 110. The coordinator writes it into __consumer_offsets, and that write is the group's durable progress.

A commit can fail on a stale generation. That failure means a rebalance happened mid-processing — the correct response is to rejoin and retry, not to hammer the commit. After a successful commit, the flow loops back to Fetch. Steps 6 and 7 repeat for the life of the consumer. The handshake in steps 2 through 5 only reruns when a rebalance forces it.

Under the hood

The Java client splits this flow across a few key classes:

KafkaConsumer — the API application code touches.
ConsumerCoordinator — group management: FindCoordinator, JoinGroup, SyncGroup, OffsetFetch, OffsetCommit, and heartbeats.
Fetcher — records from the partition leaders.
SubscriptionState — the current assignment and position.
NetworkClient — everything on the wire.

The split explains the two network paths. Group management runs through the coordinator, Broker 3. The data path runs to the leaders, Broker 1 and Broker 2. Committed progress at the end of this walkthrough: payments-0 → 110, payments-1 → 260, stored in __consumer_offsets where the next generation of the group will find it.

Rule of thumb

When a consumer hangs, the useful move is naming the step that broke: the handshake (steps 2–5), the fetch (step 6), or the commit (step 7).

It is almost never the process() code.

Takeaways

subscribe() registers interest and nothing more. poll() drives every step that follows.
Group traffic and data traffic take different paths: coordination to one broker, fetches to the partition leaders.
A rebalance is the handshake rerunning. Generation numbers are how stale members get caught.
A commit stores the next offset to read, and committed offsets in __consumer_offsets are the group's only durable memory.
Consumer problems debug fastest by step: handshake, fetch, or commit.

Kafka MCP Server: How a Copilot Agent Safely Operates a Real Kafka Cluster

Charan Reddy Rachumalla — Wed, 21 Oct 2026 17:36:20 +0000

I built an MCP server for Kafka. It is a backend that connects to Kafka brokers and exposes Kafka operations as tools, defined through the Model Context Protocol, so a Copilot agent can call them. An operator asks a plain-English operational question — a consumer group falling behind, a topic that needs a closer look — and the agent answers from the live cluster instead of from memory.

The interesting part was never the model. The interesting part was everything around it: the tools, the permissions, the rules. This is the design, layer by layer.

Layer 1: You and the copilot

The interaction starts in plain English. The operator states the problem in ordinary language. The agent interprets the intent behind the ask, then picks the right tool from the set it has been given.

That last clause carries the whole design. The agent chooses from a fixed toolset. It does not get a shell, a Kafka CLI, or a blank connection string. Its world is exactly the tools on the server, nothing wider.

Layer 2: The MCP server

The server defines each operation as a tool with a typed schema. Inputs are precise: a topic name is a string with constraints, a group ID is a group ID. There is no free-form parameter for the agent to improvise into.

Two tools show the pattern:

Describe topic returns partitions, configs, ISR, and offsets for a topic.
Consumer lag returns group lag read from live metadata — committed offsets compared against the latest offsets, per partition.

Each tool is a narrow, named operation against the cluster. The server, not the agent, owns the broker connection. That connection authenticates with SASL or OAuth and runs with least-privilege permissions: the credentials can do what the tools need and nothing beyond that.

Layer 3: The Kafka cluster

On the other side sits the cluster itself — brokers, topics and partitions, consumer groups with their membership, offsets, and lag, and the health signals: under-replicated partitions, offline partitions, alerts.

The cluster is the live source of truth. Metadata is fetched fresh on every tool call. The agent never answers from a cached snapshot or from patterns in its training data. A lag number in the response is a lag number the cluster reported a second ago.

Design rule 1: Tools are the contract

A vague schema produces improvised inputs, and improvised inputs against a production cluster are a liability. Typed tools keep the agent honest. The contract is written once, by a person, in the schema — and every call the agent makes is forced through it.

This is the same lesson APIs taught a decade ago, applied to a new kind of caller. The caller is probabilistic. The contract must not be.

Design rule 2: Live state, not guesses

An LLM can produce a fluent explanation of consumer lag with no cluster access at all. It will sound right and it may be wrong. The fix is retrieval from the system itself: answers grounded in metadata the cluster just returned.

Every operational claim the agent makes traces back to a tool result. Strip the tool results away and the agent has nothing to say about this cluster — which is exactly the correct behavior.

Design rule 3: Destructive means a human gate

Read tools can run freely. Anything that changes the cluster stops for a person. Always.

This rule is enforced in the server, not requested in a prompt. Prompts are suggestions. Server-side gating is a mechanism. An agent that cannot reach a destructive operation without human approval is an agent that can be left running next to production.

The RAG side

Tools cover live state. They do not cover knowledge: runbooks, design documents, postmortems, the accumulated written memory of a platform team. That is the retrieval side of the system.

Documents are embedded and stored in a vector store. At ask time, the relevant passages are retrieved and handed to the model — Claude, running on Amazon Bedrock — which answers grounded in the retrieved text. The agent ends up with both halves of an operator's brain: the current state of the cluster from tools, and the team's documented experience from retrieval.

Inference where the data already flows

One more piece fits naturally into this stack. Flink UDFs can run LLM inference in-stream, enriching events as they pass through the pipeline. Not every record needs a model call, and the stream is the right place to decide which ones do. Enrichment happens where the data already flows, instead of in a separate service the data has to be shipped to.

Takeaways

An agent operating infrastructure should work through a fixed set of typed tools, never through open-ended access.
Typed schemas are the contract, and the contract is what keeps a probabilistic caller predictable.
Operational answers must come from live system state, retrieved at call time — never from model memory.
Destructive operations belong behind a human gate enforced by the server, not by the prompt.
Tools give an agent the present. Retrieval gives it the past. A useful operator needs both.
The model is the easy part. The system around it is the job.

Events In, Answers Out: One Pipeline From Kafka Stream to Iceberg Table

Charan Reddy Rachumalla — Wed, 21 Oct 2026 17:35:38 +0000

Most streaming diagrams stop at the stream processor. Data flows in, something processes it, and the diagram ends with an arrow pointing at a box labeled "analytics."

That last box is where the real design work lives. A stream is not something an analyst can query. A table is. The pipeline below is the shape I keep coming back to: sources, Kafka, Flink, Iceberg via TableFlow, Athena, QuickSight. Events in, answers out.

The pipeline at a glance

Six stages, each doing one job:

Sources — applications and services emitting events
Kafka — the topics that hold those events
Apache Flink — real-time processing
Iceberg tables via TableFlow — stored on S3, cataloged in Glue
Athena — plain SQL queries
QuickSight — dashboards

No stage reaches around another. Each one consumes the output of the previous stage and produces the input for the next. That discipline is most of the design.

Stages 1 and 2: Sources and Kafka

Everything starts with applications and services emitting events — an order placed, a payment attempted, a reading reported. Those events land on Kafka topics.

Kafka earns its place here for two reasons. It decouples producers from everything downstream, so a source system never waits on analytics and never knows analytics exists. And it retains the events, so the processing layer can re-read history when logic changes or a job restarts from a failure.

The topic becomes the contract: a named, ordered, replayable record of what happened. Everything downstream trusts the topic instead of trusting the source systems.

Stage 3: Flink does the work in real time

Flink reads from the topics and processes events as they arrive. This is where the raw stream gets shaped: noise filtered out, streams joined, events enriched, values aggregated over time windows.

The important property is timing. The work happens once, at event time, while the data is fresh. The alternative is a batch job that re-reads an entire day of raw events every night and redoes the same shaping work from scratch. By the time records leave Flink, they are already in the shape the table needs.

State lives in Flink for exactly this reason. A join or a window does not care that the data arrived as a stream. It cares that the right records meet each other, and Flink holds them until they do.

Stage 4: TableFlow lands the stream in Iceberg

This is the stage that changed the architecture for me. TableFlow materializes the processed stream as an Apache Iceberg table, continuously. The data files live in Amazon S3. The table is registered in the Glue Catalog, and that catalog entry is what makes it visible to the rest of the analytics stack.

Iceberg matters because it gives files on object storage real table semantics: a defined schema that can evolve, snapshots of the table at points in time, and safe reads while new data keeps arriving. The stream stops being a stream at exactly this point. It becomes a table, and it stays current because the materialization never stops.

The practical win is what disappears. There is no separate ETL job copying Kafka data into a warehouse on a schedule. There is no second version of the pipeline maintained for the analytics side, drifting out of sync with the streaming version. The streaming path and the table are the same path.

Stages 5 and 6: Athena and QuickSight

With the table in the Glue Catalog, Athena queries it with plain SQL. There is no cluster to size and no data to load. The query runs where the data already sits, on S3.

QuickSight sits on top of that for dashboards. The people consuming the data never touch Kafka or Flink. They open a dashboard, or run a SQL query against a table that was a stream moments earlier.

That separation is the point of the whole design. The streaming stack does streaming work. The analytics stack does analytics work. Iceberg and TableFlow are the seam between them, and the seam is a table format and a catalog — boring, standard, inspectable pieces.

Takeaways

A streaming pipeline is only finished when the data lands somewhere queryable. The last box on the diagram deserves the most design attention.
Kafka works best as the durable contract between producers and processing: ordered, replayable, and decoupled from both sides.
Processing in Flink at event time removes the nightly batch re-processing step entirely.
Materializing a stream into Iceberg with TableFlow makes one pipeline serve as both the streaming path and the analytics path.
Analysts should query tables with SQL. They should never need to know a stream exists underneath.

DEV Community: Charan Reddy Rachumalla

Kafka Consumers on EKS: Designing for a Platform That Restarts Your Consumer Group

The core tension

Start with the baseline

The eight failure modes

1. Pod restart and CrashLoop

2. Session timeout — heartbeats missed

3. max.poll.interval.ms exceeded

4. Rebalance storms

5. Offset commit failures

6. Resource limits — CPU and memory

7. DNS and bootstrap servers

8. Ungraceful shutdown

Designing for the platform

Debug fast

What changes in practice

Takeaways

What's next

Tiered Storage in Apache Kafka — Hot Data on Disk, Old Data in S3 (and the Same Trick That Cuts a Databricks Bill)

The system design

The configs

1. Broker config — server.properties

2. Per-topic opt-in — topic-config.sh

3. The pod spec — kafka-on-eks.yaml

4. The only S3 permissions the brokers need — iam-policy.json

The Databricks half: storage workaround + EKS compute split

The evidence

State, Not Events: Keeping a Kafka Topic Forever With Log Compaction

What the configuration actually does

Project one: connector configs as a compacted topic

Project two: user data for log enrichment

The rule behind both

Every Step Runs Inside poll(): The Kafka Consumer Flow, End to End

Step 1: Create consumer and subscribe

Step 2: FindCoordinator

Step 3: JoinGroup

assignment protocols offered: range, cooperative-sticky

Step 4: SyncGroup

Step 5: OffsetFetch

Step 6: Fetch

Step 7: Process and commit

Under the hood

Rule of thumb

Takeaways

Kafka MCP Server: How a Copilot Agent Safely Operates a Real Kafka Cluster

Layer 1: You and the copilot

Layer 2: The MCP server

Layer 3: The Kafka cluster

Design rule 1: Tools are the contract

Design rule 2: Live state, not guesses

Design rule 3: Destructive means a human gate

The RAG side

Inference where the data already flows

Takeaways

Events In, Answers Out: One Pipeline From Kafka Stream to Iceberg Table

The pipeline at a glance

Stages 1 and 2: Sources and Kafka

Stage 3: Flink does the work in real time

Stage 4: TableFlow lands the stream in Iceberg

Stages 5 and 6: Athena and QuickSight

Takeaways

1. Broker config — `server.properties`

2. Per-topic opt-in — `topic-config.sh`

3. The pod spec — `kafka-on-eks.yaml`

4. The only S3 permissions the brokers need — `iam-policy.json`