DEV Community: Charan Reddy Rachumalla

Every Step Runs Inside poll(): The Kafka Consumer Flow, End to End

Charan Reddy Rachumalla — Wed, 21 Oct 2026 17:43:33 +0000

Every Kafka consumer application calls poll(). Very few developers can list what happens inside it.

That gap shows up at the worst time: a consumer that hangs, a group that rebalances in a loop, a commit that keeps failing. Each of those failures lives in a specific step of the flow below. The flow is seven steps. The first is setup, the next four are a handshake, and the last two are the steady state the consumer spends its life in.

Step 1: Create consumer and subscribe

Application code builds a KafkaConsumer with bootstrap.servers and a group.id, then calls subscribe() with the topic list.

subscribe() only registers interest. Nothing is fetched, nothing is assigned, no broker knows the consumer exists yet. Every step that follows is driven by poll() — it is the engine of the entire client, and the handshake below unfolds across its first calls.

Step 2: FindCoordinator

The consumer asks any bootstrap broker a simple question: which broker coordinates this group.

The answer is the broker that leads the __consumer_offsets partition assigned to the group. In this walkthrough that is Broker 3, coordinating the group payments-group. From this point, all group traffic — joins, heartbeats, offset commits — goes to Broker 3. None of it goes to the brokers holding the actual data.

Step 3: JoinGroup

The consumer sends a JoinGroup request carrying its protocol preferences and timeouts:

group.id=payments-group
session.timeout.ms=45000
rebalance.timeout.ms=300000

assignment protocols offered: range, cooperative-sticky

The coordinator starts a rebalance and elects a group leader from the members. It issues a generation number — generation 17 in this walkthrough — plus a member ID for each consumer. The generation matters later: it is how the coordinator detects members acting on a stale view of the group.

Step 4: SyncGroup

The elected leader computes the partition assignment for every member and sends it back through the coordinator. SyncGroup returns this consumer's share: partitions payments-0 and payments-1.

Only now does the consumer know what it owns. Everything before this step was negotiation.

Step 5: OffsetFetch

The consumer asks the coordinator — not the partition leaders — where the group left off. The committed offsets come back: payments-0 → 100, payments-1 → 250.

Position starts from the committed offset. If a partition has no committed offset at all, auto.offset.reset decides where to start. That one configuration quietly determines whether a brand-new group reads a topic from the beginning or only sees new records.

Step 6: Fetch

Fetch requests go to each partition's leader broker — this is the data path, separate from the coordinator path. payments-0 is fetched from Broker 1, payments-1 from Broker 2.

Each fetch starts at the consumer's position, and record batches stream back per poll(). Fetch is a long poll: the broker holds the request, waiting up to fetch.max.wait.ms to accumulate at least fetch.min.bytes of data. An idle topic does not mean a stuck consumer. It usually means the broker is patiently waiting to send a batch worth sending.

Step 7: Process and commit

Application code processes the records, then commits. The commit carries the next offset: after processing through offset 109 on payments-0, the committed offset is 110. The coordinator writes it into __consumer_offsets, and that write is the group's durable progress.

A commit can fail on a stale generation. That failure means a rebalance happened mid-processing — the correct response is to rejoin and retry, not to hammer the commit. After a successful commit, the flow loops back to Fetch. Steps 6 and 7 repeat for the life of the consumer. The handshake in steps 2 through 5 only reruns when a rebalance forces it.

Under the hood

The Java client splits this flow across a few key classes:

KafkaConsumer — the API application code touches.
ConsumerCoordinator — group management: FindCoordinator, JoinGroup, SyncGroup, OffsetFetch, OffsetCommit, and heartbeats.
Fetcher — records from the partition leaders.
SubscriptionState — the current assignment and position.
NetworkClient — everything on the wire.

The split explains the two network paths. Group management runs through the coordinator, Broker 3. The data path runs to the leaders, Broker 1 and Broker 2. Committed progress at the end of this walkthrough: payments-0 → 110, payments-1 → 260, stored in __consumer_offsets where the next generation of the group will find it.

Rule of thumb

When a consumer hangs, the useful move is naming the step that broke: the handshake (steps 2–5), the fetch (step 6), or the commit (step 7).

It is almost never the process() code.

Takeaways

subscribe() registers interest and nothing more. poll() drives every step that follows.
Group traffic and data traffic take different paths: coordination to one broker, fetches to the partition leaders.
A rebalance is the handshake rerunning. Generation numbers are how stale members get caught.
A commit stores the next offset to read, and committed offsets in __consumer_offsets are the group's only durable memory.
Consumer problems debug fastest by step: handshake, fetch, or commit.

Kafka MCP Server: How a Copilot Agent Safely Operates a Real Kafka Cluster

Charan Reddy Rachumalla — Wed, 21 Oct 2026 17:36:20 +0000

I built an MCP server for Kafka. It is a backend that connects to Kafka brokers and exposes Kafka operations as tools, defined through the Model Context Protocol, so a Copilot agent can call them. An operator asks a plain-English operational question — a consumer group falling behind, a topic that needs a closer look — and the agent answers from the live cluster instead of from memory.

The interesting part was never the model. The interesting part was everything around it: the tools, the permissions, the rules. This is the design, layer by layer.

Layer 1: You and the copilot

The interaction starts in plain English. The operator states the problem in ordinary language. The agent interprets the intent behind the ask, then picks the right tool from the set it has been given.

That last clause carries the whole design. The agent chooses from a fixed toolset. It does not get a shell, a Kafka CLI, or a blank connection string. Its world is exactly the tools on the server, nothing wider.

Layer 2: The MCP server

The server defines each operation as a tool with a typed schema. Inputs are precise: a topic name is a string with constraints, a group ID is a group ID. There is no free-form parameter for the agent to improvise into.

Two tools show the pattern:

Describe topic returns partitions, configs, ISR, and offsets for a topic.
Consumer lag returns group lag read from live metadata — committed offsets compared against the latest offsets, per partition.

Each tool is a narrow, named operation against the cluster. The server, not the agent, owns the broker connection. That connection authenticates with SASL or OAuth and runs with least-privilege permissions: the credentials can do what the tools need and nothing beyond that.

Layer 3: The Kafka cluster

On the other side sits the cluster itself — brokers, topics and partitions, consumer groups with their membership, offsets, and lag, and the health signals: under-replicated partitions, offline partitions, alerts.

The cluster is the live source of truth. Metadata is fetched fresh on every tool call. The agent never answers from a cached snapshot or from patterns in its training data. A lag number in the response is a lag number the cluster reported a second ago.

Design rule 1: Tools are the contract

A vague schema produces improvised inputs, and improvised inputs against a production cluster are a liability. Typed tools keep the agent honest. The contract is written once, by a person, in the schema — and every call the agent makes is forced through it.

This is the same lesson APIs taught a decade ago, applied to a new kind of caller. The caller is probabilistic. The contract must not be.

Design rule 2: Live state, not guesses

An LLM can produce a fluent explanation of consumer lag with no cluster access at all. It will sound right and it may be wrong. The fix is retrieval from the system itself: answers grounded in metadata the cluster just returned.

Every operational claim the agent makes traces back to a tool result. Strip the tool results away and the agent has nothing to say about this cluster — which is exactly the correct behavior.

Design rule 3: Destructive means a human gate

Read tools can run freely. Anything that changes the cluster stops for a person. Always.

This rule is enforced in the server, not requested in a prompt. Prompts are suggestions. Server-side gating is a mechanism. An agent that cannot reach a destructive operation without human approval is an agent that can be left running next to production.

The RAG side

Tools cover live state. They do not cover knowledge: runbooks, design documents, postmortems, the accumulated written memory of a platform team. That is the retrieval side of the system.

Documents are embedded and stored in a vector store. At ask time, the relevant passages are retrieved and handed to the model — Claude, running on Amazon Bedrock — which answers grounded in the retrieved text. The agent ends up with both halves of an operator's brain: the current state of the cluster from tools, and the team's documented experience from retrieval.

Inference where the data already flows

One more piece fits naturally into this stack. Flink UDFs can run LLM inference in-stream, enriching events as they pass through the pipeline. Not every record needs a model call, and the stream is the right place to decide which ones do. Enrichment happens where the data already flows, instead of in a separate service the data has to be shipped to.

Takeaways

An agent operating infrastructure should work through a fixed set of typed tools, never through open-ended access.
Typed schemas are the contract, and the contract is what keeps a probabilistic caller predictable.
Operational answers must come from live system state, retrieved at call time — never from model memory.
Destructive operations belong behind a human gate enforced by the server, not by the prompt.
Tools give an agent the present. Retrieval gives it the past. A useful operator needs both.
The model is the easy part. The system around it is the job.

Events In, Answers Out: One Pipeline From Kafka Stream to Iceberg Table

Charan Reddy Rachumalla — Wed, 21 Oct 2026 17:35:38 +0000

Most streaming diagrams stop at the stream processor. Data flows in, something processes it, and the diagram ends with an arrow pointing at a box labeled "analytics."

That last box is where the real design work lives. A stream is not something an analyst can query. A table is. The pipeline below is the shape I keep coming back to: sources, Kafka, Flink, Iceberg via TableFlow, Athena, QuickSight. Events in, answers out.

The pipeline at a glance

Six stages, each doing one job:

Sources — applications and services emitting events
Kafka — the topics that hold those events
Apache Flink — real-time processing
Iceberg tables via TableFlow — stored on S3, cataloged in Glue
Athena — plain SQL queries
QuickSight — dashboards

No stage reaches around another. Each one consumes the output of the previous stage and produces the input for the next. That discipline is most of the design.

Stages 1 and 2: Sources and Kafka

Everything starts with applications and services emitting events — an order placed, a payment attempted, a reading reported. Those events land on Kafka topics.

Kafka earns its place here for two reasons. It decouples producers from everything downstream, so a source system never waits on analytics and never knows analytics exists. And it retains the events, so the processing layer can re-read history when logic changes or a job restarts from a failure.

The topic becomes the contract: a named, ordered, replayable record of what happened. Everything downstream trusts the topic instead of trusting the source systems.

Stage 3: Flink does the work in real time

Flink reads from the topics and processes events as they arrive. This is where the raw stream gets shaped: noise filtered out, streams joined, events enriched, values aggregated over time windows.

The important property is timing. The work happens once, at event time, while the data is fresh. The alternative is a batch job that re-reads an entire day of raw events every night and redoes the same shaping work from scratch. By the time records leave Flink, they are already in the shape the table needs.

State lives in Flink for exactly this reason. A join or a window does not care that the data arrived as a stream. It cares that the right records meet each other, and Flink holds them until they do.

Stage 4: TableFlow lands the stream in Iceberg

This is the stage that changed the architecture for me. TableFlow materializes the processed stream as an Apache Iceberg table, continuously. The data files live in Amazon S3. The table is registered in the Glue Catalog, and that catalog entry is what makes it visible to the rest of the analytics stack.

Iceberg matters because it gives files on object storage real table semantics: a defined schema that can evolve, snapshots of the table at points in time, and safe reads while new data keeps arriving. The stream stops being a stream at exactly this point. It becomes a table, and it stays current because the materialization never stops.

The practical win is what disappears. There is no separate ETL job copying Kafka data into a warehouse on a schedule. There is no second version of the pipeline maintained for the analytics side, drifting out of sync with the streaming version. The streaming path and the table are the same path.

Stages 5 and 6: Athena and QuickSight

With the table in the Glue Catalog, Athena queries it with plain SQL. There is no cluster to size and no data to load. The query runs where the data already sits, on S3.

QuickSight sits on top of that for dashboards. The people consuming the data never touch Kafka or Flink. They open a dashboard, or run a SQL query against a table that was a stream moments earlier.

That separation is the point of the whole design. The streaming stack does streaming work. The analytics stack does analytics work. Iceberg and TableFlow are the seam between them, and the seam is a table format and a catalog — boring, standard, inspectable pieces.

Takeaways

A streaming pipeline is only finished when the data lands somewhere queryable. The last box on the diagram deserves the most design attention.
Kafka works best as the durable contract between producers and processing: ordered, replayable, and decoupled from both sides.
Processing in Flink at event time removes the nightly batch re-processing step entirely.
Materializing a stream into Iceberg with TableFlow makes one pipeline serve as both the streaming path and the analytics path.
Analysts should query tables with SQL. They should never need to know a stream exists underneath.