Originally published on Naveed Ahmed's Engineering Blog.
Apache Kafka is often described as a messaging system.
That's partly true, but understanding it as a distributed event log gives you a much clearer picture of how it works.
Producers write records to partitioned topics. Brokers store those records, and consumers read them at their own pace.
Unlike a traditional work queue, Kafka can retain events after they've been consumed, allowing different applications to process or replay them independently.
Let's walk through the key concepts and a small hands-on example.
1. How Apache Kafka Works
A simplified architecture looks like this:
Producer
|
v
Kafka Topic
|
+-- Partition 0
+-- Partition 1
+-- Partition 2
|
v
Consumer Group
Each partition is an ordered sequence of records.
Kafka uses offsets to identify record positions within a partition.
If you have three partitions, you can have up to three actively assigned consumers within one consumer group for that topic.
Adding more consumers than partitions doesn't create additional partition-processing parallelism.
2. What Is KRaft?
Modern Apache Kafka uses KRaft to manage cluster metadata without an external ZooKeeper deployment.
KRaft controllers form a quorum responsible for metadata decisions, while brokers handle client requests and partition storage.
This separation matters when you're investigating broker failures, partition leadership changes, or controller health.
3. Start Kafka with Docker
For a quick local test, you can start Kafka with the official Docker image:
docker run -d \
--name kafka-local \
-p 127.0.0.1:9092:9092 \
apache/kafka:4.4.0
This example uses the image's default single-node configuration for development.
Create a topic:
docker exec kafka-local \
/opt/kafka/bin/kafka-topics.sh \
--create \
--topic orders.events \
--partitions 3 \
--replication-factor 1 \
--bootstrap-server localhost:9092
Inspect it:
docker exec kafka-local \
/opt/kafka/bin/kafka-topics.sh \
--describe \
--topic orders.events \
--bootstrap-server localhost:9092
This gives you a simple environment for learning partitions and offsets.
It isn't a highly available Kafka cluster.
4. Produce and Consume Events
Write a message:
echo '{"orderId":101,"status":"created"}' | \
docker exec -i kafka-local \
/opt/kafka/bin/kafka-console-producer.sh \
--topic orders.events \
--bootstrap-server localhost:9092
Read messages:
docker exec kafka-local \
/opt/kafka/bin/kafka-console-consumer.sh \
--topic orders.events \
--from-beginning \
--timeout-ms 10000 \
--bootstrap-server localhost:9092
You now have a local Kafka workflow using a producer, topic, and consumer.
5. Production Kafka Is a Different Problem
Running one broker on a laptop is easy compared with managing a replicated cluster.
In production, pay attention to:
- Replication factor and minimum in-sync replicas
- Broker disk capacity and I/O
- Consumer group lag
- Partition leadership
- Authentication and authorization
- KRaft controller quorum health
- Recovery and upgrade procedures
For Kubernetes, operators such as Strimzi can help manage cluster lifecycle, but they don't remove the need for careful storage and capacity planning.
6. Investigating Consumer Lag
Consumer lag tells you how far a consumer group is behind the latest available partition offsets.
Use:
docker exec kafka-local \
/opt/kafka/bin/kafka-consumer-groups.sh \
--bootstrap-server localhost:9092 \
--describe \
--group inventory-service
The group must exist and have committed offsets for meaningful output.
If lag is increasing, check whether event production exceeds processing capacity.
Also investigate slow downstream dependencies, consumer exceptions, frequent rebalances, and partition counts.
Avoid resetting offsets without understanding the risk of skipping or replaying records.
Read the Complete Apache Kafka Guide
The original article goes deeper into Kafka storage internals, KRaft, Docker Compose, Kafka vs RabbitMQ, Spring Boot, Strimzi, and production runbooks.
Read Apache Kafka Architecture & Hands-On Production Guide
Final Thoughts
Kafka becomes much easier to operate once you understand how partitions, offsets, replication, and consumer groups work together.
The challenge isn't just sending messages. It's making the streaming platform reliable when workloads grow and failures happen.
What's the most difficult Kafka problem you've investigated in production?
Written by Naveed Ahmed — DevOps & Platform Engineering
Top comments (0)