The year is 2018. I'm sitting in a client's office in Gurugram, and their CTO just told me their system can't handle the incoming data flow. Their words? "We're drowning." Every database connection is maxed out, the message queue is backing up, and the monitoring dashboard looks like a heart monitor gone flat. I asked if they'd considered Kafka. Blank stare.
That conversation has repeated itself dozens of times since. If you're building anything that moves data at scale — and let's face it, that's most of us — you need to understand what Kafka is, what it does, and why it's become the ugly backbone of the modern internet.
Before Kafka, the standard approach to moving data between systems was point-to-point integration. Service A calls Service B, which calls Service C, and so on. This works fine when you have three services.
When you have thirty? It's a nightmare.
Every integration adds latency, becomes a single point of failure, and creates a web of dependencies that's impossible to untangle. Instead of integrating each system with every other system, you connect everything to Kafka and let it handle the data movement as a high-speed, fault-tolerant intermediary.
Think of it as a central nervous system for your data. Instead of every organ talking directly to every other organ, everything routes through the spine.
Apache Kafka is a distributed event streaming platform. That's the official definition. Here's what that means in practice.
Kafka does four things:
- Publishes streams of events (records) from producers
- Subscribes to those streams on the consumer side
- Stores the events durably in a distributed, fault-tolerant way
- Processes those streams in real-time
It's a system that publishes, subscribes, stores, and processes streams of events end-to-end — all with a single solution. The Apache Software Foundation describes it as an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, and mission-critical applications.
Developed originally by LinkedIn and later donated to the Apache Software Foundation, it follows the publish-subscribe model, where producers send messages to topics and consumers read from them. The project is written in Java and Scala, but you don't need to touch either to use it effectively.
Kafka is built for scale and speed. We're talking hundreds of thousands of events per second on modest hardware. Low latency. High throughput. That combination is rare, and it's why Kafka has become the standard for data infrastructure.
Before we go deeper, I need to address the elephant in the room. Yes, there are two Kafkas. No, they are not related.
Franz Kafka was a German-language Jewish Czech writer born in Prague in 1883. He's widely regarded as a major figure of 20th-century literature, famous for works like The Metamorphosis and The Trial. His stories feature isolated protagonists facing bizarre, surreal predicaments and incomprehensible bureaucratic systems.
His work expressed the anxieties and alienation felt by many in 20th-century Europe. If you've ever asked "what is kafka's ideology?", the answer is complex — he wrote about existential dread, alienation, and the absurdity of modern institutions.
Here's the irony I genuinely appreciate: the distributed system named after him often feels like it's governed by the same incomprehensible bureaucracy his stories satirize. If you've ever spent three hours debugging a consumer group rebalance, you know exactly what I mean. The tragedy of Kafka the writer is that he died young, alone, and largely unpublished. And the tragedy of Apache Kafka is that it solves problems you didn't know you had until you're rebuilding your entire architecture around it.
The naming isn't a joke. It's a warning.
If I'm going to explain how to use Kafka practically, you need to understand the vocabulary. This isn't optional — it's the foundation everything else builds on.
A topic is a named logical channel where events are published. Think of it like a file folder with an infinite stream of records inside. You create a topic called "orders," and every order event from every service goes into that topic.
Topics are partitioned. Partitions are where the actual data lives across the broker cluster. More partitions mean more parallel processing — but more partitions also mean more overhead. There's no perfect answer for partitioning strategy; it's a trade-off you make based on your use case.
Producers publish events to topics. They're the sources. A producer can be any system that generates data — an API gateway, a database change feeder, an IoT sensor, whatever.
Consumers read events from topics. Multiple consumers can read from the same topic simultaneously, and Kafka tracks each consumer's position (offset) so they don't reprocess old data unless you explicitly tell them to.
A broker is a single Kafka server. A cluster is a collection of brokers. Data is replicated across brokers for fault tolerance. If one broker dies, another takes over without data loss.
Here's a simple producer example in Python using the confluent-kafka library:
python
from confluent_kafka import Producer
import json
conf = {
'bootstrap.servers': 'localhost:9092',
'client.id': 'order-service'
}
producer = Producer(conf)
order = {
'order_id': 12345,
'user_id': 67890,
'total': 299.99,
'items': ['widget', 'gadget']
}
producer.produce(
topic='orders',
key=str(order['order_id']),
value=json.dumps(order)
)
producer.flush()
print(f"Published order {order['order_id']} to 'orders' topic")
Nothing fancy. But look at how simple it is to get data into the system. Now here's the consumer side:
python
from confluent_kafka import Consumer
import json
conf = {
'bootstrap.servers': 'localhost:9092',
'group.id': 'order-processor',
'auto.offset.reset': 'earliest'
}
consumer = Consumer(conf)
consumer.subscribe(['orders'])
try:
while True:
msg = consumer.poll(timeout=1.0)
if msg is None:
continue
if msg.error():
print(f"Consumer error: {msg.error()}")
continue
order = json.loads(msg.value())
print(f"Processing order {order['order_id']} for user {order['user_id']}")
except KeyboardInterrupt:
pass
finally:
consumer.close()
That's the whole loop. Publish, subscribe, process. The real complexity comes when you're scaling, not when you're connecting — which is exactly why Kafka is so powerful.
Here's what the marketing doesn't tell you: Kafka wasn't built for "streaming analytics" as a buzzword. It was built to solve a concrete, painful problem at LinkedIn.
At the time, LinkedIn was dealing with massive amounts of data — user activity logs, profile views, search queries. They had dozens of point-to-point integrations that were breaking constantly. The goal was to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.
The problem wasn't generating data. It was moving it.
Kafka solved that by decoupling producers from consumers. Producers don't wait for consumers to acknowledge data. Consumers don't block producers. Data is written to disk durably, and consumers can replay it at their own pace.
This is why thousands of companies — including companies you've actually heard of — use Kafka for their data pipelines.
Let me be honest about the trade-offs. Kafka isn't magic, and there are real pain points you'll hit.
When a consumer joins or leaves a group, Kafka triggers a rebalance. During a rebalance, all consumers in that group stop processing. If you have a large group and frequent rebalances, you'll see your throughput drop to zero repeatedly.
I've seen this kill production systems. A service deployed with a rolling restart could trigger rebalances that took minutes to complete, causing massive backlogs.
The fix: Configure session.timeout.ms carefully, monitor rebalance rates, and never restart all consumers in a group at once.
Kafka guarantees order only within a partition. If you need global ordering of events, you're out of luck — unless you use a single partition, which kills your parallelism.
The fix: Design your keys so that related events land in the same partition. Partitioner logic isn't optional; it's architecture.
Kafka stores events on disk. Even if consumers have read everything, data stays until the retention policy kicks in. At high throughput, that means serious storage requirements.
The fix: Plan your retention policy upfront. Tiered storage helps in newer versions, but you can't ignore disk entirely.
Kafka isn't just a message queue — it's a platform with two major extensions:
Kafka Connect handles integration with external systems for data import/export. Instead of writing custom connectors for every database or API, you use pre-built connectors for common systems.
Kafka Streams is a Java library for stream processing. It lets you write real-time processing logic — filtering, aggregation, joins — directly against Kafka topics.
Here's a Kafka Streams example for real-time aggregation:
java
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.kstream.KStream;
import org.apache.kafka.streams.kstream.KTable;
import org.apache.kafka.streams.kstream.Materialized;
public class OrderAggregator {
public static void main(String[] args) {
StreamsBuilder builder = new StreamsBuilder();
KStream orders = builder.stream("orders");
KTable orderCounts = orders
.groupBy((key, value) -> extractUser(value))
.count(Materialized.as("order-counts"));
orderCounts.toStream().to("user-order-counts");
KafkaStreams streams = new KafkaStreams(builder.build(), getConfig());
streams.start();
}
}
Here's the contrarian take: I've seen teams reach for Kafka Streams when they should've used a simple consumer with Redis. Kafka Streams is powerful, but it's not simple. You need Java expertise, which limits your team to JVM developers.
Start with a basic producer-consumer first and grow into the advanced features when you actually need them.
Let me give you a realistic path to production — the way I've actually done this, not how the docs suggest.
Install Kafka and start with a single broker. Don't worry about clusters yet.
bash
wget https://dlcdn.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
bin/kafka-topics.sh --create --topic test-events \
--bootstrap-server localhost:9092 \
--partitions 3 --replication-factor 1
Use the console tools to verify everything's working:
bash
bin/kafka-console-producer.sh --topic test-events --bootstrap-server localhost:9092
bin/kafka-console-consumer.sh --topic test-events --from-beginning --bootstrap-server localhost:9092
Don't deploy Kafka to production until you understand how your data flows, what your throughput requirements are, and what your failure scenarios look like.
You can't operate Kafka without monitoring. Track:
- Consumer lag (how far behind consumers are)
- Broker CPU and disk I/O
- Partition distribution across brokers
- Rebalance frequency
If you're using a managed service (Confluent Cloud, AWS MSK, etc.), a lot of this is handled for you. But you still need to understand it to design your system properly.
Here's the part people don't like to hear: Kafka isn't always the answer.
If you need to process a few thousand messages a day, RabbitMQ or Redis Streams will serve you better. Kafka has operational overhead — you need a cluster, monitoring, and engineering time to maintain it. If your data doesn't need replayability, durability, or massive throughput, you're paying for features you're not using.
I've advised startups to stay away from Kafka. I've also advised enterprises to migrate everything to Kafka. The variable isn't size — it's the data flow patterns you actually have.
Apache Kafka is an open-source distributed event streaming platform. It's used to build real-time data pipelines, stream processing applications, and data integration layers. Companies use it to move large volumes of data between systems reliably and at low latency.
Apache Kafka is famous for being the de facto standard for event streaming — handling high-throughput, fault-tolerant data movement. Franz Kafka, the writer, is famous for The Trial, The Metamorphosis, and other works about alienation and absurd bureaucracy.
If you mean Franz Kafka's ideology, his works critique institutional power, alienation, and the incomprehensibility of modern systems. There's a reason the distributed system shares his name.
Franz Kafka wasn't literally alone when he died in 1924 — his friend Robert Klopstock was at his bedside. But he was isolated in a deeper sense: largely unpublished, unmarried, and tormented by illness. He asked his friend Max Brod to burn his unpublished manuscripts, which fortunately didn't happen.
The tragedy of Franz Kafka is that he died young (40), affected by tuberculosis, his work largely unrecognized in his lifetime. The tragedy of Apache Kafka is what happens when your consumer lag grows unbounded during peak traffic and nobody noticed the monitoring dashboard was down.
Kafka is a tool. A powerful one, but a tool nonetheless. It won't fix bad architecture, and it won't solve problems that aren't about data movement.
What it will do is give you a reliable backbone for moving data across your systems, at scale, with acceptable latency. It's the closest thing data infrastructure has to a universal connector — a central nervous system that can handle hundreds of thousands of events per second without breaking a sweat.
I've built systems on Kafka. I've watched them process 200K events per second during peak load and stay stable. I've also watched them struggle when the cluster was misconfigured and the monitoring was ignored.
Treat Kafka with respect. Understand its strengths — distributed, scalable, fault-tolerant event streaming. Understand its weaknesses — operational complexity, the need for real expertise, the danger of misconfiguration.
And remember who it's named after. Because if you don't respect the system, it will absolutely make you feel like you've been processed by an incomprehensible bureaucracy.
Sources:
- Franz Kafka - Wikipedia
- What is Apache Kafka and How Does it Work? - GeeksforGeeks
- Apache Kafka - Wikipedia
- Franz Kafka | Biography, Books, The Metamorphosis, The Trial, & Facts - Britannica
- GitHub - apache/kafka
- What is Apache Kafka? - GeeksforGeeks
- Apache Kafka - Official Site
- Introduction - Apache Kafka
- What is Kafka? Topics, Producers, Consumers, Brokers Explained - Confluent
- Apache Kafka Tutorial - TutorialsPoint
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.
Originally published at https://sivaro.in/articles/kafka-the-data-backbone-you-cant-ignore/.
Top comments (0)