I remember the exact moment Kafka stopped being a mystery. We were at SIVARO in 2021, debugging why our customer event pipeline kept falling over. The old architecture had six microservices talking to each other directly through REST calls. The database was the bottleneck. The frontend was timing out. And I had an engineer on my team — brilliant kid — who kept muttering, "If we just used Kafka for this..."
I'd read about Kafka. I'd even used it in a side project once. But I didn't get it. Not the way you need to get a tool to bet your production infrastructure on it.
Then I spent a weekend reading the Apache Kafka documentation, realized I'd been treating it like a message queue when it's actually a distributed commit log — and everything clicked. We rewired our entire data flow around Kafka Connect within two months. Our event throughput went from failing at 2,000 events per second to handling 200,000 without breaking a sweat. That's not a flex — that's the difference between designing for the problem you have versus the problem you're about to have.
Here's what I'll cover in this piece: what Kafka actually is (not what the marketing says), how it works at a mechanics level you can actually use, how to wire it into your stack without institutionalizing chaos, and the hard truths nobody tells you — like the fact that Kafka is not a database, and treating it like one will eventually hurt you.
Let's go.
Let's start with a definition, because most people get this wrong.
Apache Kafka is an open-source, distributed event streaming platform. It's written in Java and Scala, originally built by LinkedIn in 2011, and then donated to the Apache Software Foundation. Its purpose, according to GeeksforGeeks, is to handle large-scale real-time data streams efficiently and reliably.
But that definition is sterile. Let me give you a better one.
Kafka is a distributed commit log.
Think of it like a journal. Applications (producers) write events — a user signed up, a payment was processed, a sensor reading changed — and those events get appended to the log in order. Other applications (consumers) read from that log, at their own pace, tracking their position with something called an offset. Because the log is distributed across multiple brokers (servers), it's fault-tolerant. If one broker dies, the others keep serving.
It's deceptively simple. But that simplicity is what makes it so powerful.
According to the Apache Kafka introduction, the platform provides three main capabilities:
- Publish and subscribe to streams of events
- Store streams of events durably — Kafka retains data on disk for a configurable period (days, weeks, even years)
- Process streams of events in real-time or retrospectively
What this means in practice: instead of building point-to-point integrations between every system, you connect everything to Kafka. It becomes the central nervous system of your architecture. The AWS explainer on Kafka frames it well — a streaming platform needs to handle constant influx of data and process it sequentially and incrementally. Kafka does exactly that.
Here's where the confusion starts.
When people say "Kafka," they're usually talking about one of three things:
Franz Kafka — the actual writer. The one from Wikipedia who was born in Prague in 1883 and wrote about bureaucratic nightmares and absurd existential predicaments. The Britannica biography describes his work as expressing "the anxieties and the alienation felt by many in 20th-century Europe."
Apache Kafka — the distributed event streaming platform we're talking about here, which was named after the writer because Jay Kreps (the creator) thought the platform, with its process of "poetic" transformation through a pipeline, was a bit Kafkaesque.
Kafkaesque situations — which is what your infrastructure feels like when you've built a haphazard event bus without proper schema management. Which is a real thing.
I bring this up because when you search "what is kafka?" you get both literary and technical results, and it's genuinely confusing for beginners. The writer and the platform share a name, but they couldn't be more different in spirit. Franz Kafka's characters are isolated, trapped in incomprehensible bureaucratic systems — The Trial and The Metamorphosis are prime examples of protagonists facing surreal predicaments they can't escape.
Apache Kafka, the platform, is built to prevent those situations. It's designed to give you clarity, not confusion. Though anyone who's debugged a consumer group rebalance issue at 2 AM might disagree.
Most people think Kafka is just a faster version of RabbitMQ or Amazon SQS. That's wrong.
Message queues are designed for point-to-point communication where messages are consumed and then deleted. Kafka is designed for event retention and replay. The difference is fundamental:
- Queue: Produce a message → consumer picks it up → message is gone.
- Kafka: Produce an event → stored in the log → multiple consumers can read it independently → it sits there until the retention period expires → anyone can replay it.
According to GeeksforGeeks' explanation of Apache Kafka, Kafka is "built to transport large volumes of data in real-time between systems, without needing to develop hundreds of intricate integrations." The key phrase there is without needing to develop hundreds of integrations. Kafka replaces the spaghetti diagram of system-to-system connections with a single hub.
This isn't just about throughput. It's about decoupling. When you publish an event to Kafka, you don't care who consumes it. Your order service doesn't know whether the notification service, the analytics service, or a new data science model is going to read that event. That's powerful. That's how you build systems that scale without coordinating every deployment across the org.
Let's get practical. You can't use Kafka without understanding these pieces.
A topic is a named channel for events. Think of it as a category or a stream name — user.signups, payment.processed, page.views.
Within a topic, events are split across partitions. Partitions are what give Kafka its scalability. Each partition is an ordered, immutable sequence of events. Events within a partition are guaranteed to be in order. Events across partitions are not — that's why you need to think carefully about your partitioning strategy.
Here's the rule of thumb I use: if events need to be processed in order for a given entity (like a user), partition by that entity's ID. If order doesn't matter globally, partition by a hash of the event key.
Producers write events to topics. The simplest producer looks like this in Java:
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
Producer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("user.signups", "user-123", "{\"email\":\"nishaant@sivaro.com\"}"));
Consumers read events from topics. They maintain their offset — the position in the partition where they left off. If a consumer crashes, it can restart from its last committed offset and pick up where it stopped.
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("group.id", "email-service");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("user.signups"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
for (ConsumerRecord<String, String> record : records) {
// process the event
sendWelcomeEmail(record.value());
}
consumer.commitSync();
}
This is where Kafka gets its horizontal scaling power.
Multiple consumers with the same group.id share the partition load. Each partition is assigned to exactly one consumer within the group. This means you can add more consumers to a group to scale read throughput, and Kafka handles the rebalance of partitions automatically.
The caveat: a topic with 3 partitions can max out at 3 active consumers in a group. If you add a 4th, it sits idle. That's not a bug — it's fundamental to how Kafka guarantees ordering.
A Kafka cluster is made of brokers — the servers that store the log and serve producers and consumers. Each topic's partitions are distributed across brokers with a configurable replication factor.
At a replication factor of 3 (which is what I recommend for production), each partition exists on 3 brokers. One is the leader — it handles all reads and writes. The others are followers that sync in the background. If the leader fails, a follower is promoted. This is how Kafka achieves fault tolerance without losing data.
You don't need a cluster to start. You just need Kafka running on your laptop.
Now, here's the thing: Kafka has a reputation for being painful to run locally. That reputation is deserved. The default approach involves downloading the tarball, starting Zookeeper (if you're on an older version), starting Kafka, and hoping the logs don't tell you secrets you're not ready to hear.
Since Kafka 2.8, KRaft mode (Kafka Raft metadata mode) eliminates the Zookeeper dependency — it's simpler now. If you're using a recent version (Kafka 3.x+), this works:
wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
tar -xzf kafka_2.13-3.7.0.tgz
cd kafka_2.13-3.7.0
bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/kraft/server.properties
bin/kafka-server-start.sh config/kraft/server.properties
Then, in another terminal:
bin/kafka-topics.sh --create --topic test-topic --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092
bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092
That's a running Kafka instance. You're now in the club. And I apologize in advance for the JVM memory usage — on a small dev machine, Kafka will hog about a gig of RAM. It's a lifestyle choice.
Kafka alone is just a pipe. The real value comes from its ecosystem.
Kafka Connect is the integration layer. According to the Apache Kafka Wikipedia page, it provides "the Kafka Connect component for data integration with external systems."
Think of Connect as the adapter layer that lets you pipe data from sources (databases, files, SaaS tools) into Kafka, and from Kafka to sinks (data warehouses, search engines, object stores). We've used the Debezium connector to capture change data from PostgreSQL — every insert, update, and delete gets turned into an event in Kafka. This is called Change Data Capture (CDC), and it's the most underrated pattern in modern data architecture.
A basic connector config looks like this:
{
"name": "postgres-orders",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "localhost",
"database.port": "5432",
"database.user": "postgres",
"database.password": "password",
"database.dbname": "orders",
"database.server.name": "orders-db",
"table.include.list": "public.orders",
"plugin.name": "pgoutput"
}
}
Once you wire up CDC, you're never polling your database again. That's a freedom I can't describe. Every change emits an event. Your downstream systems react immediately. The latency is near-zero.
Kafka Streams is a client library for building stream processing applications using Kafka as the source and sink. It's not another compute cluster — it runs inside your application. You write standard Java code, and the library handles stateful computations, windowing, joins, and exactly-once semantics.
Here's a real example. We process payment events and need to flag suspicious transactions — more than 5 in 60 seconds from the same payment method:
KStream<String, Payment> payments = builder.stream("payments", Consumed.with(Serdes.String(), paymentSerde));
payments
.groupBy((key, payment) -> payment.getMethodId(), Grouped.with(Serdes.String(), paymentSerde))
.windowedBy(TimeWindows.of(Duration.ofSeconds(60)))
.count()
.filter((windowedKey, count) -> count > 5)
.toStream()
.mapValues((windowedKey, count) -> new FraudAlert(windowedKey.key(), count))
.to("fraud-alerts", Produced.with(Serdes.String(), fraudAlertSerde));
That's the entire detection logic. No Spark cluster. No Flink job. No hourly batch reconciliation. It's just... code that runs.
I've spent a lot of time praising Kafka. Here's the other side.
You can't query Kafka like a database. It's a log, not a table. If you want to look up an event by key, you need to maintain a materialized view or use ksqlDB (Kafka's streaming SQL engine). If you want random access to historical data, you're going to be disappointed.
Kafka is a distributed system. That means it breaks in distributed-system ways. You'll encounter rebalancing storms, stuck offsets, broker failures, and disk full errors at the worst possible moments. The GitHub repository for Apache Kafka is a testament to how much effort goes into maintaining this thing — and it's still not trivial to run well in production.
Without proper schema management, your Kafka topics become a dump of JSON documents that mean different things to every team. Nobody documents anything. Event formats drift. Your consumers break in mysterious ways.
Use a schema registry with Avro or Protobuf. It's non-negotiable. We learned this the hard way when one production service started writing "userId" while the rest were writing "user_id" — one character difference, hours of downtime.
It's worth pausing on the name for a second. You may be asking, "what is the tragedy of kafka?" — since Franz Kafka's life was famously difficult. According to the Britannica biography, his work "expresses the anxieties and the alienation felt by many in 20th-century Europe." He died young, at 40, in 1924, largely unrecognized.
The Kafkaesque tragedy in the tech world is different. It's when you set up Kafka, tell your team you're doing "event-driven architecture," and then all hell breaks loose because you approached it as a messaging tool instead of a log. It's when you keep data in Kafka for 7 days, downstream systems fall behind, and consumers try to replay data that's gone. It's when your microservices become a distributed nightmare orchestrated through topics nobody owns.
"Was kafka alone when he died?" — yes, and that's sad, and it's not an analogy for anything. But the platform's tragedy is that it works beautifully until you misuse it, and then it fails in ways that are hard to debug because the failure is in your design, not the software.
Use it when:
- You need to decouple producers from consumers
- You need multiple applications to consume the same events independently
- You need replay capability — reading historical events for backfills or debugging
- You need high throughput with low latency (though "low" here means milliseconds, not microseconds)
Don't use it when:
- You need simple request-reply semantics — that's what HTTP is for
- You need very low latency (sub-10ms) — look at NATS or Redis Pub/Sub for that
- You're building a small project with one or two services — you're adding operational overhead for no reason
- You need database-grade querying — use a database
There's no shame in admitting Kafka is overkill for your use case. I've walked out of planning meetings where people wanted to put Kafka in front of everything for a 500-user internal tool. Don't do it.
Let me end with a concrete example from our infrastructure, because theory is cheap and production experience is not.
At SIVARO, we run a data pipeline that ingests events from multiple client services. The architecture looks like this:
- Applications publish domain events to Kafka (user actions, system metrics, payment notifications).
- Kafka Connect streams these events to a data lake (S3) for historical storage.
- Kafka Streams processes real-time events for alerting and enrichment.
- Separate consumer groups power our analytics dashboard and the client-facing admin panel.
The beauty: when we onboard a new client service, we don't rebuild integrations. We just configure a producer and define the topic. The consumer groups subscribe, and everything flows.
Did it take time to get right? Absolutely. Rebalancing strategies needed tuning. Our topic partitioning needed careful thought. We invested in observability early — the Apache Kafka project gives you a lot of metrics, but you have to expose them.
And when we did get it right, we stopped worrying about scaling. Doubling data volume? Add a couple of brokers. New consumer? Write a config. It's infrastructure that absorbs growth instead of pushing back.
Apache Kafka is an open-source distributed event streaming platform used to build real-time data pipelines and streaming applications. It's used because it provides a unified, high-throughput, low-latency way to move data between systems. Rather than integrating each system with each other, you integrate everything with Kafka. According to GeeksforGeeks, it's "built to transport large volumes of data in real-time between systems."
Two things. Franz Kafka, the writer, is famous for works like The Trial and The Metamorphosis — stories that capture bureaucracy, alienation, and the absurdity of modern life, as documented by Britannica. Apache Kafka, the platform, is famous for being the de facto standard for event streaming at scale, used by more than 70% of Fortune 500 companies for real-time data infrastructure.
For the writer, Kafka's ideology was a pessimistic exploration of how individuals are crushed by opaque, unfeeling bureaucratic systems. For the platform, there's no ideology — it's a pragmatic tool for decoupling systems. Though I'll say this: if you use Kafka well, it's the closest thing to eliminating Kafkaesque chaos in your organization's data flow.
Yes and no. The core concepts — topics, partitions, consumers — are simple. What's hard is operational experience: knowing how to configure retention, manage rebalancing, handle exactly-once semantics, and design your partitioning for real-world constraints. That comes with time and mistakes.
RabbitMQ is a message broker optimized for routing, with complex exchange types and per-message acknowledgment. Kafka is a log-based event streaming platform optimized for throughput and replay. Kafka doesn't delete messages when they're consumed — they persist for a configurable retention period, which is what makes replay possible.
For the writer, it's the tragedy of an unrecognized genius who burned his own manuscripts. For the platform, it's the tragedy of a powerful tool used poorly. Most Kafka outages aren't Kafka's fault — they're design errors, misconfigurations, or the result of ignoring the ecosystem around it.
Kafka changed how I build systems. Before it, every integration was a bespoke mess of webhooks, polling jobs, and point-to-point connections that broke in production. After it, the architecture became boring — which is exactly the point.
It's not a silver bullet. Treat it like one and you'll learn why people reference Kafkaesque bureaucracy. But understand it as a distributed commit log — a durable, replayable, partitionable record of everything that happened in your business — and you'll never build data infrastructure the same way again.
Start small. Run it locally. Wire up one consumer group. Watch it work. Then expand.
You'll thank yourself later.
Nishaant Dixit — Founder of SIVARO. Building data infrastructure and production AI systems since 2018. Built systems processing 200K events/sec.
Originally published at https://sivaro.in/articles/kafka-the-event-streaming-backbone-you-cant-ignore/.
Top comments (0)