Kafka is not a new name. It has been around for years. Every architecture blog, every system design video, every discussion in colleague and developer circle, Kafka comes up. I heard about it many times. But never really stopped to learn it. I never went beyond the surface. Not even once.
I won’t lie. I always thought Kafka was just another message queue. A fancier RabbitMQ. Something I will learn “someday.” That someday never came.
Then I joined a System Design Cohort led by Hitesh Choudhary and Piyush Garg on ChaiCode. One evening, Piyush sir started explaining Kafka.
I was listening. But my mind was doing something else.
Steve Jobs once said: “You can’t connect the dots looking forward; you can only connect them looking backward.”
That evening, my mind was connecting dots backward to the Indian railway station.
We all have been to railway station at least once in life and if you have, you know that feeling. It is not calm. It is not organised. Somehow it just work. Nobody is managing you personally. No one is telling each passenger where to go. Still, people loves to travel in trains. They reach where they have to go. The whole madness somehow just keep running every single day.
I was surprised that Kafka was solving the similar problem in software world.
Piyush sir was explaining Kafka in class and I was seeing every single thing play out at that station in my mind. For the first time, Kafka start making sense to me.
Why Kafka Exists: The Problem First
Before I tell you what Kafka is, let me tell you why it even exist.
Everyone are using IRCTC for train ticket booking. You definitely had an experience of tatkal booking chaos. Everyone wants to book ticket at same time. You finally get that booking confirmation. But did you ever wonder what all happen in that one second? Money got deducted from your account. Seats got booked for you. A message land on your phone. Your ticket was ready somewhere in their system. PNR got updated. All of this happen in few seconds. Not for you alone. For lakhs of people at same time.
Now what if any one of these steps fail or slow down or miss the request? This kind of simultaneous load is something I wrote about earlier — it is called the Thundering Herd Problem.
That is the exact problem Kafka was built to solve.
This is the problem. Modern applications are not simple anymore. They are not one service talking to one other service. They are dozens of services, each generating events, each needing to talk to each other, all at once.
And here is where most people make a mistake. They think Kafka is just a message queue. I thought the same.
It is not.
A message queue is like a postman. One sender, one receiver, message delivered and gone. Simple.
Kafka is a message broker. Not a queue. It sit between all your services like a central hub. Many things pushing data into it at same time. Many others pulling from it. Kafka store every message. Same message, multiple services can read it. Nothing get lost in between.
That difference, trust me, it matter a lot.
Kafka is the Railway Station
Close your eyes for a second.
You are standing at a busy railway station. Not a small one. A big one. Hundreds of passengers arriving every minute. Some coming from Jaipur. Some going to Mumbai. Some waiting for a local train. Everyone with a different destination.
We all have stood at a railway station at some point. That chaos is not new to us. You are looking for your platform, someone is running with luggage, someone is asking for directions, tea vendor is shouting from corner. Nobody is in charge of you personally. Still you reach. Still everyone reach.
Now think about this. That station never stop working. Too many passengers? It handle them. Someone miss their train? Another one is coming. No matter how crowded it get, station just keep doing its job.
That is exactly what Kafka does for your application.
Kafka is that station. Every message in your system is like that passenger who need to reach somewhere. Station manage thousands of people without losing single one. Kafka do exactly that for your software.
Everything connect to Kafka. Every message pass through it and from there, each message reach exactly where it need to go.
Producer: Who Sends the Message
We all have reached a railway station at some point and we all had someone who drop us there. Could be father, friend, or just an auto driver. Once you are inside the station, their job is done. They don’t come with you. They don’t worry about where you will go next.
In Kafka, that person is called a Producer. Any service that generate and send message to Kafka is a Producer. Your payment service send a payment event. Your app send a booking request. It does not matter who send it. Once message reach Kafka, producer job is finish.
Producer don’t care what happens next. Who reads the message. When they read it. That is not producer’s problem. Producer completed his job the moment message reach Kafka.
Topics: The Platforms
You are inside the station now. You are that message, remember? First thing you look for is your platform. Not randomly. You check your ticket. Platform 4. That is where your train is. Every passenger go to their specific platform. Nobody mix up.
In Kafka, these platforms are called Topics. Every message go to a specific topic. Order messages go to orders topic. Payment messages go to payments topic. Nobody mix up there either.
A Topic is a category. Think of it as a named channel. Order related messages always find their own place. Payment messages find theirs. Notifications find theirs. Nobody land in wrong place.
Think of it this way: producer does not just throw message into Kafka randomly. It send message to a specific topic. Just like passenger go to specific platform.
Order related messages will always find the “orders” platform. Payment messages will always find “payments”. No mixing. No confusion.
Partitions: The Bogies
Now here is where it get really interesting.
You found your platform. Train is right there. But when you step closer, you notice something. Train is not just one big space. It is divided into bogies. S1, S2, S3, A1, B2. Each bogie carries different passengers. Together, they all go to same destination.
In Kafka, these bogies are called Partitions.
One Topic can have multiple Partitions. They belongs to the same category, but work is split into smaller lanes. This is where Kafka actually get its real power from.
But here is the important thing Piyush sir explained that day. This is what change everything for me.
Why we need these partitions? Why not just one big bogie?
We all know Indian train classes. General, Sleeper, Third AC, Second AC and First AC. Each class has its own coach. General passengers never sit in First AC coach. First AC passengers never end up in Sleeper. Everyone belong to their own coach. That is just how it work.
Kafka follow same rule. Related messages always go to same partition. Same order ID will always goto same partition. The sequence stay intact. Every time.
Because there are multiple partitions, multiple consumers can work in parallel. More partitions, more speed. That is how Kafka scale without breaking the order.
Consumers: Who Receives the Message
Every passenger on that train is going somewhere. Someone is waiting at the destination. Could be a family member standing at exit gate. Could be a friend who came to pick up. Could be a colleague. Their job is simple. Receive and take action.
In Kafka, that person is called a Consumer.
A Consumer is any service that read messages from Kafka. Your notification service consume the order event and send push notification. Your analytics service consume the same event and update dashboard. Your inventory service consume it and reduce stock count.
Consumer sit and wait. When message arrive in their topic, they read it. They process it. They take action.
Producer send. Kafka hold. Consumer receive. That is the basic flow.
Consumer Groups: Sharing the Work
We all have travelled in a train with a big group at some point. Could be a family trip, college friends, or office team. Everyone board same train. But when train reach destination, everyone scatter. Some head to hotel. Some go to relatives place. Some go directly to the event. Nobody wait for each other. Each group know where they have to go.
In Kafka, this is called a Consumer Group.
A Consumer Group is a set of consumers that work together to read from a topic and here is what make it powerful: each partition is read by only one consumer within a group. Work is divide evenly. No two consumers in same group read the same message.
So if your “orders” topic has 4 partitions, and you have 4 consumers in a group, each consumer handle one partition. Parallel processing. Fast. Efficient.
But now here is something even more interesting.
You can have multiple Consumer Groups reading the same topic completely independently.
Your notification service is reading from orders topic. Your analytics service is also reading from same orders topic. Both get every message. One send notification. Other update the dashboard. Neither know about each other. Neither disturb each other.
Same train. Different families. Each going where they need to go.
Why Kafka is Fast and Scalable
One last thing before we wrap this up.
How does a big railway station handle lakhs of passengers every single day without shutting down?
Simple. It don’t rely on one platform. One train. One bogie. It has many platforms running in parallel. Many trains on each platform. Many bogies on each train. Work is spread out.
Kafka work the same way.
More topics for more categories. More partitions for more parallel processing. More consumers in a group to share the load. You need to handle more messages? Add more partitions. Add more consumers. The system scale horizontally.
No single point doing all the work. No bottleneck. No crash.
Any application that survive millions of events per second, trust me, has something like Kafka running quietly in the background doing this job.
Conclusion: You Are the Message
So that evening in cohort class, dots finally connected for me. Backward. Just like Steve Jobs said.
Kafka is not just another message queue , it is a message broker. A central station for your entire application. Scale does not break. Order does not get lost. Multiple teams can read same data. Nobody disturb anyone else.
Everything we went through today, Producers , Topics , Partitions , Consumers , Consumer Groups , that whole journey, it was all happening at that one busy station.
So go back to that station one last time.
This time do not stand outside watching. Step in.
You are not standing outside anymore. You are inside. You are that passenger. You are the message.
Someone dropped you at the station. That was your Producer. You walked to your platform. That is your Topic. You boarded your coach. That is your Partition. You found your seat. That is your offset , your exact position, your order guaranteed.
The train moves. It reaches the destination. Someone is waiting for you there.
That is Kafka.
Next time someone says “we use Kafka for event streaming”, you will smile. Because this time, you know exactly what is happening inside that station.
Continue the Journey with more articles
Still on the platform and enjoyed this ride? Here are more system design trains to catch:
- The Thundering Herd Problem: What happens when 25,000 students hit my system at the same time.
- Message Queue in System Design: How my server crashed in 60 seconds and what I learned from it.
- Cache Strategies in Distributed Systems: The day our cache expired and 25,000 students lost their exam.
Looking for more? All articles are available on my Medium profile and many are coming soon. Follow or subscribe to get notified when the next one drops.
Top comments (0)