Recently I had to mess with Kafka in my work, it's something I've never done and I had some difficulty understanding, so I'm thinking of bringing a series of posts about it.
In this first post I want to give a little introduction about what Kafka is and how it works
What is Kafka?
Apache Kafka is an open-source, high-performance, real-time messaging platform. It stands out for being extremely fast and scalable, it is widely used with microservices.
After this more technical description let's go to what it actually does, in today's world we have systems communicating with each other all the time, but this may not always be the best option.
Let's imagine that I have a System A and a System B, if system A sends a message and system B is down, this message will be lost, so that doesn't happen, we put an intermediary for this communication that stores and sends these messages when system B comes back.
Important terms
When we talk about Kafka we have some terms that we use, and without understanding them it would be difficult to understand how it works, so here comes a list explaining these terms:
- Broker: in computing, a broker is the server for sending messages, it is the one who triggers and receives the messages and it is he who owns the topics and partitions
- Topics: a topic is an identifier, Through this identifier it is possible to connect with the broker and send or receive messages. (For me this guy is basically like the “database table” is where our data is)
- Microservices: Microservices are a software architecture model that aims to separate a system into several smaller systems to improve performance and availability.
- Partition: Subdivision of a topic, partition is a resource for load balancing
- Cluster: In short, a cluster is a set of brokers and it contains the servers and the main instance of Kafka.
How does Kafka work ?
Kafka was written in Java and Scala, it performs data transmission asynchronously. In its structure, Kafka has consumers and producers.
Producers are those who send and distribute the data, while consumers are those who subscribe to a topic and listen to the messages that will arrive.
The Kafka Architecture described above allows for consistency in sending messages. Brokers are designed to withstand outages and failures, when a broker goes down and stays out for some reason, a backup broker takes over the activities and maintains delivery and availability without losing any information, the same thing goes for topics that have multiple partitions that are replicas of the primary. These replicas are often scattered on other machines.
Another very interesting thing about Kafka is that unlike some other messaging systems, even after the consumer reads the message, it is not thrown away, so if we have a new consumer who wanted to read the data from the beginning, he can, but of course, if desired, it is possible to configure the time this data will be available before being deleted.
Final considerations
Apache Kafka is really a fantastic system for you to be able to work with messaging tools, including working with Pub/Sub, for example.
Therefore, everyone reads a channel and a producer produces there on that channel, that is, in this topic everyone can read these messages.
If you are thinking of some solution for you to make your system communicate securely and efficiently and you still need to process these messages Apache Kafka is a great solution for you to consider.
Top comments (0)