Currently when we need to connect large information systems with different formats and volumes we will depend on tools that are in charge of being in the middle of our architecture, organizing and distributing all that information operating in high availability, this is where the integration options we have could fit very well Apache Kafka would fit our project very well.
What is Apache Kafka?
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
https://kafka.apache.org
Get Apache Kafka
At the time of writing this post it is in version 2.6
https://kafka.apache.org/downloads
Kafka Architecture
- Brokers: In charge of containing the topics of the cluster
- Publishers: They are in charge of putting the messages on the topics
- Consumers: They extract the messages from the topics to process them
- Cluster: A set of Kafka nodes
- Topics: Flow of data that have a same context
- Partitions: The corresponding divisions of a topic
- Zookeeper: Orchest all nodes in the cluster
Zookeeper
To use Kafka we need to first start Zookeeper, in the installation folder there is the shell command in the folder bin. Which is:
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Kafka
Now we start Kafka, we need to locate ourselves in the bin directory where the shell is located.
$ bin/kafka-server-start.sh config/server.properties
Creating the topic
To generate a topic we must execute:
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic my-topic
List all topics
$ bin/kafka-topics.sh --list --zookeeper localhost:2181
Get information about the topic
$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic
Delete a topic
$ bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic my-topic
Creating a consumer
$ bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092
Creating a producer
$ bin/kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092
> Mensaje 1
> Mensaje 2
Top comments (1)
Great!!!