Currently when we need to connect large information systems with different formats and volumes we will depend on tools that are in charge of being in the middle of our architecture, organizing and distributing all that information operating in high availability, this is where the integration options we have could fit very well Apache Kafka would fit our project very well.
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
At the time of writing this post it is in version 2.6
- Brokers: In charge of containing the topics of the cluster
- Publishers: They are in charge of putting the messages on the topics
- Consumers: They extract the messages from the topics to process them
- Cluster: A set of Kafka nodes
- Topics: Flow of data that have a same context
- Partitions: The corresponding divisions of a topic
- Zookeeper: Orchest all nodes in the cluster
To use Kafka we need to first start Zookeeper, in the installation folder there is the shell command in the folder
bin. Which is:
$ bin/zookeeper-server-start.sh config/zookeeper.properties
Now we start Kafka, we need to locate ourselves in the
bin directory where the shell is located.
$ bin/kafka-server-start.sh config/server.properties
To generate a topic we must execute:
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 3 --topic my-topic
$ bin/kafka-topics.sh --list --zookeeper localhost:2181
$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-topic
$ bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic my-topic
$ bin/kafka-console-consumer.sh --topic my-topic --from-beginning --bootstrap-server localhost:9092
$ bin/kafka-console-producer.sh --topic my-topic --bootstrap-server localhost:9092 > Mensaje 1 > Mensaje 2