DEV Community

Discussion on: Redis, Kafka or RabbitMQ: Which MicroServices Message Broker To Choose?

assyahid profile image
Assyahid Hasan Albana Author
According to HiveMQ CTO Dominik Obermaier, Kafka can bring lot of value to IoT solutions thanks to its capacity to process large amounts of real-time data providing high throughput and low latency.
Kafka, though, is not by itself especially geared towards IoT applications due to a fundamental mismatch between the requirements of a typical stream-processing solution and those of an IoT solution. In particular, Obermaier mentions the restricted number of topics Kafka consumers and producers may handle in comparison with potentially millions of topics used in an IoT application; the complexity of typical Kafka API implementations which does not lend itself easily to use on constrained devices; the unavailability of key IoT features such as keep alive and last will and testament and others.
This is where the HiveMQ Enterprise Extension for Kafka comes into the picture. One of its key features, indeed, is its ability to map MQTT topics to Kafka topics, thus allowing to overcome Kafka unsuitability to handle large numbers of topics. Similarly, all IoT devices will only see the MQTT broker, so they can keep using MQTT with its reduced requirements and full support for keep alive and last will and testament. Additionally, in cases where the unreliability of the network may break the connection between the broker and the Kafka clusters, the former will buffer all received messages, ensuring they are not lost.
Based on a publish/subscribe model, Kafka is one of the most widely used platforms to process and distribute real-time data streams. MQTT is a publish/subscribe protocol particularly suited to IoT applications thanks to its small footprint, real-time guarantees, and suitability for use in high-latency, low-throughput, and unreliable networks.