dev-to-uploads.s3.amazonaws.com/i/...
According to HiveMQ CTO Dominik Obermaier, Kafka can bring lot of value to IoT solutions thanks to its capacity to process large amounts of real-time data providing high throughput and low latency.
Kafka, though, is not by itself especially geared towards IoT applications due to a fundamental mismatch between the requirements of a typical stream-processing solution and those of an IoT solution. In particular, Obermaier mentions the restricted number of topics Kafka consumers and producers may handle in comparison with potentially millions of topics used in an IoT application; the complexity of typical Kafka API implementations which does not lend itself easily to use on constrained devices; the unavailability of key IoT features such as keep alive and last will and testament and others.
This is where the HiveMQ Enterprise Extension for Kafka comes into the picture. One of its key features, indeed, is its ability to map MQTT topics to Kafka topics, thus allowing to overcome Kafka unsuitability to handle large numbers of topics. Similarly, all IoT devices will only see the MQTT broker, so they can keep using MQTT with its reduced requirements and full support for keep alive and last will and testament. Additionally, in cases where the unreliability of the network may break the connection between the broker and the Kafka clusters, the former will buffer all received messages, ensuring they are not lost. conclusion
Based on a publish/subscribe model, Kafka is one of the most widely used platforms to process and distribute real-time data streams. MQTT is a publish/subscribe protocol particularly suited to IoT applications thanks to its small footprint, real-time guarantees, and suitability for use in high-latency, low-throughput, and unreliable networks.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
dev-to-uploads.s3.amazonaws.com/i/...
According to HiveMQ CTO Dominik Obermaier, Kafka can bring lot of value to IoT solutions thanks to its capacity to process large amounts of real-time data providing high throughput and low latency.
Kafka, though, is not by itself especially geared towards IoT applications due to a fundamental mismatch between the requirements of a typical stream-processing solution and those of an IoT solution. In particular, Obermaier mentions the restricted number of topics Kafka consumers and producers may handle in comparison with potentially millions of topics used in an IoT application; the complexity of typical Kafka API implementations which does not lend itself easily to use on constrained devices; the unavailability of key IoT features such as keep alive and last will and testament and others.
This is where the HiveMQ Enterprise Extension for Kafka comes into the picture. One of its key features, indeed, is its ability to map MQTT topics to Kafka topics, thus allowing to overcome Kafka unsuitability to handle large numbers of topics. Similarly, all IoT devices will only see the MQTT broker, so they can keep using MQTT with its reduced requirements and full support for keep alive and last will and testament. Additionally, in cases where the unreliability of the network may break the connection between the broker and the Kafka clusters, the former will buffer all received messages, ensuring they are not lost.
conclusion
Based on a publish/subscribe model, Kafka is one of the most widely used platforms to process and distribute real-time data streams. MQTT is a publish/subscribe protocol particularly suited to IoT applications thanks to its small footprint, real-time guarantees, and suitability for use in high-latency, low-throughput, and unreliable networks.