Apache Kafka - Fundamentals, Use cases and Trade-Offs

#kafka #opensource #architecture #eventdriven

This post will focus on open source and popular Apache kafka platform related brief fundamentals, Use Cases and Trade-offs with other popular messaging technologies.

Kafka Fundamentals - Brief

Apache kafka is an event streaming platform. It provides distributed, high-throughput, low-latency, fault-tolerant platform for handling real-time data feeds - known as events.

It can be used in different ways due to various capabilities combined in Kafka.

Message System - to Publish and Subscribe events
Stream Processing System - to Process steams of events
Storage System - to Store log of events reliably (forever, if needed)

Kafka has topics, partitions, producers and consumers, which are basics for any messaging system. For more details on Kafka 101, please refer this link, where it is very nicely explained.

Kafka has following message / event delivery guarantees between producer and consumer.

At most once — Messages may be lost but are never redelivered.
At least once — Messages are never lost but may be redelivered.
Exactly once — Each message is delivered once and only once.

Kafka Use Cases

For anything that deals with events and real-time data, Kafka is the way to go to process events and get insights - all in real-time. Kafka supports wide variety of use cases due to large ecosystem. Some of the example use cases are as below.

Financial Services - Real-time fraud detection and faster payments
Automobile - Real-time data cluster for Anomaly detection, Alerts and Personalization for better driving experience
Healthcare - Real-time health sensors data to get better insights to serve patients and save lives

Kakfa can be used as backbone to integrate various systems - be it data analytics use case or microservices communications in an event-driven architecture. Following diagram shows kafka as backbone for various systems integration.

Kafka Trade-offs with other messaging systems

For basic, single machine, message oriented, point-to-point systems communication, classical messaging suits with RabbitMQ or ActiveMQ.

But for any event-streaming, where there is event-based communication with multiple consumers and need ordered events (either same events to multiple consumers or specific events to specific consumer group), then Kafka is best suited.

Apart from being messaging system, Kafka can be leveraged as stream Processing System (to Process steams of events) and Storage System (to Store log of events reliably - forever, if needed). It also has lots of other features in Kafka ecosystem, such as kafka connectors and kafka streams (please refer kafka documentation for more details). So for event-driven, real-time systems - Kafka is most matured and complete ecosystem.

Enjoy building event-driven systems with Kafka !

Please like, share and comment your views/ opinion.

Disclaimer - This post is my personal opinion and does not reflect those of any of my past, present or future employers or affiliations.

Top comments (18)

Giuseppe • Oct 24 '20

Please, can you explain better when to use Kafka instead of Rabbit MQ? I haven't really understand, you said that Kafka is used for log storage, but in the official tutorial of RabbitMQ there is also an example about log storage. What are the differences?

Dhruvesh Patel • Oct 25 '20

Kafka uses a distributed commit log as its storage layer. Writes are appended to the end of the log and Reads are sequential starting from an offset. This suites best for event streaming use cases.

Conversely, RabbitMQ uses index-based storage systems. These keep data in a tree structure to provide the fast access necessary for acknowledging individual messages, but fast individual reads comes at the cost of write overhead, which leads to either decreased write throughput and increased write latency compared to a log.

RabbitMQ is designed to store messages for a short period of time only. Kafka can retain messages indefinitely.

Hope this helps.