loading...
Cover image for Apache Kafka - Fundamentals, Use cases and Trade-Offs

Apache Kafka - Fundamentals, Use cases and Trade-Offs

dhruvesh_patel profile image Dhruvesh Patel ・2 min read

This post will focus on open source and popular Apache kafka platform related brief fundamentals, Use Cases and Trade-offs with other popular messaging technologies.

Kafka Fundamentals - Brief

Apache kafka is an event streaming platform. It provides distributed, high-throughput, low-latency, fault-tolerant platform for handling real-time data feeds - known as events.

It can be used in different ways due to various capabilities combined in Kafka.

  • Message System - to Publish and Subscribe events
  • Stream Processing System - to Process steams of events
  • Storage System - to Store log of events reliably (forever, if needed)

Kafka has topics, partitions, producers and consumers, which are basics for any messaging system. For more details on Kafka 101, please refer this link, where it is very nicely explained.

Kafka has following message / event delivery guarantees between producer and consumer.

  • At most once β€” Messages may be lost but are never redelivered.
  • At least once β€” Messages are never lost but may be redelivered.
  • Exactly once β€” Each message is delivered once and only once.

Kafka Message Delivery Guarantees

Kafka Use Cases

For anything that deals with events and real-time data, Kafka is the way to go to process events and get insights - all in real-time. Kafka supports wide variety of use cases due to large ecosystem. Some of the example use cases are as below.

  • Financial Services - Real-time fraud detection and faster payments
  • Automobile - Real-time data cluster for Anomaly detection, Alerts and Personalization for better driving experience
  • Healthcare - Real-time health sensors data to get better insights to serve patients and save lives

Kakfa can be used as backbone to integrate various systems - be it data analytics use case or microservices communications in an event-driven architecture. Following diagram shows kafka as backbone for various systems integration.

Kakfa - Use Cases and Systems

Kafka Trade-offs with other messaging systems

For basic, single machine, message oriented, point-to-point systems communication, classical messaging suits with RabbitMQ or ActiveMQ.

But for any event-streaming, where there is event-based communication with multiple consumers and need ordered events (either same events to multiple consumers or specific events to specific consumer group), then Kafka is best suited.

Kafka vs Legacy messaging systems

Apart from being messaging system, Kafka can be leveraged as stream Processing System (to Process steams of events) and Storage System (to Store log of events reliably - forever, if needed). It also has lots of other features in Kafka ecosystem, such as kafka connectors and kafka streams (please refer kafka documentation for more details). So for event-driven, real-time systems - Kafka is most matured and complete ecosystem.

Enjoy building event-driven systems with Kafka !

Please like, share and comment your views/ opinion.

Disclaimer - This post is my personal opinion and does not reflect those of any of my past, present or future employers or affiliations.

Discussion

pic
Editor guide
Collapse
ziggo profile image
Giuseppe

Please, can you explain better when to use Kafka instead of Rabbit MQ? I haven't really understand, you said that Kafka is used for log storage, but in the official tutorial of RabbitMQ there is also an example about log storage. What are the differences?

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Kafka uses a distributed commit log as its storage layer. Writes are appended to the end of the log and Reads are sequential starting from an offset. This suites best for event streaming use cases.

Conversely, RabbitMQ uses index-based storage systems. These keep data in a tree structure to provide the fast access necessary for acknowledging individual messages, but fast individual reads comes at the cost of write overhead, which leads to either decreased write throughput and increased write latency compared to a log.

RabbitMQ is designed to store messages for a short period of time only. Kafka can retain messages indefinitely.

Hope this helps.

Collapse
chiragks profile image
Chirag Shah

Nice article . Always enjoyed your post. Keep share like this.

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Thanks for kind words.

Collapse
prabhat36539454 profile image
Prabhat Kumar

Always a great learning to go through your article.

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Glad to help.

Collapse
levirs565 profile image
Levi Rizki Saputra

Good explanation.πŸ‘

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Thanks for the feedback.

Collapse
shreya1310 profile image
Shreya1310

Very helpful article πŸ‘

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Thanks for kind words.

Collapse
phyllisroberts profile image
PhyllisRoberts

Very Well Written

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Thanks for kind words.

Collapse
joelblanda profile image
JoelBlanda

Love to read more article..
Really help full.

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Happy to help.

Collapse
amandahopkins profile image
AmandaHopkins

I am searching in Google about Apache Kafka and come to this article.. nice article..

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Thanks for feedback.

Collapse
luciolangworth profile image
LucioLangworth

Explain very well, My many doubts clear...

Collapse
dhruvesh_patel profile image
Dhruvesh Patel Author

Thanks for kind words happy to help.