DEV Community

Ali Abbas
Ali Abbas

Posted on

1

Explain Apache Kafka like I'm five

I've been lately try to understand Apache Kafka, but I'm little bit unable to comprehend what it is actually. Can anybody explain it like I'm five.

Top comments (1)

Collapse
 
qm3ster profile image
Mihail Malo

It's a message broker/"database" (what an overloaded word).
It works by having a "chronologically" ordered queue/stream of entries (eg events).
A key difference from MQTT and other MQs is that rather than entries being "consumed" by a subscriber, or only one last message being available in a "topic", the entire stream can be read from the beginning or some intermediate ID. Eg, someone came and read messages #15-#25, someone else can come later and still read messages #10-#30.
The messages can be cleaned up by expiration or a more complex compaction strategy, so they don't take up infinite space.
Kafka can even be used as "durable storage", for example New York Times allegedly stores all their articles in history in a Kafka stream.
For this, it supports replication between multiple nodes, with "writes" appearing as done for the client only after a certain number of replicas receive them.
For high throughput, the streams can also be sharded, essentially making multiple streams which are each only individually ordered, not among each other.

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay