Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Use-case
Kafka is helpful in various real-life operational and data analytics use cases.
•Messaging: This domain has its own specialized software, like RabbitMQ and ActiveMQ, but Kafka is often sufficient to handle it while providing great performance.
•Website activity tracking: Kafka can handle small, frequently generated data records like page views, user actions, and other web-based browsing activity.
•Metrics: You can easily consolidate and aggregate data that can be sorted using topics.
•Log aggregation: Kafka makes it possible to gather logs from different sources and aggregate them in one place in a single format.
•Stream processing: Streaming pipelines are one of the most important Kafka features, making it possible to process and transform data in transit.
•Event-driven architecture: Applications can publish and react to events asynchronously, allowing events in one part of your system to easily trigger behavior somewhere else. For example, a customer purchasing an item in your store can trigger inventory updates, shipping notices, etc.
Architectural components
These are the most essential high-level components of Kafka:
Record
Producer
Consumer
Broker
Topic
Partitioning
Replication
ZooKeeper or Controller Quorum
1.Record
Also called an event or message, a record is a byte array that can store any object of any format. An example would be a JSON record describing what link a user clicked while they were on your website.
Sometimes you want to distribute certain kinds of events among a group of consumers, so each event will be distributed to just one of the consumers in that group. Kafka allows you to define consumer groups this way.
A critical design approach is that, besides consumer groups, no other interconnection happens among clients. Producers and consumers are fully decoupled and agnostic of each other.
2.Producer
A producer is a client application that publishes records (writes) to Kafka. An example here is a JavaScript snippet on a website that tracks browsing behavior on the site and sends it to the Kafka cluster.
Consumer
A consumer is a client application that subscribes to records from Kafka (i.e. reads them), such as an application that receives browsing data and loads it into a data platform for analysis.
3.Broker
A broker is a server that handles producer and consumer requests from clients and keeps the data replicated within the cluster. In other words, a broker is one of the physical machines Kafka runs on.
4.Topic
A topic is a category that allows you to organize messages. Producers send to a topic, while consumers subscribe to topics of relevance, so they only see the records they actually care about.
5.Partitioning
Partitioning means breaking a topic log into multiple logs that can live on separate nodes on the Kafka cluster. This allows you to have topic logs that are too big to live on one single node.
6.Replication
Partitions can be copied among several brokers to stay safe in case one broker experiences a failure. These copies are called replicas.
7.Ensemble service
An ensemble is a centralized service for maintaining configuration information, discovery, and providing distributed synchronization and coordination. Kafka used to rely on Apache ZooKeeper for this, although newer versions have moved to a different consensus service called KRaft.
Not all event streaming software requires installing a separate ensemble service. Redpanda, which offers 100% Kafka-compatible data streaming, works out of the box because it already has this functionality built-in.
Top comments (0)