Apache Kafka

Apache Kafka is a powerful open-source distributed event streaming platform designed to handle real-time data feeds. It was originally developed by LinkedIn and is now maintained by the Apache Software Foundation. Kafka is widely used for building real-time data pipelines and streaming applications.

Core Concepts of Apache Kafka

1. Topics:

   • A Kafka topic is a category or feed name to which records are sent by producers.
   • Topics are partitioned, meaning each topic can have multiple partitions to allow parallel processing.
   • Example: A topic named user_activity could store logs of user actions on a website.

2. Partitions:

   • Each topic is divided into partitions, which are ordered, immutable sequences of records.
   • Partitions allow Kafka to scale horizontally by distributing data across multiple brokers.
   • Each record in a partition is assigned a unique offset.

3. Producers:

   • Producers are applications that send data to Kafka topics.
   • They can specify which partition to send data to or let Kafka decide based on a partitioning strategy.

4. Consumers:

   • Consumers are applications that read data from Kafka topics.
   • They can subscribe to one or more topics and process the data in real-time.

5. Brokers:

   • Kafka brokers are servers that store data and serve client requests.
   • A Kafka cluster consists of multiple brokers, and each broker is identified by a unique ID.

6. ZooKeeper:

   • ZooKeeper is used for managing and coordinating Kafka brokers.
   • It handles tasks like leader election for partitions and storing metadata.

7. Offsets:

   • An offset is a unique identifier for each record within a partition.
   • Consumers use offsets to keep track of their position in the topic.

8. Replication:

   • Kafka replicates data across multiple brokers to ensure fault tolerance.
   • Each partition has one leader and multiple replicas.

Kafka APIs

1. Producer API:

   • Allows applications to send streams of data to topics.

2. Consumer API:

   • Enables applications to read streams of data from topics.

3. Streams API:

   • Used for building real-time stream processing applications.

4. Connect API:

   • Facilitates integration between Kafka and other systems like databases or file systems.