Kafka - Overview

Agenda

• Kafka Architecture
  • Basic concepts(topic/partition/consumer group/commit log/offset)
• Delver into the process of sending a message to the broker
• Technical Highlights

Architecture - Overview

Architecture - Detail

Architecture - Basic Concepts

Topics

A Kafka topic (https://www.instaclustr.com/support/documentation/kafka/using-kafka/topic-management/) defines a channel through which data is streamed.Producers publish messages to topics,and consumers read messages from the topic they subscribe to.

Topics organize and structure messages,with particular types of messages published to particular topics.Topics are identified by unique names within a Kafka cluster, and there is no limit on the number of topics that can be created.

Partitions

Within the Kafka cluster, topics are divided into partitions (https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/), and the partitions are replicated across brokers. From each partition, multiple consumers can read from a topic in parallel. It's also possible to have producers add a key to a message-all messages with the same key will go to the same partition.

While messages are added and stored within partitions in sequence,messages without keys are written to partitions in a round robin fashion.By leveraging keys,you can guarantee the order of processing for messages in Kafka that share the same key.
This is a particularly useful feature for applications that require total control over records.
There is no limit on the number of Kafka partitions that can be created(subject to the processing capacity of a cluster).