Apache Kafka: A Beginner's Guide to Key Concepts

Apache Kafka

A busy post office keeps on delivering even when one worker is slow. The letters keep on arriving on a constant rate, delivery staff picks the letters up and drops them, and no one waits for the other.

Now, Apache Kafka works this way but now for data. It is a distributed platform for data streaming. Some application sends data to Kafka, and other read the same data from Kafka. Everything happens fast and in real time

The problem Kafka actually solves.

Before Kafka was developed, applications used to talk to each other directly. APP A sends data to APP B, which in turn send the data to APP C. It seems easy at first, but as systems grew, the massive amount of data sending becomes messy. If App C is down, App A has to wait and if App B is down, the whole process collapses.

This is where Kafka comes in, instead of the apps sending data directly to other apps, Kafka acts as an intermediate. producers send data into Kafka and consumers read that data from Kafka.

Key Concepts:

1. Events and Messages

An event is a record that something happened. A user clicked a button, a sensor recorded a temperature or a payment was processed. These messages are referred to as records in Kafka.

Every message has 3 parts:

• A key, which is an optional identifier( e.g : userID)
• A value, this is the actual data( e.g: {"user": "jane", "action": "purchase"})
• Timestamp, when it happened.

2. Topics

A topic is a named channel where the messages live. A topic is like a folder in a computer, where the messages are the files inside that folder. If one creates a topic named users, each time a new user registers, then the app where the user was registered sends a message to the user topic in Kafka.

Topics are append only, meaning new messages go at the end, and old messages still remain. No messages disappear even after being read.

A small example:

Topic: payments
├── msg 1: {"id": "a1", "amount": 50}
├── msg 2: {"id": "a2", "amount": 120}
└── msg 3: {"id": "a3", "amount": 30}

Kafka allows as many topics as one wishes to create. May it be users, inventory, payments, email-sent etc.

3. Partitions.

A Topic can hold data in large amounts. In order to handle this, Kafka splits topics into partitions. Each partition contains a numbered log of messages. For example, if payment has 3 partitions, then Kafka spreads messages across all the 3 partitions.

Each message inside a partition gets a number called an offset. offset 0 is the first message and offset 1 is the second message. The offset never changes, which allows consumers to track where they are.

Partition 0: [offset 0] [offset 1] [offset 2]
Partition 1: [offset 0] [offset 1]
Partition 2: [offset 0] [offset 1] [offset 2] [offset 3]

This is important because of parrarelism, which allows multiple consumers to read messages at the same time. If there are 10 partitions, then 10 consumers can read the data.

4. Producers

A producer is any application that sends messages to kafka. When an e-commerce backend sends order data , it is a producer, and when a mobile app sends user activity, it as also a producer.

When writing topics, producers get to pick which topic they are writing. Producers can also either chose where the message they are writing goes or let Kafka chose. If Kafka decides, it writes data in one partition, then the next, and the next, then comes back to the first. Like that.

To simplify this, one can decide what partition they wish the data to go by passing a key argument directly in the send () call. Messages with the same key, go to the same partition. e.g.

from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lamda v: json.dumps(v).encode('utf-8')
)
.
# This message goes to partition with key 'user-10'

producer.send(
    topic='payments',
    key=b'user-10',
    value={"amount": 50}
)

producer.flush()

Here you are specifying the exact partition where the message goes.

5. Consumers and Consumer Groups

A consumer reads messages from a topic. A consumer tracks the offsets, so as to determine which messages have already been processed.

A consumer Group is a team of consumers working together to read messages from a topic. Kafka assigns one partition to one consumer in the group. For example, if a topic has 4 partitions, each consumer get assigned exactly one partition, and if one consumer fails, Kafka assigns the partition to the others automatically, in a process called rebalancing.

from kafka import KafkaConsumer
import json

consumer = KafkaConsumer(
    topic='payments',
    bootstrap_servers='localhost:9092',
    group_id='payments-processor',
    value_deserializer=lambda v: json.loads(v.decode('utf-8'))
)

for message in consumer:
    print(f"Offset: {message.offset}, Data: {message.value}")

plaintext

6. Brokers and the Cluster

A broker is a single Kafka server. Brokers store the partitions and serve the producers and consumers. A cluster is a group of brokers. In production, one runs at least 3 brokers. Data is copied, or replicated, onto each broker for safety. If one broker fails, another broker has the data it needs to serve the clients.

Each partition has a leader (one of the brokers) and the replica brokers. Producers and consumers always communicate with the leader. The replicas stay in sync with the leader. If the leader broker fails, one of the replica brokers automatically becomes the new leader.

Broker 1 (Leader for partition 0)
Broker 2 (Replica for partition 0, Leader for partition 1)
Broker 3 (Replica for partition 1)

7. Retention

Messages are kept for a set amount of time. The default time is 7 days. After that time, the messages will be automatically deleted. This period can be changed on a per-topic basis. This is very powerful as it allows consumers to fall behind in processing the events and then catch up later. Another consumer could also replay all the events that occurred from the beginning of the topic. There is no limit to processing these events in real time or catching up later.

A Real Example of How Kafka Fits in

• Customer places an order - This is a backend that produces a message to orders topic.
• The kitchen app consumes from orders - starts preparing the food.
• The delivery app consumes from orders - assigns a driver.
• The billing app consumes from orders - charges the customer.

All the three consumers work independently, such they never block one another. So if the delivery app crashes, it picks up where it left off when it resumes since the offsets are saved.

Conclusion

Kafka may seem complicated at the start, but becomes simpler as one connects the process to familiar objects. Topics are folders, messages/events are files., Partitions are subfolders in the big folder. Producers are writers of these files and consumers read these files. Brokers are the office buildings that physically hold the filing cabinets. If one building burns down, another building already has a backup copy of those files. Work continues without losing anything.