Apache Kafka: Event Streaming Architecture (2026-09-04 20:31)

Apache Kafka: Event Streaming Architecture

Apache Kafka has become the de facto standard for building real-time data pipelines and event-driven systems. Originally developed at LinkedIn and open-sourced in 2011, Kafka provides a distributed, fault-tolerant platform for handling high-throughput event streams. This post explores its core architecture and the concepts you need to design robust streaming systems.

What Is Event Streaming?

Event streaming captures data in real time from event sources—databases, sensors, mobile devices, applications—and stores these events durably for retrieval, processing, and routing to different destinations. Unlike traditional messaging, Kafka retains events for a configurable period, enabling both real-time and historical processing.

Core Concepts

Topics and Partitions

A topic is a named category to which events are published. Each topic is split into partitions, which are the fundamental unit of parallelism and ordering.

• Events within a partition are strictly ordered.
• Events across partitions have no global ordering guarantee.
• Each event within a partition has a sequential offset.

Topic: user-events
├── Partition 0: [msg0][msg1][msg2][msg3] ...
├── Partition 1: [msg0][msg1][msg2] ...
└── Partition 2: [msg0][msg1][msg2][msg3][msg4] ...

Producers and Consumers

Producers publish events to topics. They can choose a partition explicitly or rely on a partitioning strategy (e.g., hashing a key).

Consumers read events from topics. They are organized into consumer groups, where each partition is consumed by exactly one consumer within a group—enabling horizontal scaling.

// Producer example
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.send(new ProducerRecord<>("user-events", "user-123", "login"));
producer.close();

Brokers and Clusters

A Kafka broker is a single server that stores data and serves client requests. A cluster consists of multiple brokers working together. Partitions are distributed across brokers for scalability, and replicated for fault tolerance.

Replication and Fault Tolerance

Each partition has a leader replica and zero or more follower replicas. Producers and consumers interact only with the leader, while followers replicate the leader's data.

• The In-Sync Replicas (ISR) set contains replicas that are fully caught up.
• If a leader fails, a follower from the ISR is promoted.
• The acks producer setting controls durability guarantees:

acks	Behavior
0	Fire-and-forget; no acknowledgment
1	Leader acknowledges only
all	Leader + all ISR acknowledge

For strong durability, combine acks=all with min.insync.replicas=2.

Consumer Offsets and Delivery Semantics

Consumers track progress via committed offsets. Kafka supports three delivery semantics:

• At-most-once: Commit offsets before processing. Risk of data loss.
• At-least-once: Commit offsets after processing. Risk of duplicates.
• Exactly-once: Achieved via idempotent producers and transactions.

// Enabling exactly-once semantics
props.put("enable.idempotence", "true");
props.put("transactional.id", "my-transactional-id");

The Role of Coordination

Historically, Kafka relied on ZooKeeper for cluster metadata and controller election. Newer versions introduce KRaft (Kafka Raft), which removes the ZooKeeper dependency by managing metadata internally using a Raft-based consensus protocol. KRaft simplifies operations and improves scalability for large clusters.

Kafka Streams and Connect

Two key components extend Kafka's capabilities:

• Kafka Streams: A client library for building stream-processing applications with stateful operations, windowing, and joins.
• Kafka Connect: A framework for integrating Kafka with external systems (databases, object stores) using reusable source and sink connectors.

// Kafka Streams word count example
StreamsBuilder builder = new StreamsBuilder();
builder.<String, String>stream("input-topic")
    .flatMapValues(value -> Arrays.asList(value.toLowerCase().split(" ")))
    .groupBy((key, word) -> word)
    .count()
    .toStream()
    .to("word-count-output");

Design Best Practices

1. Choose partition count carefully. More partitions increase parallelism but add overhead. Plan for future scale since decreasing partitions isn't supported.
2. Use meaningful keys. Keys determine partition assignment and ordering guarantees.
3. Set retention appropriately. Balance storage cost against replay requirements.
4. Monitor consumer lag. Rising lag indicates consumers can't keep up with producers.
5. Replicate for durability. A replication factor of 3 is standard for production.

Conclusion

Apache Kafka's architecture—built around distributed, partitioned, replicated logs—makes it a powerful foundation for event-driven systems. By understanding topics, partitions, replication, and delivery semantics, you can design pipelines that are scalable, durable, and resilient. As the ecosystem evolves with KRaft and richer stream-processing