Most data systems are built on direct connections. Service A calls Service B's API. If B is down, A fails. If A sends data before B is ready, the data is lost. Every integration is a point-to-point contract that breaks the moment either side changes.
Kafka solves this by introducing a persistent, distributed log layer between producers and consumers.
What Kafka Is
Kafka is a distributed event store and stream-processing platform. A Kafka cluster consists of brokers, servers that store topic partitions. Producers write records to topics. Consumers read records from topics. Producers and consumers never interact directly.
# Producer
producer.send('orders', key='user_42', value={'id': 123, 'item': 'laptop'})
# Consumer
for msg in consumer:
process(msg.value)
Each record has a key, value, timestamp, and optional headers. The key determines partition assignment, records with the same key always go to the same partition, preserving order for that key.
The Commit Log
Kafka's core data structure is the commit log which is an append-only, ordered sequence of records. Records are written sequentially and never mutated after write. This makes writes extremely fast and simplifies replication.
Each partition is a single log file split into segments. Old segments are deleted or compacted based on retention policy. Because the log is append-only, Kafka avoids the random I/O that plagues traditional databases.
Brokers, Replication, and ISR
Each partition is replicated across multiple brokers for fault tolerance. One broker is the leader for a given partition; the rest are followers. Producers write to the leader. Followers replicate data from the leader.
Kafka uses an in-sync replica (ISR) set, which is the subset of followers that are fully caught up with the leader. The leader acknowledges a write only when all ISR replicas have confirmed it. If the leader fails, an ISR follower is elected as the new leader. This guarantees that acknowledged records are never lost.
Persistent Logs, Not Queues
Traditional message queues delete a message after consumption. Kafka retains messages on disk based on configurable policies; time-based (e.g., 7 days) or size-based (e.g., 50 GB). Each consumer maintains its own offset: a monotonically increasing integer pointing to its position in the log.
A consumer commits its offset periodically. If the consumer crashes, it resumes from the last committed offset. A new consumer starting from offset 0 replays the entire partition history. Multiple consumers reading the same partition process independently, they don't compete for messages.
Kafka also supports log compaction, where only the most recent record for each key is retained. This is useful for restoring state: instead of replaying every event, a consumer gets the final value for every key.
Consumer Groups
A consumer group is a set of consumers that coordinate to read a topic. Each partition is assigned to exactly one consumer in the group. If you have 6 partitions and 3 consumers, each consumer gets 2 partitions. If you have 6 partitions and 6 consumers, each gets 1. If you have 6 partitions and 10 consumers, 4 sit idle — you cannot parallelize beyond the partition count.
Consumer groups enable horizontal scaling of consumption. Rebalancing occurs when consumers join or leave, partition ownership is redistributed. During a rebalance, consumers are temporarily paused, which is why frequent joins/leaves should be avoided.
Partitions and Ordering
Partitions are the unit of parallelism. Records within a partition are strictly ordered by offset. Records across partitions are unordered. This is a fundamental trade-off: Kafka guarantees order within a partition but not across them.
The partition key is hashed to determine partition assignment: hash(key) % num_partitions. This ensures all records with the same key land in the same partition, preserving order for that entity (e.g., all events for a specific customer ID).
Exactly-Once Semantics
Kafka provides exactly-once semantics (EOS) through idempotent producers and transactional consumption. The producer includes a unique identifier and sequence number in each record. The broker deduplicates retries. Transactions allow atomic writes across multiple partitions — either all records in a batch are committed or none are.
EOS introduces performance overhead and is unnecessary for many use cases. At-least-once delivery with idempotent downstream processing is often the simpler choice.
Retention and Tiered Storage
Kafka's retention is configurable per topic. Once a record exceeds the retention threshold, it's eligible for deletion. This prevents unbounded disk usage.
Tiered storage (Kafka 3.0+) moves older segment data to cheaper object storage (S3, GCS) while keeping metadata on the broker. Consumers can still read old data transparently ,the broker fetches it from the remote store. This allows retaining months or years of data without provisioning expensive broker storage.
Kafka Ecosystem Components
Kafka ships with several built-in components:
- Kafka Connect — imports and exports data from external systems using configurable connectors (source connectors pull data in, sink connectors push data out). Connectors are available for databases (JDBC, Debezium), S3, Elasticsearch, and hundreds of others.
- Kafka Streams — a Java library for building stream-processing applications. It handles partitioning, state management, and exactly-once processing without requiring a separate cluster.
-
KSQL — an interactive SQL interface over Kafka topics. You can run
CREATE STREAM orders WITH (kafka_topic='orders', value_format='json');and query it in real time.
Where Confluent Comes In
Confluent was founded by the original creators of Kafka. It provides enterprise features on top of the open-source project:
- Schema Registry — enforces a schema contract (Avro, Protobuf, JSON Schema) for every record. Producers and consumers reference schema IDs rather than raw schemas, preventing format mismatches and enabling schema evolution with compatibility rules.
- Control Center — a UI for monitoring cluster health, consumer lag (the delay between record production and consumption), throughput by topic and partition, and broker metrics.
- Confluent Cloud — a fully managed Kafka service. Cluster provisioning, broker scaling, rebalancing, and upgrades are handled automatically. You pay for throughput and storage rather than managing infrastructure.
- Enterprise connectors — certified versions of Kafka Connect connectors with SLAs and support.
Running Kafka in production requires tuning replication factor, partition count, segment size, retention policy, and broker heap settings. Confluent Cloud abstracts most of this, while self-managed Kafka gives you full control at the cost of significant operational complexity.
When to Use Kafka
Kafka is appropriate when:
- Multiple systems need to consume the same data stream independently
- Data must be replayed or re-processed after consumption
- Producers and consumers operate at different speeds and you need buffering
- You need a unified, immutable audit log of events
- You're building event-driven or event-sourcing architectures
Kafka is not appropriate for:
- Request-response patterns (use an API gateway)
- Sub-millisecond message delivery (use a message queue like RabbitMQ)
- Simple scheduled task execution (use a cron job or Airflow)
- Systems with very low throughput where operational overhead isn't justified
Bottom Line
Kafka replaces fragile point-to-point integrations with a persistent, distributed, scalable event log organized into topics and partitions. Records are immutable, replicated across brokers for fault tolerance, and retained for configurable periods. Consumers coordinate through consumer groups and maintain their own position via offsets. Confluent adds schema management, monitoring, and managed hosting for teams that don't want to operate Kafka clusters themselves.
Top comments (0)