How Kafka Works: A Quick Walkthrough with Java

Your order service needs to tell the invoice service, the email service, and the analytics pipeline that an order happened. You could wire them together directly - but every new consumer means changing the producer, and if a consumer is down, messages get lost.

Kafka takes a different approach: the producer drops the message into a durable log, and each consumer reads that log whenever it is ready, at its own pace, without the producer even knowing who they are. Let's see the whole flow working, with a real Kafka broker and a real Java client.

The five pieces that matter

• Topic: a named stream of messages, like greetings. Producers write to it, consumers read from it.
• Partition: a topic is split into partitions. Each partition is an ordered, append-only log. Partitions are how Kafka scales: different consumers can read different partitions in parallel.
• Offset: a message's position in its partition - 0, 1, 2, ... Once written, a message never changes; consumers just move forward through the offsets.
• Producer: the app that appends messages. If you give the message a key, the key's hash decides which partition it lands in, so all messages with the same key stay in order in the same partition.
• Consumer group: consumers with the same group id share the partitions between them and Kafka tracks how far each group has read.

Here is the whole journey, animated - three messages go in, and a consumer pulls them back out:

Running a real broker

The official Kafka quickstart runs a single-node broker without ZooKeeper (KRaft mode) in three commands, which is exactly what I did - this article's output is from Apache Kafka 4.4.0 on JDK 21:

# 1. generate a cluster id
KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)"

# 2. format the log directories
bin/kafka-storage.sh format --standalone -t $KAFKA_CLUSTER_ID -c config/server.properties

# 3. start the broker
bin/kafka-server-start.sh config/server.properties

Startup logged Kafka Server started (kafka.server.KafkaRaftServer), and the broker was ready on localhost:9092.

A Java producer and consumer in one file

This complete program creates a topic with 3 partitions, produces 3 keyed messages, and reads them back. It uses only the kafka-clients jar that ships in the Kafka download's libs folder - no extra dependencies:

import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.NewTopic;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.apache.kafka.common.serialization.StringSerializer;

import java.time.Duration;
import java.util.List;
import java.util.Properties;

public class KafkaWalkthrough {

    static final String TOPIC = "greetings";

    public static void main(String[] args) throws Exception {
        createTopic();
        produce();
        consume();
    }

    static void createTopic() throws Exception {
        Properties p = new Properties();
        p.put("bootstrap.servers", "localhost:9092");
        try (AdminClient admin = AdminClient.create(p)) {
            admin.createTopics(List.of(new NewTopic(TOPIC, 3, (short) 1))).all().get();
            System.out.println("[setup] created topic '" + TOPIC + "' with 3 partitions");
        }
    }

    static void produce() throws Exception {
        Properties p = new Properties();
        p.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        p.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        p.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        try (KafkaProducer<String, String> producer = new KafkaProducer<>(p)) {
            String[][] msgs = {
                {"vishnu", "hello kafka"},
                {"asha", "kafka is a log"},
                {"ravi", "partitions scale reads"},
            };
            for (String[] m : msgs) {
                var meta = producer.send(new ProducerRecord<>(TOPIC, m[0], m[1])).get();
                System.out.printf("[producer] sent key=%s value='%s' -> partition=%d offset=%d%n",
                        m[0], m[1], meta.partition(), meta.offset());
            }
        }
    }

    static void consume() {
        Properties p = new Properties();
        p.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        p.put(ConsumerConfig.GROUP_ID_CONFIG, "walkthrough-group");
        p.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        p.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
        p.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(p)) {
            consumer.subscribe(List.of(TOPIC));
            int got = 0;
            long deadline = System.currentTimeMillis() + 15000;
            while (got < 3 && System.currentTimeMillis() < deadline) {
                ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
                records.forEach(r -> {
                    System.out.printf("[consumer] read key=%s value='%s' from partition=%d offset=%d%n",
                            r.key(), r.value(), r.partition(), r.offset());
                });
                got += records.count();
            }
        }
    }
}

I compiled it with javac -cp "libs/*" KafkaWalkthrough.java and ran it against the live broker. This is the real, unedited output:

[setup] created topic 'greetings' with 3 partitions
[producer] sent key=vishnu value='hello kafka' -> partition=0 offset=0
[producer] sent key=asha value='kafka is a log' -> partition=0 offset=1
[producer] sent key=ravi value='partitions scale reads' -> partition=0 offset=2
[consumer] read key=vishnu value='hello kafka' from partition=0 offset=0
[consumer] read key=asha value='kafka is a log' from partition=0 offset=1
[consumer] read key=ravi value='partitions scale reads' from partition=0 offset=2

Four honest observations from that output:

1. All three keys hashed to partition 0. With only three partitions and three keys, that is chance, not a rule - other keys land in partitions 1 and 2. The guarantee is narrower: the same key always goes to the same partition.
2. Offsets count up within a partition. 0, 1, 2 - the append-only log in action.
3. The consumer replayed history. The producer had already finished and exited when the consumer started. Nothing was lost, because the topic stores messages; AUTO_OFFSET_RESET_CONFIG = "earliest" told the new consumer group to start from offset 0.
4. Order is per-partition. The consumer got the messages in offset order. Across partitions there is no global order - another reason keys matter when order matters.

When Kafka is the right tool

• Decoupling services: the producer does not know or care who consumes.
• Buffering bursts: consumers catch up after a spike or an outage instead of dropping work.
• Multiple independent readers: analytics, email, and search can each have their own consumer group over the same topic.
• Replay: because messages persist, a new service can rebuild its state by reading from offset 0.

It is more machinery than a plain queue, so for one producer talking to one consumer with no replay needs, a simpler tool can be the better call. Kafka earns its keep when the log itself - durable, ordered per partition, readable by many groups - is what you need.

Sources

• Apache Kafka documentation: introduction
• Apache Kafka quickstart

Written with AI assistance; code examples were compiled and run against a real single-node Kafka 4.4.0 broker on JDK 21 before publishing. The animation is an illustration of the message flow, not a screen recording.