Did You Know? Kafka Stores Your Messages on Disk 🤔

Most people assume that a high-performance messaging system like Kafka must be keeping everything in memory to stay fast.

But here's the truth — every single message you produce to Kafka gets written to disk.

First, let's go step by step — I'll start by creating a topic using this command:

Next, let's try producing a message using this command — I'll send the word "hellooooooo" to Kafka:

Kafka will then store that message inside the container at this path:

/var/lib/kafka/data/test-topic-0/

00000000000000000000.log
The actual data file. This is where Kafka stores your messages — including the "hellooooooo" we just produced. Every message gets appended here sequentially.

00000000000000000000.index
An index file that maps offset → position in the .log file. So when a consumer wants message at offset 5, Kafka looks here first to find exactly where to jump in the .log file — without scanning the whole thing.

00000000000000000000.timeindex
Similar to .index but maps timestamp → offset. Useful when you want to find messages from a specific point in time, like "give me all messages after 10:00 AM."

leader-epoch-checkpoint
Tracks which leader epoch this partition is on. Think of it as a version counter — every time a new leader is elected (e.g. after a broker crash), the epoch increments. This helps prevent data inconsistency.

partition.metadata
Stores basic metadata about this partition — like which topic ID it belongs to. Kafka uses this internally to manage the partition correctly.

Finally, let's peek inside the .log file to see what the actual stored data looks like:

ET???????j???j&helloooooooooo

You can see the word "helloooooooooo" is right there — but it's not stored as plain text. The message is wrapped with binary metadata (the weird ET???????j???j& part) which includes things like:

Offset
Timestamp
Checksum
Message size

So Kafka doesn't just dump your raw text — it wraps it in a binary format called a Record Batch before writing to disk. That's why you see those strange characters around your actual message.