Did You Know? Kafka Stores Your Messages on Disk 🤔

Macky — Tue, 15 Sep 2026 13:18:37 +0000

Most people assume that a high-performance messaging system like Kafka must be keeping everything in memory to stay fast.

But here's the truth — every single message you produce to Kafka gets written to disk.

First, let's go step by step — I'll start by creating a topic using this command:

Next, let's try producing a message using this command — I'll send the word "hellooooooo" to Kafka:

Kafka will then store that message inside the container at this path:

/var/lib/kafka/data/test-topic-0/

00000000000000000000.log
The actual data file. This is where Kafka stores your messages — including the "hellooooooo" we just produced. Every message gets appended here sequentially.

00000000000000000000.index
An index file that maps offset → position in the .log file. So when a consumer wants message at offset 5, Kafka looks here first to find exactly where to jump in the .log file — without scanning the whole thing.

00000000000000000000.timeindex
Similar to .index but maps timestamp → offset. Useful when you want to find messages from a specific point in time, like "give me all messages after 10:00 AM."

leader-epoch-checkpoint
Tracks which leader epoch this partition is on. Think of it as a version counter — every time a new leader is elected (e.g. after a broker crash), the epoch increments. This helps prevent data inconsistency.

partition.metadata
Stores basic metadata about this partition — like which topic ID it belongs to. Kafka uses this internally to manage the partition correctly.

Finally, let's peek inside the .log file to see what the actual stored data looks like:

ET???????j???j&helloooooooooo

You can see the word "helloooooooooo" is right there — but it's not stored as plain text. The message is wrapped with binary metadata (the weird ET???????j???j& part) which includes things like:

Offset
Timestamp
Checksum
Message size

So Kafka doesn't just dump your raw text — it wraps it in a binary format called a Record Batch before writing to disk. That's why you see those strange characters around your actual message.

How and why : introduce

Macky — Tue, 15 Sep 2026 06:10:39 +0000

Hey everyone! 👋

I'm starting a new series where I'll be sharing things about the technology behind banking systems — what they use, how they handle massive loads, and why certain tech choices are made.

But here's the thing — this series isn't just about the fancy stuff. I actually want to focus on the simple, foundational concepts that most people overlook, especially in the AI era where everyone just searches for a quick answer and moves on.

And that's kind of the point. When we stop digging into how and why things work, we slowly lose that understanding. A search result gives you an answer — but it doesn't always give you the intuition behind it.

That said — I'm not against AI at all. In fact, I'll be using AI to help me write code and speed up the process of experimenting with different approaches. AI is a great tool for moving faster, but understanding the fundamentals is what helps you actually know what to build and why it works.

So whether you're a developer, someone curious about fintech, or just want to understand what's actually running under the hood of your banking app — this series is for you.

Stay tuned, and feel free to follow along. More coming soon! 🚀

DEV Community: Macky

Did You Know? Kafka Stores Your Messages on Disk 🤔

How and why : introduce