Initial scope was simple: just make it fast. No fsync, no fancy durability. Just raw speed.
I was bored. Asked AI "what should I build in C++?" It said "KV store". Okay, let's do it.
This is the full story. Not a clean tutorial, just a war story of all the dumb decisions and tradeoffs I made in one week.
Day 1-2: Per-Line KV - 300k w/s, Slow as Hell
The dumbest version: key:value\n per line. Read with fgets, scan from top to bottom.
Slow. Want to read 1 key? You have to scan everything. O(n). 100MB file = pain.
Lesson: need binary format, not text.
The Binary Format
Ended up with my own format:
[Checksum 4B][Status 1B][Type 1B][Key Len 1B][Value Len Varint][Key][Value]
Some decisions that looked dumb but actually made sense:
1. Key Len limited to 256 (1 byte).
Why? Just an engine decision. I haven't found a use case where keys are 1KB long. If 1 byte is enough, why use 4 bytes? Save 3 bytes per record x 10 million = 30MB saved.
2. Value Len was 32 bytes at first.
I wanted to support 500MB values. Super wasteful. A small 10-byte value had to carry a 32-byte header.
Then I learned varint. Manipulate MSB as a continuation flag.
Small (<127): 0xxxxxxx -> 1 byte
Large: 1xxxxxxx 0xxxxxxx -> 2 bytes
From 32 bytes to 1 byte for small data. Heaven.
3. Checksum as a recovery mode.
I only knew the concept as "compare". At first it was reversed, then XOR, then CRC. Not just for validation, but to know which record is corrupted after a crash.
Day 3-4: Indexing - From Scanning to HashMap
Still slow. Every read still scans the file.
Solution: HashMap. But at first I stored key -> value in the HashMap. RAM exploded.
Changed to: key -> offset. HashMap only holds the position in the file, not the value itself. Bitcask-style. RAM-friendly, read is just seek(offset).
Day 4-5: I/O - Ring Buffer & Page Faults
Reading per payload was still slow. One read = one syscall.
Tried ring buffer: read per chunk, erase-read method. If payload is incomplete at the end of the chunk, re-read from the last payload offset.
Still slow. Learned about 4KB page faults. Every 4KB, the OS has to load a new page.
- 4KB chunk: thousands of syscalls
- 8MB chunk: hundreds of syscalls
- 16MB chunk: crazy fast
Why? Because 16MB = 4096 page faults at once, not one by one.
This is what took me from 300k to 8M w/s (raw, no safety).
Day 6-7: mmap - The Final Cheat Code
AI recommended: "try mmap".
Mind blown. The file doesn't need to be read anymore. The file is RAM.
char* data = (char*) mmap(nullptr, file_size, PROT_READ, MAP_PRIVATE, fd, 0);
No more string allocations. Everything is a full ref &[u8]. Zero-copy.
Raw speed: 8 million writes/sec.
But not safe. Added checksum + active flag for recovery:
Final: 1.3 million writes/sec, 31μs read on a 612MB file (10 million records).
Dropped from 8M? Yeah. But that's the safety tax. A race car without brakes is useless.
Tradeoffs I Intentionally Made
- No fsync / flush. On purpose. My scope was just speed. If I used fsync, 1.3M would become 10k because you have to wait for the disk to actually spin. Fsync can be added later as
Saku::flush()if needed. - No compaction. Append-only piles up garbage. Compaction needs file rewrite, would drop write speed to 200k. Let it pile up first, benchmark pure write path.
- Key max 256 bytes. Size tradeoff. If you need more, just change that 1 byte to varint too.
Final Numbers
Write: 10,000,000 ops in 7.47s -> 1,337,999 writes/sec
Read: 612.6 MB file, key "example long key 9999999" -> 31μs
CPU: AMD Ryzen 7 5825U, 32GB RAM, Debian 13
Repo: https://github.com/isnakode/saku
make bench N=10000000 to try it.
What I Actually Learned
- AI gives you the map, but you're the one walking in the jungle. Segfaults at 2 AM can't be AI'd.
- Varint, checksum, mmap are not memorization, they are tradeoffs: size vs speed vs safety.
- Limiting to 1 byte is an engineering decision, not laziness.
- I went from
read() read() read()-> ring buffer -> mmap, and watched syscalls drop from thousands to hundreds. You can't learn that from docs, you feel it.
Top comments (0)