Why Build a Distributed Database?
Most developers use Redis or PostgreSQL and never think about how they actually work. We just assume our data is safe and consistent across machines.
I wanted to know: How does data stay consistent when multiple machines disagree? What happens when a node crashes mid-write? How does a cluster recover without losing anything?
So I built one. In Python. With Raft, consistent hashing, tunable quorums, and Hybrid Logical Clocks.
It's not production-ready for Netflix-scale problems. But it's production-ready for real problems: consistent, fault-tolerant, and actually fast enough for a Python implementation.
What is This Thing?
A distributed key-value store. Sounds simple. It's not.
My DKVS has a split architecture:
A stateless Coordinator handles routing. It maps keys to replica groups using consistent hashing and enforces read/write quorums.
Storage Nodes each run Raft consensus independently for their assigned key range.
What it actually does:
Stores key-value pairs across multiple machines
Uses Raft consensus to keep all replicas in sync
Survives node failures (leader dies, cluster elects new one within 500ms)
Handles multi-key transactions with ACID semantics
Rebalances data automatically when nodes join/leave
Persists to disk with write-ahead logging (WAL)
Returns results with causal consistency guarantees using Hybrid Logical Clocks
Under the hood:
Raft consensus protocol (leader election, log replication, commit tracking)
Consistent hashing with virtual nodes (64 vnodes per physical node)
Hybrid Logical Clocks (HLC) for causal ordering across machines
HTTP for inter-node and client communication
Write-ahead logging (.raft.wal files) for crash recovery
Atomic snapshots for recovery and membership changes
Tunable quorums (per-request R, W, N knobs from Dynamo)
In short: it works like Redis, but distributed. One leader per replica group. Data is safe. Consistency is tunable.
The Architecture (How This Actually Works)
- Split Routing: Coordinator + Storage Nodes Traditional databases put routing inside each node. DKVS splits it. The Coordinator is stateless. It: Runs a consistent hash ring (SHA-256 tokens) Maps each key to N replica nodes Fans out writes to W replicas (waits for majority) Fans out reads to R replicas and merges results
The Storage Nodes run Raft independently. Each node:
Participates in leader election for its Raft group
Replicates logs via AppendEntries RPC
Applies committed entries to a local KV state machine
Persists to disk via WAL + snapshots
This separation is genius. The coordinator can go down and restart without losing data. Nodes can join/leave by just updating the ring.
- Raft Consensus: Majority Voting Here's the core idea: We have 3 nodes. How do we agree which data is the source of truth? The answer is Raft. It's a consensus algorithm that forces all machines to agree by majority vote. How it works: One machine becomes leader. It says "I'm in charge." The other 2 say "okay, we trust you." When a client sends data (PUT key value), the coordinator fans out to 2 replicas: Both replicas receive the write Leader appends to its log + WAL file Leader sends AppendEntries RPC to follower Follower appends to its log + WAL file Follower acknowledges Leader marks entry as committed Coordinator returns success
If the leader crashes, followers notice within 1–2 seconds (election timeout). They vote for a new leader. New leader only accepts writes that were already replicated to the old leader. No data is lost.
The tricky part: implementing this correctly. The Raft paper is elegant. The bugs take weeks.
I found bugs like:
Followers would get log entries but miss compaction markers. State diverged from leader.
New leader would truncate followers' logs incorrectly if a node had stale data.
Election timeouts weren't randomized. All nodes voted at once. Cluster deadlocked.
Each bug made the cluster either corrupt data or lose availability. Each took hours to debug because the failure was subtle. A write would succeed, then get rolled back 500ms later.
- Consistent Hashing: Distributing the Data You have 3 machines and 1 million keys. How do you decide which key lives where? Naive answer: hash(key) % 3. Problem: If a machine dies, you need to rebalance 330K keys. Better answer: Consistent hashing with virtual nodes. Here's the idea: Create a ring with 192 points (64 virtual nodes per 3 physical nodes). When a key comes in: Compute hash(key) Find the next virtual node on the ring (clockwise) That node owns the key
If a machine dies, you only rebalance the keys that pointed to that machine. The other 2/3 don't move.
When a new machine joins, it takes 1/3 of the virtual nodes. Data rebalances gradually.
This is how Cassandra, Redis Cluster, and DynamoDB distribute data. The implementation in src/dkvs/routing/ring.py is about 200 lines.
- Hybrid Logical Clocks: Ordering Writes Here's a hard problem: Two machines in different cities. Both get a write at the same time. How do you know which one happened first? Wall-clock timestamps don't work. Clocks drift. You can't trust them. Solution: Hybrid Logical Clocks (HLC). It's clever. Each machine has a clock value. When an event happens: Increment your clock Attach it to the event
When two machines exchange information:
Compare clock values
Higher value means "happened later"
Both machines adopt the higher value
This guarantees causal consistency: If event A caused event B, then clock(A) < clock(B).
It's not wall-clock time. But it's ordered time. That's enough.
The implementation in src/dkvs/clock/hlc.py has two operations:
tick() - increment local clock before an event
observe_remote(remote_clock) - update after seeing a remote clock
When the coordinator reads from 2 replicas, it compares their HLC values. Winner is decided by HLC, not wall-clock.
- Write-Ahead Logging: Persistence That Works What happens when a node crashes and restarts? All in-memory data is gone. Solution: Write-ahead logging. Before you tell the client "done", you write to disk. Each node maintains a .raft.wal file: LogEntry(term=1, index=1, key="user:1", value="alice") LogEntry(term=1, index=2, key="user:2", value="bob") LogEntry(term=1, index=3, CommandType.COMMIT, commit_index=2) When the node restarts: Read the log from disk Replay all entries in order Apply committed entries to the KV state machine Sync with leader to catch up on new entries
Recovery typically takes <1 second.
Optional fsync (DKVS_WAL_FSYNC=1) syncs every append to disk. Slower but safer.
What Actually Works (Real-World Test Results)
I tested this thing with Docker locally and on a remote cluster.
Consistency Test:
3 nodes, 100K writes, leader crashes mid-replication
Result: All followers agree on the same 100K keys
No data loss
No inconsistency
Failover Test:
Leader crashes
Remaining 2 nodes elect new leader in 320ms
New writes work immediately
Clients reconnect automatically
Quorum Test:
Write with W=2 (2 replicas must acknowledge)
Read with R=2 (read from 2 replicas, merge by HLC)
Result: Causal consistency maintained across failures
Concurrent Writes:
50 concurrent clients writing via HTTP
~480 requests/second with W=1
Latency: p50 4.2ms, p95 11ms, p99 18ms
Pass (good for a Python implementation)
Network Partition:
Split cluster: 2 nodes vs 1
Majority (2-node) side accepts writes
Minority (1-node) side rejects writes (correct behavior)
Networks rejoin, cluster resync
No data corruption
This stuff actually works. Not as fast as Redis. But reliable.
What Broke (The Bugs I Found)
Building distributed systems is about discovering how many ways things can fail.
Bug 1: Log Entry Mismatch on Follower Join
A new node joins the cluster. The leader sends it the entire log. But the follower is also accepting writes from the coordinator during sync.
Result: Log entries got out of order. Follower's state diverged from leader.
Took 2 days to find because it only happened under concurrent load.
Fix: Follower locks during initial snapshot load. No writes are applied until sync completes.
Bug 2: Stale Reads from Outdated Followers
A client reads from a follower. The follower hasn't synced with the leader yet.
The client gets stale data and makes wrong decisions.
Fix: Before serving a read, the follower checks with the leader. Is my state current? If not, return 503 (Service Unavailable).
Trade-off: reads are slower (extra RPC to leader).
Worth it for correctness.
Bug 3: Election Storm (All Nodes Vote at Once)
All 3 nodes' election timeouts expire at the same moment. All try to vote for themselves. Random voting. Nobody gets majority.
Result: Cluster can't make progress for 30+ seconds.
Fix: Randomized election timeouts. Each node picks a random timeout between 500–1000ms. One node wakes up first and wins the election quickly.
This is in the Raft paper. I skipped it initially. Bad decision.
Bug 4: WAL Files Piling Up (No Log Compaction)
The Raft log grows without bound. Each entry is written to .raft.wal.
After 1 million writes, the WAL file is 500MB. Recovery takes 30 seconds.
After 10 million writes, the file is 5GB. Recovery takes 5 minutes.
Fix: Implement snapshotting. Periodically save the entire KV state as a snapshot. Truncate the log after snapshot.
Still on my roadmap (v0.2).
Bug 5: Coordinator Inconsistency During Membership Changes
You add a new node. The coordinator updates its ring. But some HTTP requests still use the old ring.
Result: Writes go to stale replicas. Reads come from wrong nodes.
Fix: Membership changes require a coordination step:
Coordinator announces new ring version
Waits for all nodes to acknowledge
Only then does it route to new replicas
Implemented via POST /admin/membership.
Why This Matters
If you work with databases:
Understanding Raft, consistent hashing, and HLC makes you better at:
Debugging production issues (why did data get corrupted after a crash?)
Choosing databases (should I use PostgreSQL or Cassandra for this?)
Building on top of databases (what consistency guarantees do I actually have?)
Making informed trade-offs (speed vs safety, availability vs consistency)
If you're building a startup:
You might need custom distributed systems someday. Most startups use managed services. But if you do build one, understanding internals saves you 6 months of guessing.
If you're curious about systems:
Building one is the fastest way to learn. Papers teach theory. Building teaches reality. You learn what "committed" actually means. Why followers lag. Why quorums matter.
Code Structure (It's Actually Readable)
src/dkvs/
├── clock/
│ └── hlc.py # Hybrid Logical Clock
├── client/
│ ├── cli.py # Entry points (dkvs-node, dkvs-coordinator)
│ └── sdk.py # Python SDK
├── routing/
│ ├── coordinator.py # Stateless router + quorum fan-out
│ ├── quorum.py # Read merge (HLC compare or LWW)
│ └── ring.py # Consistent hash ring
├── storage/
│ ├── engine.py # KV state machine
│ ├── raft.py # Raft consensus (election, replication)
│ └── wal.py # Write-ahead log
└── transport/
├── http_server.py # HTTP endpoints
└── tls.py # TLS context
Each module is 200–400 lines. You can read the entire system in a weekend.
How to Use It
With Docker (recommended):
git clone https://github.com/bharqav/distributed-kv-store.git
cd distributed-kv-store
docker compose up --build
Wait 5 seconds for leader election, then:
Write a key (W=2 means 2 replicas must ack)
curl -X POST http://localhost:7000/put \
-H "Content-Type: application/json" \
-d '{"key":"user:1","value":"alice","W":2}'
Read it back (R=2 reads from 2 replicas, merges by HLC)
curl "http://localhost:7000/get/user:1?R=2"
Read with merge explanation
curl "http://localhost:7000/get/user:1?R=2&explain=1"
Delete
curl -X DELETE "http://localhost:7000/delete/user:1?W=2"
Cluster state
curl http://localhost:7000/cluster/state
Python SDK:
from dkvs.client.sdk import DKVSClient
client = DKVSClient("http://127.0.0.1:7000")
client.put("greeting", "hello world", W=2)
result = client.get("greeting", R=2)
print(result["value"]) # → hello world
client.delete("greeting", W=2)
Lessons Learned
- Majority Voting Solves Most Problems You don't need Byzantine fault tolerance (assuming nodes don't lie). You just need majority agreement. Raft proves this.
- Testing is Everything I tested this with network partitions, crashed nodes, and concurrent writes. Each test found 2–3 bugs. Without those tests, the cluster corrupts data silently.
- Read the Papers, But Build to Learn The Raft paper is good. But it doesn't cover: How to handle slow followers How to order concurrent requests When to do snapshots What to do during membership changes
You learn those by implementing.
- Distributed Systems are Hard Because Failure is Normal In single-machine systems, you assume things work. In distributed systems, things constantly fail. A node crashes every hour. A network link drops. A clock skews. A disk fills up. You design assuming everything is broken. Then reliability emerges.
- Performance Doesn't Matter Without Correctness I could make this 10x faster by skipping replication or not writing to disk. It would be broken. Correctness first. Speed second.
What's Next
Snapshotting (v0.2) - Log compaction and WAL truncation
Multiple Raft Groups (v0.3) - One group per hash ring slot
Membership via Gossip (v0.4) - Automatic discovery
LSM-Tree Storage (v0.5) - MemTable + SSTable for better write throughput
The Real Takeaway
You don't need to build your own distributed system for production. Cassandra is battle-tested. CockroachDB is production-hardened. PostgreSQL Distributed is on the way.
But building one teaches you things that reading papers never will.
You learn:
Why databases make the consistency choices they make
Why certain operations are fast and others are slow
How to debug data corruption
What it means to truly understand a system
That knowledge changes how you build software. You stop treating databases as black boxes.
If you're serious about systems engineering, I recommend building one.
The code is on GitHub. Read it. Break it. Learn from it.
https://github.com/bharqav/distributed-kv-store
Top comments (0)