Distributed consensus is one of those problems that sounds simple until you actually try to build it. "Just get a few servers to agree on something" — how hard can that be?
Pretty hard, it turns out. Hard enough that MIT's graduate distributed systems course (6.824) uses building Raft as its flagship assignment. So I built one myself: RaftKV, a distributed key-value store backed by a from-scratch implementation of the Raft consensus algorithm (Ongaro & Ousterhout, 2014) — the same algorithm behind etcd (which powers Kubernetes), CockroachDB, and Consul.
No consensus libraries. No frameworks. Pure Python standard library, from the ground up.
Why build this instead of using an existing one?
Because using Raft and understanding Raft are completely different skills. I wanted to actually understand:
- How does a cluster agree on who's in charge (leader election) without a central coordinator?
- How do you replicate data safely when nodes can crash mid-write?
- How do you guarantee you never lose committed data, even during a network partition?
The only way to really learn this is to implement it and then try to break it.
The core pieces
Leader election. Every node starts as a follower. If a follower doesn't hear from a leader within a randomized timeout, it becomes a candidate and requests votes from its peers. The randomization matters — if every node timed out at exactly the same moment, you'd get repeated split votes forever. Once a candidate gets votes from a majority, it becomes the leader.
Log replication. The leader is the only node that accepts writes. It appends the write to its own log, then replicates it to followers. Once a majority of nodes have the entry, it's considered committed — safe, permanent, will survive any single node crashing.
Crash-safe persistence. Every node writes its state to disk using atomic write-to-tmp-then-rename (the same pattern I used in an earlier project, Vektr, for its vector index persistence). This guarantees a crash mid-write never corrupts the node's state.
The part that actually matters: does it survive failure?
Anyone can write code that works when nothing goes wrong. The real test of a consensus implementation is whether it stays correct when things fail. So I built a fault-injection test suite that does exactly what a chaos-engineering tool would do to a real production cluster:
Test 1 — Kill the leader mid-operation. Does a new leader get elected? Does any committed data get lost?
Test 2 — Partition the network. Split the cluster into two groups that can't talk to each other. Does the smaller group correctly refuse to accept writes (to prevent split-brain data corruption), while the larger group keeps working?
Test 3 — Crash and restart a node. Does it correctly catch up on everything it missed once it rejoins?
All three passed — on real threads, real sockets, real process kills. Not mocked.
Finding a real bug through testing
Here's the part I'm most proud of. While running the network partition test repeatedly, I noticed something: sometimes the majority side took over 3 seconds to elect a new leader, when it should take milliseconds.
I dug in and found the actual bug: my election-timer loop was calling into the election logic while still holding the node's internal lock — and that election logic made blocking network calls to every peer (up to 500ms each). While a node was busy running its own election, it couldn't respond to incoming votes from other nodes, because responding required the same lock. Two nodes could end up blocking each other.
The fix: release the lock before making any network calls.
The result:
| Metric | Before | After |
|---|---|---|
| Leader re-election time | occasionally 3,000+ ms | 5.9ms average |
| Re-election success rate (8 trials) | 7/8 | 8/8 |
That's a 99.8% reduction in stall time, found not by reading the code carefully, but by actually running the fault-injection tests over and over until the intermittent failure showed up.
The numbers
Benchmarked on a 3-node localhost cluster:
- Write throughput: 38.9 ops/sec, p99 latency 28.6ms (full consensus path: propose → replicate → majority commit)
- Leader re-election: 5.9ms average, 8.5ms max
- Read throughput: 2,195 ops/sec (served locally, no consensus round-trip needed)
One more concrete optimization worth mentioning: I initially had the leader wait for the next periodic heartbeat tick before replicating a new write, rather than replicating immediately. Fixing that alone doubled write throughput — from 19.6 to 38.9 ops/sec — and cut p99 latency from 67.7ms to 28.6ms.
What's next
A few things I'd add if I kept building this:
- Log compaction/snapshotting — right now the log grows unbounded, which is fine for a demo but not for a long-running production cluster
- Read-index or lease-based reads — currently any node can serve a read, which means a follower that's slightly behind on replication could serve a stale value. Production systems solve this properly.
- Membership changes — adding or removing nodes from a live cluster without downtime
Try it yourself
The whole thing is open source, zero dependencies, and includes a live React dashboard that visualizes leader elections and failover in real time — you can literally watch the cluster elect a new leader the moment you kill the old one.
Code: github.com/sameer-sde/raftKv
If you're learning distributed systems, I'd genuinely recommend building your own Raft implementation before reaching for etcd or Consul in a real project. Reading the Raft paper is useful. Debugging your own broken leader election at 2am because you didn't handle a term comparison correctly is how it actually sticks.
Top comments (0)