The server in charge just crashed. In the Raft authors' own tests, with fixed election timers, the other four servers argued for more than ten seconds about who should take over. Then they added five milliseconds of randomness, and the median wait dropped to 287 milliseconds.
Raft is the algorithm that lets a group of machines agree on one truth, even when two of five of them die. It runs inside etcd, the key-value store Kubernetes keeps its cluster data in. This post walks through how it works, following one example the whole way. The animated version:
One copy isn't enough
A ticket shop is selling the last seat of a sold-out concert: seat 7A. If the shop keeps its bookings on one server, life is simple, until that server dies and the shop closes.
So you keep copies on several servers. Now any one of them can die and the others carry on. But copies bring a new problem: Priya books seat 7A on one server, and a second later Leo books the same seat on another. Both servers said yes. The seat is sold twice. The copies have to agree.
Same commands, same order
Raft's answer starts with a log. Every change is written down as a numbered command, like "seat 7A goes to Priya". Each server keeps its own copy of the log and plays the commands into its own copy of the data, one by one. If every server has the same commands in the same order, every server ends up in exactly the same state. The paper calls this a replicated state machine.
The whole job of consensus is to keep those logs identical while servers crash, restart and lose messages. The trick is majorities: a change only counts once more than half the servers have it. With five servers, that's three, so five servers can lose any two and keep working. That's why five is the typical size.
Who's in charge
Raft elects one leader. Every write goes through it, and log entries only flow from the leader to the others. Each server is a leader, a follower or a candidate. Followers are passive: they answer, unless the leader goes quiet.
The leader sends a steady heartbeat to every follower. Time is split into numbered terms, each starting with an election and having at most one leader. A server that sees a higher term catches up at once, and an old leader that sees one steps down.
An election
Every follower runs an election timer, reset by each heartbeat. When server one, the leader for term 4, dies, the heartbeats stop. Server three's timer runs out first. It moves to term 5, becomes a candidate, votes for itself and asks everyone else for their vote.
Each server gets one vote per term, first come, first served. Servers two and four say yes: three of five, a majority. Server three leads term 5 and starts sending heartbeats. Because every server votes once per term and a win needs three votes, two leaders can never win the same term.
Split votes and the random timer
If two servers time out at the same moment, they can split the vote and nobody reaches three. With identical timers, they collide again and again. That's why fixed timeouts took over ten seconds. The fix: each server picks its timeout at random, from a range like 150–300 ms. One server almost always wakes first and wins before anyone else wakes up.
The timeout needs to be much longer than one round of messages (about 0.5 to 20 ms) and much shorter than the months between server failures.
Committing a write
Priya taps buy. Her request goes to the leader (a follower would just redirect her). The leader appends index 7, term 5: 7A → Priya to its log and sends it to all four followers at once in an AppendEntries message. The same message with no entries is the heartbeat.
Servers two and four write it and say yes. With the leader's own copy, that's three of five, so the entry is committed: it survives even if the leader dies a second later. Priya sees her ticket. A slow server five holds nobody up; the leader keeps retrying until it catches up.
When logs disagree
Crashes leave messes: missing entries, or extra entries that were never committed. So every AppendEntries also carries the index and term of the entry just before the new ones. If the follower doesn't have that exact entry, it refuses. The leader steps back one entry and tries again until the logs agree, and the follower replaces everything after that point with the leader's entries. The leader never changes its own log.
Not just anyone can lead
Imagine the leader crashed the instant Priya's booking committed, and server five, which never got it, asks for votes. If it won, Priya's booking would be wiped out. So Raft adds one rule: a server refuses its vote if its own log is more up to date than the candidate's. Compare the last entries: the later term wins; if the terms match, the longer log wins.
Priya's entry is on three servers, and any candidate needs three votes. Two groups of three out of five always share at least one server, and that server refuses to vote for a log without the booking. So every new leader already has every committed entry.
The leader that didn't know
If the network splits with the leader on the small side, the big side elects a new leader for term 6. For a moment two servers think they're leader, but the old one can only reach two of five, so nothing it writes commits. When the network heals, it sees term 6 and steps down, and its uncommitted entry is overwritten. Bad timing can slow Raft down, but it can't make two servers disagree.
What it costs, and where it runs
Every write waits for a round trip to a majority, and each follower writes the entry to disk before it answers. Every write goes through one leader. Six servers survive no more failures than five. If three of five go down, the cluster stops taking writes rather than risk two different answers. And Raft assumes servers crash or go silent; it doesn't defend against a server that lies on purpose.
Raft was designed to be understandable: in a study of 43 students who learned both, 33 scored better on Raft than on Paxos. That clarity helped it spread: etcd, HashiCorp's Consul and TiKV all run on it.
Takeaways
- Same commands in the same order give the same state on every server.
- A write counts once a majority holds it: 3 of 5 survives two failures; use odd cluster sizes.
- Elect one leader on a random timer; one vote per server per term.
- Only an up-to-date server can win, so every leader has every committed entry.
- Safety never depends on timing.
Watch the full animated breakdown: https://youtu.be/J_qzUMR3JbA
Sources: Diego Ongaro and John Ousterhout, In Search of an Understandable Consensus Algorithm (Extended Version), https://raft.github.io/raft.pdf · https://raft.github.io/ · etcd FAQ · Kubernetes architecture docs · HashiCorp Consul consensus docs.
Practice designing systems like this yourself at https://systemdesignlab.in
Top comments (0)