Inside Raft - The State, the RPCs, and the Rules That Make It Safe

A detour before Part 2 of the CockroachDB series. CockroachDB, etcd (which powers Kubernetes itself), Kafka's newer KRaft mode, and Consul all lean on the same underlying idea - Raft. This post goes past "what Raft does" and into the actual state machine: the variables every node tracks, the exact fields in its two RPCs, and the specific rules that make it provably safe.

The Setup, Quickly

Raft solves consensus: getting a cluster of machines to agree on one identical, ordered log of events, even when machines crash or the network misbehaves. It does this with one leader at a time (all writes go through it), elected by the other nodes via majority vote, with a term number that increases every election and acts as a tie-breaker whenever there's confusion about who's in charge.

That's the 30-second version. Before we open the hood, let's walk through what this actually looks like in practice, in plain language - no variable names yet, just the shape of the thing.

First, a Mental Picture

Imagine five people trying to keep an identical shared notebook, but they're in different rooms and can only pass notes to each other, and any one of them might fall asleep at any moment without warning.

Getting a leader in the first place: Everyone starts out just... waiting to hear from a leader. If a while goes by and nobody's heard anything, one of them gets impatient, declares "I'll be in charge for this round," and shouts out to the other four asking them to back them. If at least three of the five (a majority) say yes, that person is now the leader for this round. If two people get impatient at the exact same moment and both ask for votes, it's possible the vote splits and nobody gets a majority - in which case, everyone just waits again and tries once more with a new round.

Writing something down, once there's a leader: Once someone's in charge, all new entries go through them. The leader writes the new entry in their own notebook, then sends a copy to the other four. As soon as at least two of those four (making a majority of five, including the leader) confirm "got it, written down," the leader considers that entry locked in - permanent, can't be undone. Only then does it tell the person who asked for the write, "done, it's safe."

What happens if the leader vanishes mid-task: Say the leader writes something, gets confirmation from just one other person, and then falls asleep before hearing back from anyone else, and before telling the other three what's going on. Eventually the others notice the silence and hold a new round of "who's in charge now." Here's the reassuring part: whoever does get picked as the new leader is guaranteed - just by the math of majorities - to be someone who already had every entry that was actually confirmed as locked-in before. Nothing that was ever promised as "done" gets lost, even though the room that made that promise is now asleep.

What if the group splits into two rooms that can't hear each other: Three people in one room, two in another, and the wall between them is soundproof. The room of three can still reach a majority among themselves and keep working normally. The room of two can never reach a majority of the whole group of five, no matter how many rounds they try - so they simply can't confirm any new writes until the wall comes down. That's not a bug; it's the system deliberately refusing to let two rooms both think they're "in charge" and start writing conflicting things.

That's the entire idea, no jargon required: one leader at a time, decided by majority vote, writes only count once a majority has a copy, and the group would rather pause part of itself than risk disagreeing with itself.

Now, the vocabulary. Everything above has a name and a very specific implementation in real Raft - this is where it gets genuinely interesting, because the "how do you actually build this so it never breaks" part is where most of the clever engineering lives.

The State Every Node Actually Holds

This is the part most explanations skip, and it's the foundation for everything else. Every single node in a Raft cluster - leader or follower - persistently stores exactly three things:

• currentTerm - the latest term this node has seen, ever. Starts at 0, only ever increases.
• votedFor - which candidate (if any) this node voted for during the current term. This is what stops a node from voting twice in the same election.
• log[] - the actual sequence of entries, where each entry stores the command itself and the term it was created in.

These three are persisted to disk before responding to any RPC - if the node crashes and restarts, it needs to remember exactly this much to rejoin safely. Losing votedFor, for instance, could let a restarted node accidentally vote twice in the same term, which is exactly the kind of bug that breaks the majority-safety guarantee.

On top of that, every node also tracks two small pieces of volatile state (fine to lose on restart):

• commitIndex - the highest log index this node knows to be safely committed.
• lastApplied - the highest log index this node has actually applied to its own state.

And here's a detail that's easy to miss: the leader alone keeps two extra arrays, one entry per follower:

• nextIndex[] - the leader's best guess at the next log entry it needs to send to that specific follower.
• matchIndex[] - the highest entry the leader actually knows is replicated on that follower, confirmed by a reply.

This pair is what makes replication precise instead of a leader just blindly hoping followers are caught up. Every follower can be at a different point in the log, and the leader tracks each one individually.

RPC #1: RequestVote - Field by Field

When a node's election timeout fires, it becomes a candidate and sends RequestVote to everyone else, carrying:

• term - the candidate's new term.
• candidateId - who's asking.
• lastLogIndex and lastLogTerm - a snapshot of how caught-up the candidate's own log is.

That last pair is the interesting part. A receiving node doesn't just vote for whoever asks first - it checks: is the candidate's log at least as up-to-date as mine? Specifically, if the candidate's lastLogTerm is lower than the voter's own last log term, or equal but with a lower lastLogIndex, the voter refuses the vote, even if it hasn't voted for anyone else yet.

This single rule is what prevents a node with an incomplete log from ever becoming leader. Remember, only a majority can elect a leader, and separately, any committed entry is guaranteed to exist on a majority of nodes - so at least one voter in any successful election is guaranteed to already hold every committed entry, and that voter will reject any candidate who doesn't measure up. This is the real mechanism behind the "a new leader always has everything committed" guarantee from the earlier post - it's enforced right here, at vote time, not assumed after the fact.

RPC #2: AppendEntries - Field by Field

This is the leader's replication (and heartbeat) call. Its fields:

• term - leader's current term.
• leaderId - so followers know who to redirect clients to.
• prevLogIndex and prevLogTerm - the index and term of the entry immediately before the new ones being sent.
• entries[] - the actual new log entries (empty for a plain heartbeat).
• leaderCommit - the leader's current commitIndex, so followers know what's safe to apply.

The prevLogIndex/prevLogTerm pair is the single most important detail in this whole post. When a follower receives this RPC, before accepting any new entries, it checks: do I have an entry at prevLogIndex with exactly prevLogTerm? If not, it rejects the whole request.

This check is only possible because of something called the Log Matching Property, which Raft guarantees as an invariant: if two logs contain an entry with the same index and the same term, then every single entry before that point in both logs is guaranteed to be identical too. That's a strong claim - it means a single index+term comparison is a valid proxy for "is your entire history in agreement with mine," without ever needing to compare full histories. That's what makes the consistency check in AppendEntries cheap instead of expensive.

What Happens When Logs Actually Diverge

Say a follower crashed for a while, or was on the losing side of a network partition, and its log has entries the new leader doesn't recognize (left over from an old, now-abandoned leader). Here's the real repair mechanism:

1. The leader initializes nextIndex for that follower optimistically - usually right after its own last log entry.
2. It sends AppendEntries with that prevLogIndex/prevLogTerm. The follower rejects it - no match.
3. The leader decrements nextIndex for that follower by one and tries again, with an earlier prevLogIndex.
4. This repeats until it finds a prevLogIndex/prevLogTerm pair the follower actually agrees with - the last point where the two logs genuinely match.
5. From there, the leader sends everything after that point, and - critically - the follower deletes any conflicting entries it had locally past that point, replacing them with the leader's version.

So a follower's log doesn't get gently patched - it gets forcibly overwritten from the point of disagreement onward. This sounds aggressive, but it's exactly what's needed: any entries that only existed on the old, abandoned leader's log and never reached a majority were never actually committed, so discarding them loses nothing that was ever promised to a client.

(Real implementations usually don't decrement one-by-one for efficiency - the original paper mentions optimizations to jump back faster - but the one-at-a-time version is the easiest to actually reason about, and it's what most explanations, including this one, use as the baseline mental model.)

The Rule That Trips Up Almost Everyone: Committing Across Terms

Here's a subtle safety rule that's easy to get wrong if you're implementing this yourself. You'd think: "once a majority of nodes have an entry, it's committed - done." But Raft adds an extra restriction: a leader can only directly commit an entry from its own current term.

Why? Imagine a leader replicates an entry to a majority, but crashes before marking it committed and telling anyone. A new leader gets elected in a later term - and because of the election safety rule above, this new leader is guaranteed to have that entry in its log. But here's the trap: that new leader cannot safely conclude "a majority has this, so it's committed" purely by term-counting logic, because a future scenario could still overwrite it under specific edge cases the Raft paper walks through in detail.

The safe rule Raft actually uses: a new leader will never directly mark an old-term entry as committed just because a majority holds it. Instead, it waits until it has replicated at least one entry from its own current term to a majority. At that point, because of the Log Matching Property, everything before that new entry - including the old, previously-uncommitted one - becomes safely committed too, as a side effect.

This is genuinely one of the cleverer, less obvious pieces of the algorithm, and it's the kind of detail that separates "I watched a conference talk about Raft" from "I actually read the paper."

Tying the Mechanics Back Together

Put together, here's what actually happens end-to-end for one write, now with the real machinery visible:

1. A client sends a command to the leader.
2. The leader appends it to its own log[], tagged with its currentTerm.
3. It sends AppendEntries to each follower, with prevLogIndex/prevLogTerm pointing at whatever the leader believes that follower's nextIndex position is.
4. Followers check the Log Matching invariant, accept or reject accordingly, and reply.
5. The leader updates matchIndex[] for followers that accepted, and once a majority (self included) have replicated the entry - and at least one entry from the current term has reached that majority - it advances commitIndex.
6. The next AppendEntries (or heartbeat) carries the updated leaderCommit, telling followers it's now safe to apply the entry to their own state.

Every piece we walked through - the persisted votedFor, the prevLogIndex check, the backward-stepping nextIndex repair, the current-term commit rule - exists to make one specific promise airtight: once a client is told "your write succeeded," that write survives any single-node failure, any leader crash, and any network partition, with mathematical certainty rather than best-effort hope.