DEV Community

subhansh
subhansh

Posted on

I Built a Full Raft Consensus Engine in Rust at 17 — Here's Every Gritty Detail

#rust #raft #consensus #distributedsystems

I Built a Full Raft Consensus Engine in Rust at 17 — Here's Every Gritty Detail

I'm 17. This is my first distributed systems project.

Raft is the consensus algorithm that powers etcd, Consul, TiKV, and a dozen other production systems. The paper makes it look straightforward — leader election, log replication, snapshots. But implementing it from scratch? Every paragraph in the paper expands into dozens of edge cases. Figure 8 alone (the "previous term entries can't be committed directly" rule) caused me three rewrites.

This implementation is 4,400+ lines of Rust across 7 crates, 52 tests, real gRPC networking, disk persistence with sled, log compaction via InstallSnapshot, and a chaos harness that kills nodes and partitions networks. No etcd code. No TiKV dependencies. Pure paper-to-production.

What This Actually Is

A complete, tested Raft implementation that you can run right now:

# In-process 3-node demo
cargo run --example kv-demo

# Or run a real cluster across 3 terminals
cargo build --release
# Terminal 1: cargo run --release -p raft-node -- --node-id 1 --peers 2@127.0.0.1:8081,3@127.0.0.1:8082 --port 8080
# Terminal 2: cargo run --release -p raft-node -- --node-id 2 --peers 1@127.0.0.1:8080,3@127.0.0.1:8082 --port 8081
# Terminal 3: cargo run --release -p raft-node -- --node-id 3 --peers 1@127.0.0.1:8080,2@127.0.0.1:8081 --port 8082
Enter fullscreen mode Exit fullscreen mode

What works:

  • Leader election with randomized timeouts (300-500ms) and proper log-up-to-date checks
  • Log replication with Figure 8 compliance — previous-term entries commit indirectly only
  • Fast backup optimization — followers return conflict_term/conflict_index, leader skips entire terms in one RTT
  • Persistence before RPC replies (Figure 2 requirement) — current_term, voted_for, log entries flushed to sled
  • Snapshotting with InstallSnapshot RPC — log compaction, boundary handling, 15 tests
  • Chaos testing — network partitions, node kills, leader isolation, process-level chaos with 5 real OS processes

What's not done (honest gaps):

  • Dynamic membership changes (joint consensus)
  • ReadIndex/LeaseRead for linearizable reads
  • WAN optimization / batching
  • WAL-based persistence (sled works but isn't crash-consistent like a WAL)

Architecture — 7 Crates, Clean Boundaries 

raft/
├── Cargo.toml
├── crates/
│   ├── raft-core/          # Pure logic — 40 unit tests
│   │   ├── types.rs
│   │   ├── state.rs
│   │   ├── election.rs
│   │   ├── replication.rs
│   │   └── snapshot.rs
│   ├── raft-rpc/           # gRPC transport (tonic)
│   ├── raft-storage/       # Persistence abstraction
│   │   ├── RaftStorage trait
│   │   ├── SledStorage
│   │   └── MemStorage
│   ├── raft-node/          # Standalone binary
│   └── raft-chaos/         # Chaos harness
└── examples/
    ├── kv-demo/
    └── process-chaos/
Enter fullscreen mode Exit fullscreen mode

Why this separation matters: The core logic has zero I/O. No network, no disk, no time. Pure functions. Unit tests are deterministic and fast. Storage backends are swappable without touching consensus logic. The chaos harness injects failures at the network layer — not the logic layer.

Leader Election — The Part Everyone Gets Wrong

The paper says: "If election timeout elapses without receiving AppendEntries, become candidate." Simple, right?

Wrong. Here's what actually happens in election.rs:

pub fn on_election_timeout(&mut self) -> Vec<NodeAction> {
    self.become_candidate();  // term++, vote for self

    let args = RequestVoteArgs {
        term: self.current_term,
        candidate_id: self.id,
        last_log_index: self.last_log_index(),
        last_log_term: self.last_log_term(),
    };

    self.peers.iter()
        .map(|&peer| NodeAction::SendRequestVote(peer, args.clone()))
        .collect()
}
Enter fullscreen mode Exit fullscreen mode

But the vote granting logic is where bugs live (§5.4.1):

fn is_log_up_to_date(&self, candidate_last_term: u64, candidate_last_index: u64) -> bool {
    let my_last_term = self.last_log_term();
    let my_last_index = self.last_log_index();

    if candidate_last_term != my_last_term {
        candidate_last_term > my_last_term  // Higher term wins
    } else {
        candidate_last_index >= my_last_index  // Same term? Longer log wins
    }
}
Enter fullscreen mode Exit fullscreen mode

The subtlety that catches everyone: A candidate with term 3 and log length 1 beats a follower with term 2 and log length 100. Term always wins. This prevents the "stale leader with massive log" problem.

My test suite hammers this:

  • 12 election tests covering grants, rejections, stale terms, split votes, term conversion
  • test_no_two_leaders_same_term spins up 5 nodes simultaneously — all become candidates, all vote for themselves, none win (majority = 3, each has 1 vote). Beautiful.

Log Replication — Where Figure 8 Bites You

Leader receives command → appends to local log → sends AppendEntries → majority replicates → commit → apply. Standard stuff.

Except Figure 8. The paper shows a scenario where a leader commits an entry from a previous term, then crashes. New leader doesn't have that entry. Data loss.

The rule: A leader can only directly commit entries from its current term. Previous-term entries commit indirectly when a current-term entry after them commits.

My implementation in replication.rs:

pub fn advance_commit_index(&mut self) {
    if self.state != NodeState::Leader { return; }

    let mut match_indices: Vec<u64> = self.match_index.values().copied().collect();
    match_indices.push(self.last_log_index());
    match_indices.sort_unstable_by(|a, b| b.cmp(a));

    for &n in &match_indices {
        if n > self.commit_index {
            if let Some(entry) = self.get_entry(n) {
                // FIGURE 8 RULE: only commit current-term entries directly
                if entry.term == self.current_term {
                    let replicated = match_indices.iter().filter(|&&idx| idx >= n).count();
                    if self.is_majority(replicated) {
                        self.commit_index = n;
                        break;
                    }
                }
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

The test that proves it works (test_advance_commit_index_figure8):

// Leader has: index 1 (term 1), index 2-3 (term 2)
// Followers replicated up to index 1 (term 1 entry)
leader.match_index.insert(2, 1);
leader.match_index.insert(3, 1);
leader.advance_commit_index();
// commit_index stays 0 — term 1 entry NOT committed directly

// Now followers replicate index 2 (term 2)
leader.match_index.insert(2, 2);
leader.match_index.insert(3, 2);
leader.advance_commit_index();
// commit_index = 2 — term 2 entry commits, drags term 1 along indirectly
Enter fullscreen mode Exit fullscreen mode

This test alone caught 3 bugs in my early implementation. Write the Figure 8 test first. Always.

Fast Backup — The Optimization That Makes Raft Practical

Without fast backup, a follower 10,000 entries behind takes 10,000 round-trips to catch up. The paper's appendix describes the fix: followers return conflict_term and conflict_index on rejection. Leader jumps next_index past the entire conflicting term.

My implementation:

pub fn handle_append_entries_reply(
    &mut self,
    peer: NodeId,
    reply: AppendEntriesReply,
) -> Option<NodeAction> {
    if reply.success {
        // Update match_index, next_index normally
        ...
    } else {
        // FAST BACKUP
        if let Some(conflict_term) = reply.conflict_term {
            let mut new_next = reply.conflict_index.unwrap_or(1);
            for entry in self.log.iter().rev() {
                if entry.term == conflict_term {
                    new_next = entry.index + 1;  // Jump PAST the term
                    break;
                }
            }
            self.next_index.insert(peer, new_next.max(1));
        } else if let Some(conflict_index) = reply.conflict_index {
            self.next_index.insert(peer, conflict_index.max(1));
        } else {
            // Fallback: decrement by 1 (slow path)
            let current = self.next_index.get(&peer).copied().unwrap_or(1);
            self.next_index.insert(peer, (current - 1).max(1));
        }
        ...
    }
}
Enter fullscreen mode Exit fullscreen mode

Test proof (test_fast_backup_convergence):

  • Leader has terms: [1, 1, 3, 5] at indices [1, 2, 3, 4]
  • Follower rejects at index 4, says "I have term 1 at index 4, first index of term 1 is 1"
  • Leader jumps next_index from 5 → 3 (skips all of term 1 in one round-trip)

Real-world impact: A node 1M entries behind catches up in ~log(terms) round-trips, not O(entries). This is the difference between "toy" and "production."

Persistence — The "Before Reply" Rule

Figure 2 is explicit: persist before responding. My grpc_server.rs enforces this:

async fn request_vote(&self, request: Request<ProtoRequestVoteArgs>) -> ... {
    let core_args = proto_to_core_request_vote_args(args);
    let mut node = self.state.node.write().await;
    let reply = node.handle_request_vote(core_args);

    // PERSIST BEFORE REPLY — Figure 2 requirement
    let _ = self.state.storage.save_current_term(node.current_term);
    let _ = self.state.storage.save_voted_for(node.voted_for);

    Ok(Response::new(core_request_vote_reply_to_proto(reply)))
}

async fn append_entries(&self, request: Request<ProtoAppendEntriesArgs>) -> ... {
    let core_args = proto_to_core_append_entries_args(args);
    let mut node = self.state.node.write().await;
    let reply = node.handle_append_entries(core_args);

    // Persist log entries BEFORE responding
    if reply.success {
        for entry in &node.log {
            if entry.index >= core_args.prev_log_index {
                let _ = self.state.storage.save_log_entry(entry);
            }
        }
    }
    let _ = self.state.storage.save_current_term(node.current_term);
    Ok(Response::new(core_append_entries_reply_to_proto(reply)))
}
Enter fullscreen mode Exit fullscreen mode

SledStorage uses bincode serialization with a 64MB cache. Each write flushes to disk. On restart, the node loads current_term, voted_for, and the full log — resumes exactly where it left off.

Crash recovery test: Kill a node mid-replication, restart it, verify it has the same committed state. Works.

Snapshotting — Log Compaction Done Right

Logs grow forever. Snapshots fix this. Figure 12 + Figure 7.

Leader side (state.rs):

pub fn snapshot(&mut self, include_index: u64) -> Option<(Vec<u8>, u64, u64)> {
    // Validations: must be committed, applied, past last snapshot
    if include_index <= self.last_included_index
        || include_index > self.commit_index
        || include_index > self.last_applied {
        return None;
    }

    // Collect all applied commands up to include_index
    let mut data = Vec::new();
    for entry in &self.log {
        if entry.index <= include_index {
            data.extend_from_slice(&entry.command);
            data.push(b'\n');
        }
    }

    let last_included_term = self.get_entry(include_index)
        .map_or(self.last_included_term, |e| e.term);

    // Discard compacted entries
    self.log.retain(|e| e.index > include_index);
    self.last_included_index = include_index;
    self.last_included_term = last_included_term;

    if self.last_applied < include_index {
        self.last_applied = include_index;
    }

    Some((data, include_index, last_included_term))
}
Enter fullscreen mode Exit fullscreen mode

Follower side (handle_install_snapshot):

pub fn handle_install_snapshot(&mut self, args: InstallSnapshotArgs) -> InstallSnapshotReply {
    // Rule 1: Stale term → reject
    if args.term < self.current_term { ... }

    // Rule 2: Higher term → become follower
    if args.term > self.current_term { self.become_follower(args.term); }

    // Rule 3: Snapshot behind our snapshot → reject (stale)
    if args.last_included_index <= self.last_included_index { ... }

    // Rule 4: Install it
    let existing_term = self.get_entry(args.last_included_index).map(|e| e.term);

    // If boundary matches, keep entries after; else discard ALL log
    if existing_term == Some(args.last_included_term) {
        self.log.retain(|e| e.index > args.last_included_index);
    } else {
        self.log.clear();
    }

    self.last_included_index = args.last_included_index;
    self.last_included_term = args.last_included_term;

    // Advance commit/applied past snapshot
    if self.last_applied < args.last_included_index {
        self.last_applied = args.last_included_index;
    }
    if self.commit_index < args.last_included_index {
        self.commit_index = args.last_included_index;
    }

    InstallSnapshotReply {
        term: self.current_term,
        reset_election_timer: true,
        snapshot_data: Some(args.data),
        last_included_index: Some(args.last_included_index),
    }
}
Enter fullscreen mode Exit fullscreen mode

15 snapshot tests cover: basic compaction, stale rejection, uncommitted rejection, unapplied rejection, boundary preservation, empty log after compact, InstallSnapshot basic, stale term/index rejection, log discard, matching boundary kept, mismatched boundary discarded, term_at_index with snapshot, apply after snapshot, multiple snapshots.

Chaos Testing — Breaking It On Purpose

Unit tests verify correctness. Chaos tests verify resilience.

raft-chaos spins up real nodes (in-process or real processes), then:

  • Partitions networks bidirectionally (HashSet<(NodeId, NodeId)>)
  • Kills nodes (clears peers, sets to follower)
  • Heals partitions
  • Verifies safety (no split-brain) and liveness (new leader elected)

The 5 chaos tests:

Test What It Verifies
test_chaos_leader_election 3-node cluster elects leader within 5s
test_chaos_command_submission Commands replicate to all nodes' state machines
test_chaos_node_survives_majority Kill leader → new election in majority partition
test_chaos_partition_minority_cannot_commit 2-node minority partitioned → no commits, majority continues
test_chaos_partition_leader_isolated Isolate leader → majority elects new leader → heal → old leader steps down

Process chaos (examples/process-chaos): Spawns 5 real OS processes via cargo run --release -p raft-node, kills the leader process, verifies new election. This catches bugs the in-process harness misses (file descriptor limits, port binding races, sled locking).

The KV Demo — See It Run 

cargo run --example kv-demo
Enter fullscreen mode Exit fullscreen mode

Output:

=== 3-node Raft cluster started ===
Nodes on ports [9001, 9002, 9003]
Waiting for leader election...
Leader elected: node 2
Submitted: "name Alice" -> index=1, term=1
Submitted: "name Bob" -> index=2, term=1
Submitted: "lang Rust" -> index=3, term=1
Submitted: "lang Python" -> index=4, term=1

=== Cluster state ===
Node 1: state=Follower, term=1, log_len=4, commit=4, applied=4, store={"name": "Bob", "lang": "Python"}
Node 2: state=Leader, term=1, log_len=4, commit=4, applied=4, store={"name": "Bob", "lang": Python"}
Node 3: state=Follower, term=1, log_len=4, commit=4, applied=4, store={"name": "Bob", "lang": "Python"}

All nodes consistent: true
Enter fullscreen mode Exit fullscreen mode

Real consensus. Real replication. Real state machines.

Spec Reference

Built from Ongaro & Ousterhout's "In Search of an Understandable Consensus Algorithm" (2014) — the Raft paper. Every field name, every rule, follows Figure 2 directly.

Test plan follows MIT 6.5840 (Distributed Systems) Raft lab structure — each stage isolates a bug class:

  • Stage A: Leader election
  • Stage B: Log replication
  • Stage C: Persistence + crash recovery
  • Stage D: Log compaction / snapshotting
  • Stage E: Chaos testing

Links

GitHub: https://github.com/subhanshh/raft-rs

Email: me@subhansh.dev

Top comments (0)