A month ago I wrote a post-mortem about a failover test that stopped the wrong node: preview.8 shipped election code that nothing in the server called. I ended it with a promise. The article about elections would wait until elections actually run.
In 0.1.0-preview.9 they run. Stop the leader of an RF=3 group and the other two elect a new one, the client keeps working, and the leader-stop tests kill the node that matters. This is that article.
Part 1 covered the replicated write path: synchronous mutations, majority commit, write-ahead idempotency intent, and the honest CommitOutcomeUnknown answer when the owner dies mid-commit. All of that assumes something it never proves: that exactly one node speaks for a replica group at a time.
Without that guarantee, two survivors of a partition can both believe they lead, accept different writes for the same key, and return success for histories that can never be reconciled. Majority commit on top of two leaders is just durable disagreement.
Squirix is still an experimental preview, not a production-ready cache. Its APIs, wire format, and storage formats may change during 0.x. This article describes the current design, not a compatibility promise.
Opt-in, and only for three or more
Automatic failover is off by default. Two settings under Squirix:Cluster turn it on, and they come together:
{
"Squirix": {
"Cluster": {
"ReplicaCount": 3,
"ReplicationEnabled": true,
"AutomaticFailoverEnabled": true,
"QuorumReadsEnabled": true
}
}
}
Validation refuses one without the other, and refuses both for groups of one or two replicas. Every node needs the same values, and they are set on empty data directories: the switches are part of the topology fingerprint.
With failover off, nothing changes from preview.8: the owner of each key leads its group in term one, reads are local, and losing the owner makes its keys Unavailable until it comes back. Everything below is about the switched-on mode.
The failure mode to avoid
Imagine five nodes split 3–2 by a network cut. The minority side bumps its own term number and declares itself leader. It accepts a write for key K. Meanwhile the majority side elects a real leader and accepts a different write for K. The partition heals. Which write won? Both clients got success, and the log has a fork nothing can merge.
The contract that prevents it borrows the oldest Raft lesson:
no node leads on its own authority
|
v
a candidate wins only with a majority of the configured static membership
|
v
votes come only from members whose log can vote, but that never shrinks the quorum
|
v
a leader that stops hearing from a majority steps down
A lone node can inflate any number it likes. Without a majority behind it, the number means nothing.
When an election starts
A follower campaigns when it has heard neither from a leader nor granted a vote for one election timeout: one second plus up to one second of seeded jitter. The leader sends a heartbeat to every idle follower every 100 ms, so a live leader keeps everyone quiet. The timer is per member, but the group's detection is the earliest timer among the survivors, so a larger group notices a dead leader sooner: RF=5 trims the mean by about 0.13 s against RF=3 and cuts the tail beyond two seconds roughly eightfold.
A campaign starts with a pre-vote for the next term, which changes no term anywhere. A member refuses it with leader-contact if it heard from a live leader within the last election timeout. That one rule matters more than it looks: a node cut off from the leader alone would otherwise keep raising the term and depose a perfectly healthy leader every time it reconnects. With the rule, it campaigns in vain until it can hear the leader again.
Only with a majority of pre-votes does the candidate persist the new term and its own vote, then ask for real votes. Each vote request has a 250 ms timeout, and a round ends as soon as its outcome is decided, so a dead member does not slow a round that already has its majority. That early end made promotion with a silent old leader much faster in this release: about 0.3 seconds instead of 2.3.
Term one always belongs to the owner of a group, which leads it without votes after the group's first start. The first election is for term two.
Terms that survive the process
Every replica group stores a durable current_term and at most one voted_for per term, in both slots of its metadata file. A vote is persisted before it is sent — the same journal-first instinct as preview.6, applied to elections. A node that crashes after voting remembers, after restart, who it promised the term to. Double votes across restarts are not a matter of discipline; the disk forbids them.
Voting adds a second gate: log freshness. The voter compares the candidate's last log term, then its last log index, with its own, and refuses a stale candidate with stale-log. A new leader that never saw a committed entry cannot win, so it cannot overwrite that entry — the classic "elected but amnesiac" failure.
candidate (lastTerm=4, lastIndex=120) --> voter (lastTerm=5, ...)
|
+--> refuse: stale-log
Some members cannot vote at all. A copy that came back on an empty disk is unseeded: it grants no vote and takes no entries until the group repairs it. A copy whose log is not ready refuses every ballot. A copy waiting for a state transfer still votes, because its log is real, but it can never stand for election.
Authority must be earned, then kept
Winning an election is a moment. Serving traffic is a stream.
A winner does not serve anything yet. It first appends a leader-noop record in its own term and waits until a majority commits it. Until then every write is refused before anything is appended, with Unavailable and "nothing was read or written". Only that current-term commit lets the leader treat older entries as committed. This is Raft's leader-completeness rule stated as a startup step, and it closes the window where a new leader serves history it does not yet own.
Keeping authority is the other half. A leader steps down once fewer than a majority, itself included, answered within one election timeout, and at once when any reply carries a higher term. A deposed leader's messages carry a stale term, and followers refuse them with stale-term instead of merging them.
Some leaders are alive but useless: one that cannot get its own term committed, one whose apply loop is stuck, one whose own log appends keep failing. Each of those gives the group up after 30 seconds so another member can take it. A stopping leader hands the group over before it drains, instead of letting it wait out an election timeout.
Reads that refuse to guess
With quorum reads on, a read on the leader goes through five steps:
1. check authority in the current term
2. read_index = commit_index
3. confirm with a majority in that term (requests sent after step 2)
4. wait until applied_index >= read_index
5. return the value
A minority side or a former leader cannot pass step 1 or step 3. It answers Unavailable (read_quorum_unconfirmed, read_index_unapplied) or a stale-term refusal — never an old value presented as current. There are no time-based lease reads: authority without a quorum is never enough, even when clocks agree.
Finding the leader
A client should not have to know who leads. Any node accepts a single-key call, including a node outside the key's group, and forwards it to the leader it knows. If the answer is stale-owner or stale-term, it reroutes once to the leader the answer names. All the leader waits of one call share a three-second attempt budget, and the client's retries share one 15-second deadline. Without a leader the call fails with Unavailable and nothing is written, so a retry with the same operation_id is safe.
client --> any node --> leader it knows (term 7, current is 9)
|
+--> stale-term: one reroute to the named leader, then fail visibly
Bounded matters. "Try the next node forever" turns every leadership change into a thundering herd from the clients.
Every replication RPC also carries the term, the topology fingerprint and the configuration generation in its header. A message from another world — wrong group, older configuration, a node below the minimum cluster version — is refused before it touches the log.
Why RF=2 stays a mirror
The same majority rule that saves RF>=3 condemns RF=2 to honesty.
With two members the majority is two. Lose either node and the survivor holds one vote out of two. It cannot elect itself, cannot commit, cannot promote. That is not a missing feature; it is the protocol refusing to invent authority it does not have. Validation does not even let you switch failover on for RF=2.
RF=2, both alive: synchronous mirror, every write on both nodes
RF=2, one lost: writes stop, committed data stays readable locally, no promotion
RF>=3, one lost: with failover on, the other two elect a leader and keep serving
Two-node majority systems that promote on peer loss are single-node systems with extra steps — and extra data loss.
Coming back
A node that returns after an outage catches up from the leader's log. If the leader's log no longer holds what it needs — it was compacted, or the node came back on an empty disk — the leader sends a state transfer: its snapshot plus every cache entry of the group. A wiped copy also forgot whom it voted for, so before repairing it the leader confirms with a majority of the other members that no newer term exists. That check makes the forgotten vote harmless.
A returning node does not take the group back. It stays a follower: its peers refuse its pre-vote while they hear the current leader, and nobody votes for a stale log. Membership is not readiness to lead.
A model that is allowed to say no
Before any election code landed, I froze the protocol in an ADR with an executable C# model (src/squirix.protocol-model, never shipped, never referenced by product code). It has immutable state, pure transitions, a deterministic BFS explorer, and invariants that report minimal counterexample traces.
The explorer covers elections, commit, and quorum reads for RF=2 through RF=5, with crash points before and after every durable write of term, vote, log, and commit index, and with loss, duplication, reorder, and one-way isolation of a node. Newer profiles add disk loss and repair. A profile that hits its state cap without a violation exits differently from a clean fixed point, so residual risk stays visible instead of silently passing.
Just as important are the broken variants that must produce counterexamples:
- grant a second vote in the same term;
- compare log length before the last term when voting;
- commit an old-term entry without a current-term commit;
- serve a read without a majority confirm, or before apply catches up;
- repair a wiped copy without the term ceiling.
If the explorer ever stops finding those violations, the harness is broken, not the protocol sound. Conformance tests run real product components and ask the model whether it can reach the same trace.
The post-mortem was about the gap between this model and the server. The model says the protocol is safe; it never said the server runs it. What closes the gap now are the E2E tests: an RF=3 and an RF=5 cluster whose leader is stopped, a leader isolated by a partition, a five-node 2/3 split with the leader in the minority, a wiped follower repaired by state transfer, and a former leader that restarts and stays a follower. The tests that stop a node assert that it was the leader.
What preview.9 does not promise
Membership is static: no dynamic membership, rebalancing, or joint consensus. Election timings are fixed per member; the group's detection still shortens with more replicas. The model covers single-node isolation; the five-node split is covered by a test, not by the model, and richer partitions remain residual risk. A group that lost the disks of a majority of its members at once is not repaired.
Closing thought
Commit protocols ask: do enough nodes hold this write? Election protocols ask the harder question first: who is allowed to ask?
The hard part is refusing authority precisely:
- no majority, no leadership — even if the node feels healthy;
- stale log, no votes — even with supporters;
- no current-term commit, no serving old history;
- no quorum confirm, no current read;
- a live leader, no pre-vote for anyone else.
Preview.9 makes each refusal durable, fenced, and observable — and, this time, switched on. The final article in this series covers what operators see: the opt-in, metrics, readiness, and an RF=3 compose example you can run.
If you build leader election or quorum reads in .NET, comments are welcome on where you draw the same lines — especially vote persistence, pre-vote, and rerouting budgets.
Links
Squirix docs:
- GitHub
- Release
v0.1.0-preview.9 - Replication consensus ADR
- Clustering
- Configuration (automatic failover)
- Consistency
In this series:
- Part 1, from one owner to a replica set
- The failover test that stopped the wrong node
- Part 3, operating a replica set (forthcoming)
Try it:
dotnet add package squirix --version 0.1.0-preview.9
dotnet add package squirix.server --version 0.1.0-preview.9
dotnet tool install --global squirix.server.tool --version 0.1.0-preview.9
Top comments (4)
I took the rerouting-budget invitation literally, because the post quotes 0.3 s and 2.3 s while the comment on
ElectionTimerOptionssays a failover "waits one to two seconds for the timeout and the jitter before it campaigns". Those are two different intervals, and only the second one is what a client sees.Detection is not an average follower's timeout, it is the earliest: whichever survivor's
ElectionTimeout + jitter.Next(MaxJitter)fires first campaigns, and with the leader dead the others grant its pre-vote, since none of them heard a live leader inside the last election timeout. With your defaults (1 s plus uniform [0, 1] s, baseline reset by a 100 ms heartbeat), detection is the min over the survivors, and campaign-to-serving is the 0.3 s you measured:Two things fall out that I did not see stated:
LeaderWaitTimeout = min(ElectionTimeout + MaxJitter, PerAttempt - LeaderRefusalMargin) = min(2.000, 2.000). The cap does not bind at your defaults, so it clips nothing today - but it means it starts clipping the leader wait silently the moment anyone raises either timeout. And the slowest plausible detection is 2.1 s (1 s + 1 s jitter + the heartbeat baseline), so the 12.3% in the table are the failovers where that wait expires before a promotion can answer and the call spends a second attempt. That is the margin doing precisely what its comment says it is for; it just had no rate attached to it.Caveats, since the arithmetic is mine and the cluster is yours: I did not run it. I reproduced the constants from
cc10c8cand combined them with your measured 0.3 s, ignoring RTTs. If that 0.3 s already includes detection rather than covering campaign-to-serving, every row shifts down by the detection term and only the tail comparison in (1) survives.Thanks, this is the kind of comment the invitation was for. Checked against the code: your constants match (1 s timeout, up to 1 s jitter, 100 ms heartbeat, leader wait 2 s), and the 0.3 s in the post is campaign to promotion only, so your rows stand as written. A client sees the earliest survivor's timer plus those 0.3 s.
One detail that cuts the other way from what one might hope: a graceful stop does not skip detection. The stopping leader goes silent at the start of its shutdown instead of after the drain, but it tells nobody to campaign, so the survivors still wait out their timers. Your table covers both the kill and the stop.
The RF point is right and was missing from the post: "fixed" is true of a member's timer, and the group's detection is a min over them. I'll add that.
Good catch on the graceful stop, and the code says why it costs the full timer:
LeaderExithas no member for a planned exit (None,Deposed,AuthorityTimedOut,ApplyTimedOut,AppendTimedOut), and the stop path ends inRetireAsync→_leadership.RetireAsync(...), which retires leadership locally. So on the survivors' side a clean stop and a kill are the same event — silence — and the enum is the evidence that nothing was ever on that path to tell them otherwise.That makes the planned case the one where the timer is pure waste, and it is worth putting a size on it. Detection is 1.382 s of the 1.682 s client-visible mean at RF=3, so 82% of a planned failover is waiting for a node that already said goodbye (RF=5: 1.248 of 1.548, 81%). A three-node rolling upgrade pays ~5.0 s of that wait to fail over three times, against ~0.9 s if each stop could hand the timer over with the leadership.
Two things I would not skip before building it, though:
Your model is the reason I believe (2) is cheap to check and (1) is not; the round-count effect of a split vote is probably easier to measure in the E2E tests than to reason about.
You read the stop path right: RetireAsync retires locally and nobody downstream hears a planned exit. Filed it: github.com/squirix/squirix/issues/...
Your two conditions are in the issue as written. One more thing the hint has to carry: pre-vote. A member refuses a pre-vote with leader-contact while it heard the leader within the last election timeout, and the survivors heard it a heartbeat ago, so a plain "campaign now" to the successor loses its pre-vote and waits out the timer anyway. The successor's pre-vote has to say the leader asked, and voters have to grant it despite recent contact. That bypass is the only guard the transfer adds, which is also why your broken variant is the right test: a hint outside a retire must yield a counterexample.