DEV Community

Alex E
Alex E

Posted on AI-assisted

From one owner to a replica set: synchronous replication in Squirix preview.8

#dotnet #distributedsystems #csharp #opensource

From one owner to a replica set: synchronous replication in Squirix preview.8

The preview.7 article covered what happens when a node should stop accepting durable work: journal disk quotas with JOURNAL_DISK_QUOTA, and per-principal backpressure keyed by JWT subject. That work made overload explicit on one node.

It does not answer the next uncomfortable question: what happens when the node that owns your key is gone?

Up to preview.7, Squirix durability was per node. A pipelined binary WAL, group commit, snapshot recovery, and durable idempotency outcomes (preview.6) survive a process crash — but only if the same disk comes back. Lose the owner and the key is unavailable until that node returns. No other node can serve it, because no other node has it.

That question shaped 0.1.0-preview.8: replica sets with synchronous replicated mutations and majority commit. RF=1 keeps the old single-owner behavior. RF>=3 survives single-node loss on the remaining majority.

Squirix is still an experimental preview, not a production-ready cache. Its APIs, wire format, and storage formats may change during 0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.

The failure mode to avoid

Imagine a three-node cluster holding a hot key on node A. Node A loses power. The journal on A is intact, but A is unreachable. Clients keep retrying the bootstrap endpoints, and every retry routes to the same answer: the owner is down, the data is on its disk, nobody else may serve it.

The target contract is boring and explicit:

client mutation, operation_id = 4f...
                |
                v
owner appends to its log + replicates to the group
                |
                v
majority durably acknowledges (2 of 3)
                |
                v
owner applies in memory, returns success
                |
                v
owner dies -> remaining majority still holds the committed prefix
                |
                v
a new leader elected by the surviving majority serves the key
Enter fullscreen mode Exit fullscreen mode

No acknowledged replicated write lives on exactly one disk. If the client got success at RF>1, a majority holds it. (RF=1 is the exception that proves the rule: its majority is one node, so a single disk still holds the only copy.) That is the whole point of making replication synchronous on the mutation path rather than shipping logs in the background and hoping.

This is different from the "commit unknown" case in preview.6. There, durable bytes may exist on one node while the client saw only a timeout. Here, the ambiguity spans nodes: the owner may have replicated far enough that the outcome survives its own death. Clients need a stable answer for that case too, which is where the new CommitOutcomeUnknown contract comes in (more below).

From single owner to replica set

The Squirix server already routes each key to one owner with static consistent-hash routing. Preview.8 does not replace that. It widens that ownership into a replica group.

The pieces landed in dependency order across the preview.8 window:

  • Who replicates what: a physical replica ring and topology fingerprint. Key ownership comes from the existing consistent-hash ring, while replicas are the next RF−1 distinct node IDs on a separate sorted ring. The fingerprint hashes the canonical configuration, so a mismatched peer fails visibly instead of diverging silently;
  • One knob, three behaviors: ReplicaCount activation guards. RF=1, RF=2, and RF>=3 get different runtime behavior instead of one blurred mode:
    • RF=1 keeps the previous single-owner path with no replication, elections, or group checks running at all;
    • RF=2 is a synchronous mirror whose majority of two stops writes when either member is lost, with no replacement elected;
    • only RF>=3 gets majority commit, with the surviving majority electing a new leader;
  • Replication stays server-internal: a separate contract with mTLS identity. It lives on the internal listener and is deliberately separate from the shared SquirixCache.proto, so cache clients never see replication RPCs;
  • Followers that can always catch up: an ordered log plus snapshots. Every follower append is journal-backed with no memory-only shortcut, and replica-group snapshots carry idempotency state, so a lagging follower catches up by installation rather than replay alone;
  • Success means a majority has it: a durable commit pipeline. It culminates in synchronous replicated mutations, with RF=1 behavior preserved: the server returns success only after a majority of the group durably acknowledges, and applies to memory only the committed prefix.

What differs in practice is what losing a member costs:

RF=1:  owner only, no replication, no failover
RF=2:  owner + 1 follower, majority = 2, mirror stops on any loss
RF>=3: owner + followers, majority commit, survivors elect a new leader
Enter fullscreen mode Exit fullscreen mode

Static membership is a deliberate constraint. Peers are configured explicitly; there is no dynamic rebalancing in this preview. That removes a whole class of membership races from the first replicated release and lets the safety work focus on election, commit, and leadership change.

The write path: majority before success

For a replicated mutation, the ordering rule extends the preview.6 journal-first rule across nodes:

Do not return success before a majority of the configured group has durably acknowledged the mutation.

The path looks like this:

validate and admit the mutation
            |
            v
reserve a log index in the group
            |
            v
append locally + replicate to followers
            |
            v
wait for durable majority acknowledgements
            |
            v
advance commit_index, apply in memory
            |
            v
return success
Enter fullscreen mode Exit fullscreen mode

Each mutation completes only when covered by that majority-durability step — the same "completed means covered" idea as preview.6 group commit, lifted from one fsync to a quorum of them.

Two details matter in practice:

  • Log-index reservation can be refused when exhausted. That refusal is definitive for the attempt: nothing was appended, nothing was applied. It fails closed before durability, like memory admission and journal quota in earlier previews.
  • Followers validate appends before copying entry payloads and gate tail reads, so a corrupt or out-of-range frame never becomes a committed prefix by accident.

Expiration follows the same pipeline rather than sneaking around it. Expiration candidates get their own operation identity and flow through the commit coordinator, so an expiration the leader observes becomes a committed entry applied identically on every replica — a follower never expires a key on its own.

Write-ahead intent and the honest unknown

Preview.6 already gave every mutation an opaque operation_id with a request fingerprint, propagated across owner-routing hops so a retry on another entry node still deduplicates on the key owner.

Preview.8 tightens the ordering inside that story: the server records a write-ahead idempotency intent before executing the mutation, not after. If the process dies between "accepted" and "applied," recovery still knows the operation existed and can resolve its outcome instead of executing it twice.

Even so, replication adds a case where the server cannot honestly say "committed" or "rejected." The owner replicated to some followers, crashed before learning whether a majority persisted the entry, and the client saw a timeout. The outcome may have committed on the survivors.

That is the new CommitOutcomeUnknown contract. Mutations can now throw CommitOutcomeUnknownException when the durable commit outcome is ambiguous. Retrying with the same operation_id stays idempotent: either the original outcome is returned or the mutation executes exactly once under the same identity.

client --operation_id=4f--> owner (crashes mid-commit)
   |
   +--retry, same operation_id--> new leader
                                      |
                                      +--> original outcome, or exactly-once execution
Enter fullscreen mode Exit fullscreen mode

This is not a distributed transaction, and it is not presented as one. It is the same idempotency idea from preview.6, extended to the window where the owner set changes under the client.

Catch-up: logs first, snapshots when behind

A follower that restarts or falls behind does not get a special path. It gets the same log, from the point it diverged.

The follower log keeps an ordered, durable prefix with explicit term and index on every entry. Catch-up replays from the follower's last durable index: matching prefix is kept, conflicting suffix is replaced, missing suffix is appended. Only when the gap exceeds what the log retains does the leader send a snapshot.

A snapshot is one atomic, checksummed file with the committed state and the record of completed operations, so retries stay idempotent after install.

A follower installs a snapshot only from its own group, and only when the topology fingerprint matches and the configuration generation is compatible. Anything else is refused before a single byte is applied.

Installing replaces the follower's committed prefix. Entries past the snapshot boundary may stay, but only when the follower's entry exactly at the boundary matches the snapshot's last-included index and term. On any divergence at that boundary, the follower's tail is discarded and replayed fresh from the leader log. Compaction afterwards trims only the journal prefix the snapshot covers, keeping the log header and the snapshot itself installable.

Three concrete cases — in all of them the leader holds entries 1 through 6, and a primed entry (like 3') means the same index written in an older term:

leader:   [1][2][3][4][5][6]

Case 1 — small gap, conflicting tail.
The follower is almost current, but its last entry belongs to an older term:
follower: [1][2][3']           -> drop 3', copy 3..6 from the leader
result:   [1][2][3][4][5][6]

Case 2 — gap larger than the retained log.
The follower is so far behind that replaying entry by entry is no longer possible:
follower: [1]                  -> install the snapshot (state through 5),
                                  then replay entry 6 from the leader
result:   snapshot[1..5] + [6]

Case 3 — the snapshot boundary itself was written in another term.
The follower's tail overlaps the snapshot but disagrees with it at the boundary:
follower: [1][2][3][4][5']     -> install the snapshot (state through 5),
                                  discard the conflicting 5', replay 6
result:   snapshot[1..5] + [6]
Enter fullscreen mode Exit fullscreen mode

The idempotency state rides along on purpose. A snapshot with values but without retry identity lets a delayed retry execute twice after old segments are compacted away. Group snapshots carry committed outcomes so catch-up preserves retry semantics, not just data.

What preview.8 changes

The replication work in 0.1.0-preview.8 includes:

  • replica sets RF 1–5 with static membership and a physical replica ring;
  • synchronous replicated mutations with durable majority commit; RF=1 behavior preserved;
  • write-ahead idempotency intent before mutation execution;
  • the CommitOutcomeUnknown contract for ambiguous durable outcomes;
  • ordered follower-log recovery, catch-up, and snapshot installation with idempotency state;
  • a server-only replication gRPC contract on the internal listener with mTLS identity.

Some of that improves availability. Most of it exists because awkward timing now spans nodes: an owner that dies after replicating halfway, a follower that returns with a conflicting suffix, a retry that arrives at a new leader. The happy path — three healthy nodes agreeing — is necessary, but the cases above are where the design is decided.

What preview.8 does not promise

This work makes single-node loss survivable for RF>=3, but Squirix is still early software.

Replication is synchronous and per key group; multi-key operations are not transactions across owners. RF=2 never promotes after peer loss. Quorum reads and leader election details will be covered in part 2; the operator surface (opt-in, metrics, RF=3 compose demo) in part 3 — links below.

The on-disk format may change during 0.x. Synchronous replication is not a substitute for backups or recovery testing. Preview.8 is a stronger foundation for availability, not a production-readiness claim.

Closing thought

Durability work asks: what survives a crash? Replication work asks: what survives the loss of the machine that holds the answer?

The difficult part is keeping the same honesty when the owner changes:

  • success means a majority holds it, not one disk;
  • unknown means unknown, with a stable identity to retry;
  • a follower that returns gets the log it missed, or a snapshot that preserves retry identity — never a guess.

Preview.8 makes those boundaries explicit across nodes. The next article goes one level deeper into how the cluster decides who may speak for a group: terms, votes, fencing, and why RF=2 stays a mirror.

If you run replicated .NET state or care about quorum commit semantics, comments are welcome on where this model still surprises — especially majority-commit latency, CommitOutcomeUnknown handling, and snapshot-vs-log catch-up thresholds.

Links

Squirix docs:

Earlier in this series:

Next (forthcoming):

  • Who is the leader? Elections, fencing, and why RF=2 can't fail over (part 2)
  • Operating a replica set (part 3)

Try it:

dotnet add package squirix --version 0.1.0-preview.8
dotnet add package squirix.server --version 0.1.0-preview.8
Enter fullscreen mode Exit fullscreen mode

Top comments (0)