Operating a replica set: opt-in, metrics, and an RF=3 Docker demo in Squirix preview.9

Part 1 covered synchronous majority commit and catch-up. Part 2 covered elections, fencing, and why RF=2 cannot fail over.

There is a third question before any of that becomes useful: how does an operator turn it on, tell whether it is working, and check what happens when a leader dies?

That question matters especially after the failover test that stopped the wrong node. Preview.8 had replication, but its election code was not wired into the server. Preview.9 connects it. Three copies alone do not mean automatic failover.

This final part uses 0.1.0-preview.9: explicit switches, persistence and mTLS prerequisites, topology checks, live metrics, and the RF=3 Compose example. Squirix remains experimental software. Its APIs, wire format, and storage formats may change during 0.x.

Three copies are not an HA switch

Setting ReplicaCount to three chooses how many copies hold a key. It does not enable durable storage, grant permission to replicate, or turn on elections.

Replication has its own opt-in: Squirix:Cluster:ReplicationEnabled, default false, or the standalone-host flag --enable-replication. RF>1 without it refuses startup. RF>1 also requires persistence and inter-node mTLS. Automatic failover adds two settings, both off by default: AutomaticFailoverEnabled and QuorumReadsEnabled.

Here is the relevant fragment of a settings file; node identity, peers, and certificate configuration still need to be supplied:

{
  "Squirix": {
    "Cluster": {
      "ReplicaCount": 3,
      "PersistenceEnabled": true,
      "ReplicationEnabled": true,
      "AutomaticFailoverEnabled": true,
      "QuorumReadsEnabled": true
    }
  }
}

Validation requires the last two switches together and refuses them for RF=1 or RF=2. With both off, replication works, but losing a key's owner makes its keys unavailable until that owner returns. With both on, a surviving majority can elect a replacement, and reads confirm authority with a quorum.

Every node must agree on these settings. The failover switches are part of the topology fingerprint and must be chosen when the cluster is created, on empty data directories. They have no CLI flags; --enable-replication alone does not enable failover.

With a complete settings file and the cluster mTLS environment configured, the standalone commands are:

squirix-server validate-config --settings ./Squirix.settings.json --strict
squirix-server run --settings ./Squirix.settings.json --persist --data-dir ./data --enable-replication

Configuration validation catches invalid settings. It does not prove that peers are reachable or that the cluster can elect a leader. That needs a running cluster and a failure test.

Two listeners, two kinds of identity

Application clients authenticate with JWT on the primary HTTPS listener. Replication uses the internal mTLS listener, a cluster CA, and node certificates whose CN matches NodeId. An application's JWT is not a replication identity.

If a node certificate lists extended key usages, both serverAuth and clientAuth must be allowed. Certificate chains are built without network downloads: required intermediates must be in the configured CA bundle or the OS store.

In Compose, host clients use https://localhost:5001, :5002, and :5003. Each maps to the primary listener on container port 5000. Peer URIs use Docker service names and that same primary port, for example https://squirix-node-a:5000.

The transport derives the internal endpoint by substituting the configured mTLS port, 5100 in the example. Actual internode traffic goes there. Putting 5100 in Peers[].Uri instead changes the peer identity the topology expects. Each node's Cluster.Uri must match its own peer entry.

One address lets a host client reach the node, another identifies the peer in the topology, and the internal transport endpoint is derived from it.

A restart does not authorize a new topology

RF>1 activation writes a topology stamp into the data directory. Later starts compare the configured topology with that stamp. A different replica count, peer set, configuration generation, or failover setting is not silently accepted over the existing replica logs.

The fingerprint also includes the minimum cluster package version and replication policy constants. The supported cluster uses one homogeneous build; mixed versions are not a rolling-upgrade strategy. A changed fingerprint can prevent a new version from starting on an old RF>1 directory even after every node has been upgraded.

Before an upgrade, check the release-specific migration instructions and run the target version's doctor against a backup copy of the data, using the intended settings and mTLS environment. Do not edit or delete topology.stamp to make a mismatch disappear: the group logs still belong to the old topology.

For a disposable local demo, fresh volumes provide a fresh start. For data you need to keep, deleting the volumes is data loss, not migration.

Preview.9 specifically changes key placement to XxHash64 and changes journal, manifest, snapshot, and replica-group storage formats. Preview.8 data directories, including RF=1, do not start under preview.9. There is no in-place format migration for this upgrade; use the release's full-cluster restart and empty-directory procedure with a data reload plan.

Start every member before waiting for readiness

The first activation of a fresh replica group needs every configured member, not just a majority. The original owner checks the other copies, offers each a place in the group, and completes its own copy only after all accepted. Until activation completes, the affected keys refuse service and the unseeded copies fail readiness.

Start all members before waiting for them to become ready. An orchestrator that waits for node A's readiness before starting B and C can prevent the initial group from activating. This first-start requirement is different from an activated RF=3 group continuing on two survivors after a failure.

A returning member on an empty disk is unseeded and cannot vote or accept ordinary appends. A surviving leader repairs it with state transfer; its empty copy is not permission to create a competing group.

Readiness answers less than you might expect

The host exposes /health, /health/live, /health/ready, /health/ready/details, and Prometheus metrics at /metrics.

Readiness catches conditions such as unfinished journal recovery, a failed journal pipeline, or a detected ring mismatch. It is not a promise that every replica group can commit. A lagging follower or a growing group log does not, by itself, make /health/ready unhealthy. Journal disk quota can reject writes while readiness stays healthy.

Replica readiness does reject an unready log, an unseeded or incomplete local copy, a topology mismatch, a stale term, or a leader fenced for missing majority contact. A healthy follower does not need leader authority to pass its own readiness check. Readiness is therefore local admission information, not a substitute for a group-level write probe.

The details payload adds replicaGroups: log size, retained entries, snapshot index, and the leader's view of its other copies. Their states are recovering, catching-up, or ready; only ready copies count toward the write quorum. The copy list is empty on followers and before a leader's commit coordinator has started.

For live replication, these are the first metrics I would inspect:

Metric	What it tells you
squirix_replication_role	Per-group role: 0 follower, 1 pre-candidate, 2 candidate, 3 leader awaiting its term commit, 4 leader with authority.
squirix_replication_leader_authority	Whether this node leads the group and holds write authority.
squirix_replication_quorum_available	Whether a ready majority exists. Writes also need leader authority.
squirix_replication_lag_entries	Peer lag according to the leader's latest verified information.
squirix_replication_unknown_commit_total	Commits whose outcome was unknown after the local append.
squirix_replication_state_transfers_total	State-transfer attempts and declined captures, with outcome and reason labels.

Leadership is per group. A node can lead one group and follow another. The leader reports peer lag; a follower does not supply the same peer series. Some gauges are refreshed by the readiness probe, so keep that probe running alongside the metrics scrape.

Even a quorum gauge at 1 deserves context: in RF=3 with one copy down, two ready members can commit, but the group has no remaining failure margin.

Doctor reads the disk; metrics describe the running group

doctor does not contact the cluster to prove a live quorum exists. It reads the configured topology, the data directory's activated stamp, and durable replica metadata, without starting the node. It can also inspect metadata while a node is running.

squirix-server doctor --settings ./Squirix.settings.json --persist --data-dir ./data

Its report includes the fingerprint, configuration generation, and per-group term, commit index, applied index, apply lag, and member state. MISMATCH, UNREADABLE, or MISSING durable state makes it exit with code 1. A never-activated directory can exit successfully: code 0 does not prove a working cluster.

Use it to investigate what a node would start with. Use readiness details, metrics, client operations, and failure tests to investigate what the running cluster can do.

Run the RF=3 example

The source Compose example includes three nodes, named data volumes, per-node settings, persistence, mTLS, and both failover switches. Start with a fresh checkout of the release tag:

git clone --branch v0.1.0-preview.9 --depth 1 https://github.com/squirix/squirix.git squirix-preview9-demo
cd squirix-preview9-demo/docker
docker compose up --build -d
docker compose ps

This is a local development demo. Its JWT keys and mTLS certificates are public test fixtures, and its HTTPS certificate is for development. Replace that material before adapting the layout to another environment.

Probe the primary listener from the host:

curl -k https://localhost:5001/health/ready
curl -k -H "Authorization: Bearer <jwt>" https://localhost:5001/health/ready/details
curl -k -H "Authorization: Bearer <jwt>" https://localhost:5001/metrics

Use a JWT matching the Compose issuer, audience, and signing key. The three basic health routes are anonymous; remote metrics and readiness details require authentication. Host-to-container traffic through a published port counts as remote. The -k flag is only for this development certificate.

Connect a client with multiple bootstrap endpoints and keep reading and writing a known key. Before stopping anything, identify its replica group and current leader. Record that leader and confirm its authority. If the selected group's leader is node A, the failure sequence is:

docker compose kill node-a

Through surviving endpoints, verify that the same key's group acquires a new leader and resumes reads and writes. Read the value acknowledged before the failure, then write and read a new value. A transient refusal during the election is expected; permanent unavailability is not the result this scenario is meant to demonstrate.

Bring the stopped node back without removing its volume:

docker compose start node-a

Watch its catch-up and check that the acknowledged values remain available. It returns as a follower of that group; restarting the original owner does not restore its leadership. If node B or C leads the selected group, substitute that service in both commands.

The key detail is the relationship between the key and the stopped node. Killing an arbitrary node can leave the key's leader alive and demonstrate only follower loss. That was the mistake in the earlier test. This walkthrough describes what to verify; the shell commands alone do not establish that failover worked.

Keep the retry contract intact

Multiple bootstrap endpoints help a client reach a surviving node. They do not replace a majority, and they do not make an interrupted mutation's result obvious.

CommitOutcomeUnknownException means the durable outcome is ambiguous. The operation may have committed. Retry under the same operation_id, within the supported retry window, rather than inventing a new identity and risking a second mutation. Loss of an acknowledgement is not proof that no write happened.

RF=2 remains a synchronous mirror: both members are needed for writes, and losing either does not trigger promotion. RF=3 with failover enabled can continue on two surviving members after an election, provided they can communicate and retain the required durable state. Losing a majority's disks is outside that guarantee.

What this preview leaves to the operator

Membership is static. There is no dynamic membership, automatic rebalancing, or general rolling-upgrade promise. Quotas remain per node. Replication adds network and durable-write latency; quorum reads add their own confirmation round. Backups, capacity planning, and recovery exercises remain necessary.

The series has now asked three questions: do enough nodes hold a write, who may speak for the group, and can the operator deliberately enable and observe it? The last one is where protocol guarantees become something you can check: the settings agree, the data directory matches, the leader has authority, a majority is ready, and the failure exercise stops the node that actually leads the key under test.

Links

In this series:

• Part 1: From one owner to a replica set
• Part 2: Who is the leader?
• The failover test that stopped the wrong node

Release-specific references:

• Squirix on GitHub
• Release v0.1.0-preview.9
• Configuration
• Containerization and Compose
• Diagnostics and metrics
• Operational runbook
• Replication consensus ADR

For the standalone CLI used above:

dotnet tool install --global squirix.server.tool --version 0.1.0-preview.9