DEV Community: Alex E

From one owner to a replica set: synchronous replication in Squirix preview.8

Alex E — Tue, 15 Sep 2026 06:56:02 +0000

From one owner to a replica set: synchronous replication in Squirix preview.8

The preview.7 article covered what happens when a node should stop accepting durable work: journal disk quotas with JOURNAL_DISK_QUOTA, and per-principal backpressure keyed by JWT subject. That work made overload explicit on one node.

It does not answer the next uncomfortable question: what happens when the node that owns your key is gone?

Up to preview.7, Squirix durability was per node. A pipelined binary WAL, group commit, snapshot recovery, and durable idempotency outcomes (preview.6) survive a process crash — but only if the same disk comes back. Lose the owner and the key is unavailable until that node returns. No other node can serve it, because no other node has it.

That question shaped 0.1.0-preview.8: replica sets with synchronous replicated mutations and majority commit. RF=1 keeps the old single-owner behavior. RF>=3 survives single-node loss on the remaining majority.

Squirix is still an experimental preview, not a production-ready cache. Its APIs, wire format, and storage formats may change during 0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.

The failure mode to avoid

Imagine a three-node cluster holding a hot key on node A. Node A loses power. The journal on A is intact, but A is unreachable. Clients keep retrying the bootstrap endpoints, and every retry routes to the same answer: the owner is down, the data is on its disk, nobody else may serve it.

The target contract is boring and explicit:

client mutation, operation_id = 4f...
                |
                v
owner appends to its log + replicates to the group
                |
                v
majority durably acknowledges (2 of 3)
                |
                v
owner applies in memory, returns success
                |
                v
owner dies -> remaining majority still holds the committed prefix
                |
                v
a new leader elected by the surviving majority serves the key

No acknowledged replicated write lives on exactly one disk. If the client got success at RF>1, a majority holds it. (RF=1 is the exception that proves the rule: its majority is one node, so a single disk still holds the only copy.) That is the whole point of making replication synchronous on the mutation path rather than shipping logs in the background and hoping.

This is different from the "commit unknown" case in preview.6. There, durable bytes may exist on one node while the client saw only a timeout. Here, the ambiguity spans nodes: the owner may have replicated far enough that the outcome survives its own death. Clients need a stable answer for that case too, which is where the new CommitOutcomeUnknown contract comes in (more below).

From single owner to replica set

The Squirix server already routes each key to one owner with static consistent-hash routing. Preview.8 does not replace that. It widens that ownership into a replica group.

The pieces landed in dependency order across the preview.8 window:

Who replicates what: a physical replica ring and topology fingerprint. Key ownership comes from the existing consistent-hash ring, while replicas are the next RF−1 distinct node IDs on a separate sorted ring. The fingerprint hashes the canonical configuration, so a mismatched peer fails visibly instead of diverging silently;
One knob, three behaviors: ReplicaCount activation guards. RF=1, RF=2, and RF>=3 get different runtime behavior instead of one blurred mode:
- RF=1 keeps the previous single-owner path with no replication, elections, or group checks running at all;
- RF=2 is a synchronous mirror whose majority of two stops writes when either member is lost, with no replacement elected;
- only RF>=3 gets majority commit, with the surviving majority electing a new leader;
Replication stays server-internal: a separate contract with mTLS identity. It lives on the internal listener and is deliberately separate from the shared SquirixCache.proto, so cache clients never see replication RPCs;
Followers that can always catch up: an ordered log plus snapshots. Every follower append is journal-backed with no memory-only shortcut, and replica-group snapshots carry idempotency state, so a lagging follower catches up by installation rather than replay alone;
Success means a majority has it: a durable commit pipeline. It culminates in synchronous replicated mutations, with RF=1 behavior preserved: the server returns success only after a majority of the group durably acknowledges, and applies to memory only the committed prefix.

What differs in practice is what losing a member costs:

RF=1:  owner only, no replication, no failover
RF=2:  owner + 1 follower, majority = 2, mirror stops on any loss
RF>=3: owner + followers, majority commit, survivors elect a new leader

Static membership is a deliberate constraint. Peers are configured explicitly; there is no dynamic rebalancing in this preview. That removes a whole class of membership races from the first replicated release and lets the safety work focus on election, commit, and leadership change.

The write path: majority before success

For a replicated mutation, the ordering rule extends the preview.6 journal-first rule across nodes:

Do not return success before a majority of the configured group has durably acknowledged the mutation.

The path looks like this:

validate and admit the mutation
            |
            v
reserve a log index in the group
            |
            v
append locally + replicate to followers
            |
            v
wait for durable majority acknowledgements
            |
            v
advance commit_index, apply in memory
            |
            v
return success

Each mutation completes only when covered by that majority-durability step — the same "completed means covered" idea as preview.6 group commit, lifted from one fsync to a quorum of them.

Two details matter in practice:

Log-index reservation can be refused when exhausted. That refusal is definitive for the attempt: nothing was appended, nothing was applied. It fails closed before durability, like memory admission and journal quota in earlier previews.
Followers validate appends before copying entry payloads and gate tail reads, so a corrupt or out-of-range frame never becomes a committed prefix by accident.

Expiration follows the same pipeline rather than sneaking around it. Expiration candidates get their own operation identity and flow through the commit coordinator, so an expiration the leader observes becomes a committed entry applied identically on every replica — a follower never expires a key on its own.

Write-ahead intent and the honest unknown

Preview.6 already gave every mutation an opaque operation_id with a request fingerprint, propagated across owner-routing hops so a retry on another entry node still deduplicates on the key owner.

Preview.8 tightens the ordering inside that story: the server records a write-ahead idempotency intent before executing the mutation, not after. If the process dies between "accepted" and "applied," recovery still knows the operation existed and can resolve its outcome instead of executing it twice.

Even so, replication adds a case where the server cannot honestly say "committed" or "rejected." The owner replicated to some followers, crashed before learning whether a majority persisted the entry, and the client saw a timeout. The outcome may have committed on the survivors.

That is the new CommitOutcomeUnknown contract. Mutations can now throw CommitOutcomeUnknownException when the durable commit outcome is ambiguous. Retrying with the same operation_id stays idempotent: either the original outcome is returned or the mutation executes exactly once under the same identity.

client --operation_id=4f--> owner (crashes mid-commit)
   |
   +--retry, same operation_id--> new leader
                                      |
                                      +--> original outcome, or exactly-once execution

This is not a distributed transaction, and it is not presented as one. It is the same idempotency idea from preview.6, extended to the window where the owner set changes under the client.

Catch-up: logs first, snapshots when behind

A follower that restarts or falls behind does not get a special path. It gets the same log, from the point it diverged.

The follower log keeps an ordered, durable prefix with explicit term and index on every entry. Catch-up replays from the follower's last durable index: matching prefix is kept, conflicting suffix is replaced, missing suffix is appended. Only when the gap exceeds what the log retains does the leader send a snapshot.

A snapshot is one atomic, checksummed file with the committed state and the record of completed operations, so retries stay idempotent after install.

A follower installs a snapshot only from its own group, and only when the topology fingerprint matches and the configuration generation is compatible. Anything else is refused before a single byte is applied.

Installing replaces the follower's committed prefix. Entries past the snapshot boundary may stay, but only when the follower's entry exactly at the boundary matches the snapshot's last-included index and term. On any divergence at that boundary, the follower's tail is discarded and replayed fresh from the leader log. Compaction afterwards trims only the journal prefix the snapshot covers, keeping the log header and the snapshot itself installable.

Three concrete cases — in all of them the leader holds entries 1 through 6, and a primed entry (like 3') means the same index written in an older term:

leader:   [1][2][3][4][5][6]

Case 1 — small gap, conflicting tail.
The follower is almost current, but its last entry belongs to an older term:
follower: [1][2][3']           -> drop 3', copy 3..6 from the leader
result:   [1][2][3][4][5][6]

Case 2 — gap larger than the retained log.
The follower is so far behind that replaying entry by entry is no longer possible:
follower: [1]                  -> install the snapshot (state through 5),
                                  then replay entry 6 from the leader
result:   snapshot[1..5] + [6]

Case 3 — the snapshot boundary itself was written in another term.
The follower's tail overlaps the snapshot but disagrees with it at the boundary:
follower: [1][2][3][4][5']     -> install the snapshot (state through 5),
                                  discard the conflicting 5', replay 6
result:   snapshot[1..5] + [6]

The idempotency state rides along on purpose. A snapshot with values but without retry identity lets a delayed retry execute twice after old segments are compacted away. Group snapshots carry committed outcomes so catch-up preserves retry semantics, not just data.

What preview.8 changes

The replication work in 0.1.0-preview.8 includes:

replica sets RF 1–5 with static membership and a physical replica ring;
synchronous replicated mutations with durable majority commit; RF=1 behavior preserved;
write-ahead idempotency intent before mutation execution;
the CommitOutcomeUnknown contract for ambiguous durable outcomes;
ordered follower-log recovery, catch-up, and snapshot installation with idempotency state;
a server-only replication gRPC contract on the internal listener with mTLS identity.

Some of that improves availability. Most of it exists because awkward timing now spans nodes: an owner that dies after replicating halfway, a follower that returns with a conflicting suffix, a retry that arrives at a new leader. The happy path — three healthy nodes agreeing — is necessary, but the cases above are where the design is decided.

What preview.8 does not promise

This work makes single-node loss survivable for RF>=3, but Squirix is still early software.

Replication is synchronous and per key group; multi-key operations are not transactions across owners. RF=2 never promotes after peer loss. Quorum reads and leader election details will be covered in part 2; the operator surface (opt-in, metrics, RF=3 compose demo) in part 3 — links below.

The on-disk format may change during 0.x. Synchronous replication is not a substitute for backups or recovery testing. Preview.8 is a stronger foundation for availability, not a production-readiness claim.

Closing thought

Durability work asks: what survives a crash? Replication work asks: what survives the loss of the machine that holds the answer?

The difficult part is keeping the same honesty when the owner changes:

success means a majority holds it, not one disk;
unknown means unknown, with a stable identity to retry;
a follower that returns gets the log it missed, or a snapshot that preserves retry identity — never a guess.

Preview.8 makes those boundaries explicit across nodes. The next article goes one level deeper into how the cluster decides who may speak for a group: terms, votes, fencing, and why RF=2 stays a mirror.

If you run replicated .NET state or care about quorum commit semantics, comments are welcome on where this model still surprises — especially majority-commit latency, CommitOutcomeUnknown handling, and snapshot-vs-log catch-up thresholds.

Links

Squirix docs:

GitHub: https://github.com/squirix/squirix
Release v0.1.0-preview.8: https://github.com/squirix/squirix/releases/tag/v0.1.0-preview.8
Replication consensus ADR: https://github.com/squirix/squirix/blob/main/docs/architecture/replication-consensus.md
Clustering: https://github.com/squirix/squirix/blob/main/docs/clustering.md
Bootstrap client failover: https://github.com/squirix/squirix/blob/main/docs/bootstrap-client-failover.md

Earlier in this series:

What happens after a write? (preview.6): https://dev.to/alexander-efremov/what-happens-after-a-write-reworking-squirixs-wal-in-preview6-5ha4
When the disk fills up (preview.7): https://dev.to/alexander-efremov/when-the-disk-fills-up-pressure-and-tenant-isolation-in-squirix-preview7-3dpa

Next (forthcoming):

Who is the leader? Elections, fencing, and why RF=2 can't fail over (part 2)
Operating a replica set (part 3)

Try it:

dotnet add package squirix --version 0.1.0-preview.8
dotnet add package squirix.server --version 0.1.0-preview.8

When the disk fills up: pressure and tenant isolation in Squirix preview.7

Alex E — Sun, 26 Jul 2026 15:03:06 +0000

In my preview.6 article, I wrote about what happens after a write is accepted: binary journal frames, group commit, crash recovery, and durable retries. That work made the durability boundaries easier to name.

It does not answer the next uncomfortable question: what happens when the node should stop accepting more durable work?

A journal that never refuses appends will eventually fill the disk. A shared concurrency gate that never distinguishes callers will let one noisy tenant starve everyone else. From the outside, both failures can look like "the cache is down," even while the process is still alive.

Those questions shaped 0.1.0-preview.7: journal disk quotas that reject durable writes with a stable error, and per-principal backpressure keyed by JWT subject (or connection id when there is no subject).

Squirix is still an experimental preview, not a production-ready cache. Its APIs and storage formats may change during 0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.

The failure mode we wanted to avoid

Imagine a node that keeps appending journal segments until the volume is full. Depending on the OS and I/O path, the next write might throw, hang, or leave a torn tail. Operators then see process crashes, opaque I/O exceptions, and readiness flaps that take the node out of load balancers — exactly when health and metrics are most useful.

The contract we want instead is boring and explicit:

journal on-disk size approaches configured cap
                |
                v
durable append that would exceed the cap is rejected
                |
                v
client gets JOURNAL_DISK_QUOTA
(gRPC ResourceExhausted)
                |
                v
/health/ready stays healthy
/health/ready/details exposes journalDisk pressure

Rejection is definitive for that attempt. The process stays up. Readiness stays scrapeable so operators can reclaim space or raise the limit without first fighting a dead host.

That is different from the "commit unknown" case in preview.6. There, durable bytes may already exist and the client only saw a timeout. Here, the server refused the append before it crossed the durability boundary. Clients should not invent success; they should back off until operators reclaim headroom or raise the cap.

A hard cap with soft observability

Pipelined journal settings already expose JournalMaxTotalBytesMb — a hard on-disk size for journal segments, not snapshots or manifests. Preview.7 maps that cap into controlled write rejection. Segment-count and per-segment size caps can also reject with the same JOURNAL_DISK_QUOTA code.

The soft high-water for readiness details is fixed at 80% of the hard limit. Soft high-water is observability only: writes that still fit under the hard cap continue. Hard limit (critical) is the observed state when usedBytes >= maxBytes.

0% -------- 80% (high) -------- 100% (critical / hard cap)
|              |                      |
|   writes OK  |  writes OK if fit;   |  writeRejectionActive=true
|              |  oversize append     |  (usage already at cap)
|              |  still rejected      |
|              |  with JOURNAL_DISK_  |
|              |  QUOTA               |

The important timing detail: rejection uses usedBytes + appendBytes > maxBytes. An append can therefore be rejected while state is still high and writeRejectionActive is still false. writeRejectionActive becomes true only once usage has already reached the hard cap.

A few details matter in practice:

The quota covers journal segments only.
Rejected durable writes do not crash the process.
/health/ready remains healthy so the node can still expose pressure state.
/health/ready/details includes journalDisk with state, maxBytes, usedBytes, highWaterBytes, and writeRejectionActive.

The operator path is intentionally unexciting: confirm pressure, trigger or wait for snapshot plus compaction/retention, raise JournalMaxTotalBytesMb only after confirming disk capacity, and treat JOURNAL_DISK_QUOTA like other capacity rejections.

Why readiness stays healthy under quota

It is tempting to mark a node not-ready when the journal is full. That can remove it from traffic, but it also removes the cheapest way to inspect why durable writes stopped.

Preview.7 keeps /health/ready healthy and puts pressure into details:

load balancer / orchestrator
        |
        | GET /health/ready           --> 200 (still ready)
        |
        | GET /health/ready/details
        v
  journalDisk.state = critical
  writeRejectionActive = true

Cache clients already receive gRPC ResourceExhausted with public code JOURNAL_DISK_QUOTA. That is the write-path signal. Readiness is the operator-path signal. Mixing them turns every capacity event into a membership event.

HTTP on the primary listener is limited to health and metrics in v0.1; there is no public REST cache mutation surface. Shared error helpers can project the same logical error as HTTP 429 in tests and internal adapters, but application clients should key off the gRPC status and JOURNAL_DISK_QUOTA code.

Backpressure is not memory pressure

Squirix already separates several admission policies. Preview.7 does not merge them; it makes one of them fairer under multi-tenant load.

Transport limits still protect auth, payload size, and deadlines. Memory pressure still admits work based on estimated cache size. Journal disk quota still protects on-disk journal growth. Runtime backpressure protects concurrent cache operations, queues, slowdown, and optional rate limits.

Runtime backpressure sits after validation and before memory admission. A rejection there happens before logical cache operations enter memory admission or clustered/local paths, so rejected work does not append journal records, mutate local memory, update memory accounting, or record idempotency outcomes.

That is the same "fail closed before durability" idea as memory admission — applied to concurrency and rate, not to estimated bytes.

Isolating callers by principal

Global in-flight and queue limits protect the node. They do not stop one authenticated client from consuming the whole budget.

Preview.7 resolves a backpressure client id for each cache operation and applies per-client concurrency and rate limits against that id:

request
  |
  +--> HttpContext missing? ----------- yes --> runtime
  |
  +--> authenticated subject? --------- yes --> jwt:{subject}
  |
  +--> connection id present? --------- yes --> conn:{connectionId}
  |
  +-------------------------------------------> runtime

Authenticated external callers with a subject land in jwt:{subject} (NameIdentifier, then inbound sub).
Anonymous or subject-less HTTP callers land in conn:{connectionId}.
In-process callers without an HttpContext share one runtime bucket.

v0.1 external auth is JWT-only. Inter-node cluster forwarding uses mTLS on the internal listener and typically lands in the conn: or runtime bucket rather than a shared external JWT subject. That is deliberate: external tenant isolation should not treat owner-forwarded RPCs as one giant client.

Anonymous loopback clients on distinct TCP connections also do not share one JWT principal bucket. Without a principal, connection identity is the best stable key the host has.

What preview.7 changes

The pressure work in 0.1.0-preview.7 includes:

mapping the journal total-size hard cap to controlled durable write rejection (JOURNAL_DISK_QUOTA);
exposing soft high-water (80%) on readiness details without blocking writes that still fit;
keeping readiness healthy under quota so pressure remains observable;
keying per-client backpressure off JWT subject, then connection id, then the shared runtime bucket;
tightening path validation and symlink helpers, and clearing analyzer/Sonar debt on client and server hot paths.

Some of that is housekeeping. Most of it exists because awkward timing is not only about crashes. A full volume, a noisy neighbor, or a readiness flap at the wrong moment can erase the operational story as thoroughly as a torn journal frame.

What preview.7 does not promise

This work makes overload and disk exhaustion easier to reason about, but Squirix is still early software.

Quotas are per node and cover journal segments on that node; there is no cluster-wide disk accountant. In-process callers share runtime, and internal mTLS paths are not the same as external JWT tenants. Quota buys a clear error and time to act; it is not a substitute for capacity planning, compaction, backups, or recovery testing.

Preview.7 is stronger operator-facing pressure control on top of the preview.6 durability foundation, not a production-readiness declaration.

Closing thought

Durability work asks: what survives a crash? Pressure work asks: what happens when survival is no longer free?

A journal that always accepts appends will eventually lose the ability to flush cleanly. A concurrency gate that treats every caller as interchangeable will eventually amplify a noisy neighbor into a whole-node outage.

The difficult part is naming the refusal precisely:

rejected before durability (journal quota and backpressure both fail closed early);
rejected for this principal only when the signal is backpressure — journal disk quota remains node-wide;
still ready enough to inspect;
safe for clients to treat as definitive for that attempt.

Preview.7 makes those refusals more explicit. That gives operators a capacity policy instead of unexplained I/O chaos, and it gives future multi-tenant work a clearer place to hang fairness without rewriting the journal story.

If you operate multi-tenant .NET services or care about cache overload semantics, I would be glad to hear where this model still surprises you. Feedback on quota versus readiness, JWT versus connection isolation, and client reactions to JOURNAL_DISK_QUOTA is especially welcome.

Links

dotnet add package squirix --version 0.1.0-preview.7
dotnet add package squirix.server --version 0.1.0-preview.7

What happens after a write? Reworking Squirix's WAL in preview.6

Alex E — Sun, 19 Jul 2026 07:51:43 +0000

In my
first Squirix article,
I wrote about why Squirix keeps a strict boundary between the client and the server. Applications use a typed client
over gRPC; the server owns cache state, routing, persistence, recovery, and operational endpoints.

That boundary makes ownership clear. It does not answer the uncomfortable question underneath it: what exactly happens
between accepting a write and telling the client it succeeded?

Preview.6 goes one level deeper into that path. What if the process dies after writing to disk but before sending the
response? What survives a torn journal tail? How can a client retry without applying the same mutation twice?

Those questions shaped the release: a pipelined binary write-ahead log, tighter snapshot and journal recovery, and
idempotent mutation outcomes that survive a restart.

Squirix is still an experimental preview, not a production-ready cache. Its APIs and storage formats may change during
0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.

Follow one write

The basic ordering rule is simple:

Append the mutation to the journal before applying it in memory, and do not return success before the required
durability boundary.

For a normal durable mutation, the path looks like this:

validate and admit the mutation
            |
            v
append a binary WAL record
            |
            v
cross the configured durability boundary
            |
            v
apply the mutation in memory
            |
            v
return success

The order is deliberate. If the server changed memory first and appended to the journal afterward, a crash in between
could make an acknowledged write disappear after restart.

Idempotent RPCs take one deliberate variation. Squirix appends the mutation, applies it in memory to produce the
response, appends that response as an idempotency outcome, then waits for a single durability boundary covering both
records. The response is not returned before that wait completes.

The mutation and its retry result share one durability commit, without pretending the network response is atomic with
the disk write.

Journal-first ordering avoids acknowledged-but-unrecoverable writes, but it exposes a different case. Suppose the
journal becomes durable and the process dies before the response reaches the client. Recovery may replay the mutation
even though the client observed only a timeout or broken connection.

That outcome is not an ordinary failure. It is commit unknown: the operation may have committed, but the client did
not receive a definitive answer.

This is different from a rejection before the journal. If the memory-admission gate rejects a growing entry, Squirix
has not appended anything and has not changed cache state. That rejection is definitive. Once durable bytes may exist,
the answer has to be more careful.

Moving file I/O off request threads

Preview.6 replaces the previous journal path with a pipelined binary WAL backend. The goal was not simply to make file
writes faster. It was to give journal work one owner and make backpressure visible.

Request threads do not independently open files, encode records, and call fsync. They submit append work to a bounded
ring. A dedicated background thread named squirix-journal-io owns the journal event loop:

request threads
   |    |    |
   v    v    v
+-------------------+
| bounded WAL ring  |
+-------------------+
          |
          v
+------------------------+
| single journal thread  |
+------------------------+
          |
          v
   binary WAL segments

The journal thread drains accepted work, encodes and batches frames, writes segment bytes, services durability
deadlines, performs flushes, and coordinates segment rolls.

The queue is bounded because storage can always fall behind incoming traffic. An unbounded queue would turn that
mismatch into unbounded memory growth; the ring turns it into visible backpressure instead.

The single writer also gives segment state one clear owner. Offsets, batches, durability state, and roll decisions do
not need to be coordinated among arbitrary request threads.

Why the journal is binary

The new WAL writes binary frames rather than passing persistence records through JSON text.

That gives the storage path predictable record sizes, explicit field validation, CRC32C checksums, and span-based
encoding for the hot parts of the format. It also makes versioning a property of the storage format rather than an
accidental consequence of a general-purpose serializer.

This is partly a performance choice, but the main benefit is control. Recovery needs to distinguish a complete record
from arbitrary bytes at the end of a file. A framed binary format can say exactly how long a record is and whether its
checksum is valid.

The event loop also coalesces encoded records into write batches. Writing bytes and declaring those bytes durable are
separate steps, which is where group commit enters the design.

Sharing an `fsync` with group commit

Flushing every mutation separately gives straightforward semantics, but it also makes each request pay the full
durability cost.

With group commit enabled, several appends can share one flush:

append A ─┐
append B ─┼── write batch ── fsync ── complete A, B, and C
append C ─┘

Each waiter is completed only after a durability flush covers the relevant appends. The journal thread flushes the
current write batch, performs the storage flush, and then completes the captured batch of waiters. If the flush fails,
every waiter in that batch fails; no caller is told that partial durability was enough.

The group is bounded by configuration—a maximum wait and a maximum batch size—trading a small, controlled delay for
fewer durability flushes. Workloads that prefer the strictest per-write latency can disable group commit.

The important contract is not merely that an fsync happened somewhere. It is that a completed waiter is covered by
that durability flush.

The response can still get lost

Even with correct WAL ordering, the server cannot make the network response atomic with the disk write.

The failure window is easiest to see as a short sequence:

1. append operation A
2. flush operation A
3. crash before the response reaches the client
4. recover operation A from the WAL
5. client retries operation A

Without a stable request identity, step 5 could apply the same logical mutation twice. The risk is greatest for
mutations whose effect is not naturally idempotent.

Squirix mutation RPCs therefore carry an opaque operation_id. The server associates the ID with a request fingerprint
and the serialized response. A retry with the same ID and fingerprint can return the original result. Reusing the ID
for a different request is rejected.

In preview.6, that identity survives restart. Recovery rebuilds successful durable outcomes from operation metadata in
the journal and from retained idempotency records in snapshots. The operation ID is also preserved when a receiving
node forwards a mutation to the key owner.

client
  |
  | operation_id = 4f...
  v
receiving node
  |
  | same operation_id
  v
owner node
  |
  +--> durable mutation record
  |
  +--> durable outcome metadata

This is not a distributed transaction, and I will not present it as one. It gives a supported mutation a stable identity
across forwarding, timeouts, retries, and restarts—exactly the situations where commit unknown becomes operationally
important.

Recovery starts by distrusting the files

A snapshot is not valid merely because a file with the right name exists. Squirix reads and validates the complete
snapshot into a temporary load result before applying its entries. The startup gate remains closed until snapshot
restore and journal replay have both completed.

The current recovery flow is:

Read the manifest.
Locate the referenced snapshot.
Decode cache entries and retained idempotency records into a temporary load result.
Validate the complete snapshot.
Apply the validated snapshot state.
Replay journal records after the snapshot watermark.
Open the startup gate.

manifest
   |
   v
snapshot ── validate all frames ── apply recovered state
   |
   v
last applied sequence N
   |
   v
replay WAL records where sequence > N

The watermark matters. A snapshot represents state through a particular journal sequence. Replaying records at or
before that sequence would apply mutations twice; skipping later records would lose committed state.

Preview.6 freezes the metadata that describes the snapshot cut so that the entries and their sequence boundary refer
to the same logical point.

Snapshot publication follows the usual temporary-file pattern: write and flush a temporary file, publish the completed
snapshot, and only then update the manifest reference. A crash during the temporary write must not make an incomplete
snapshot authoritative.

If the referenced snapshot is missing, unreadable, truncated, or fails checksum validation, Squirix discards all
snapshot-derived state. It falls back to journal-only recovery only when the required journal history is still
available. Otherwise recovery fails instead of guessing across a gap. Preview.6 also fixes the valid fallback path so
replay starts from the earliest available segment when there is no snapshot watermark.

A torn tail does not erase the valid prefix

A process can stop halfway through the final WAL frame. Recovery must not interpret those bytes as a complete mutation,
but it should not discard every earlier record either.

[valid frame][valid frame][valid frame][partial bytes...]
                                      ^
                               recovery stops here

Journal replay proceeds through complete, validated records in segment order. During startup, Squirix scans the active
segment to the last valid frame boundary and truncates an incomplete tail before reopening it for writes.

This is intentionally conservative:

never invent a record from partial bytes;
never replay beyond an invalid boundary;
preserve the valid committed prefix.

The same principle applies to snapshots: do not start applying entries until the complete file has been decoded and
validated. A clean journal fallback is easier to reason about than trusting a partly decoded snapshot.

Compaction must preserve retry semantics

Compaction cannot keep only the latest key/value state.

Imagine that operation A committed, its result was retained for deduplication, and old journal segments were then
compacted away. If the snapshot contained cache values but not the idempotency record, a delayed retry of A could lose
its original identity and execute again.

Snapshots therefore include retained idempotency outcomes alongside cache entries. After compaction and restart,
Squirix needs to recover both:

cache state
+
retry identity and durable outcomes

Durability is not only the ability to reconstruct values. It also means reconstructing enough protocol state to handle
uncertain retries safely.

What preview.6 changes

The durability work in 0.1.0-preview.6 includes:

a pipelined binary WAL;
a dedicated single-writer journal event loop;
bounded append backpressure;
configurable group commit;
journal-only recovery from the earliest available segment when the journal topology is valid;
a stable snapshot cut and replay watermark;
durable operation outcomes across restart;
operation-ID propagation between nodes;
retained idempotency records in snapshots;
compaction cancellation during shutdown;
lower allocation pressure on WAL hot paths.

Some changes improve throughput. Most exist because of awkward timing: a crash after a journal flush, a lost response,
a truncated final frame, compaction followed by an old retry, or shutdown while persistence work is active.

That is where durability engineering lives. The happy path is necessary, but it is rarely the part that keeps me
thinking after the code is written.

What preview.6 does not promise

This work makes the durability model easier to reason about, but Squirix is still early software.

The on-disk format may change during 0.x. Durability is per node; preview.6 does not add replication or automatic
failover. Operations spanning multiple owners are not globally atomic transactions. Serializer compatibility and
restart behavior still need to be validated for the mutation paths a workload depends on.

A WAL is not a substitute for replication, backups, or recovery testing. Preview.6 is a stronger durability foundation,
not a production-readiness declaration.

Closing thought

Appending bytes is the easy part of a write-ahead log. Deciding what those bytes mean after a crash is the real work.

The difficult part is naming every boundary precisely:

accepted;
appended;
flushed;
applied in memory;
visible to the client;
recoverable after restart;
safe to retry.

Preview.6 makes those boundaries more explicit. That gives future work—replication, failover, and more advanced cluster
behavior—a durability model it can build on instead of work around.

If you work on storage engines or distributed .NET systems, I would be glad to hear where you think the remaining sharp
edges are. The project is open source, and feedback on the WAL, recovery semantics, and retry model is especially
welcome.

Why Squirix uses a strict client/server architecture for a .NET distributed cache

Alex E — Sun, 07 Jun 2026 07:12:52 +0000

Squirix 0.1.0 is an early preview of a .NET distributed cache. A typed client SDK talks to a remote server over gRPC; the server owns state, routing, durability, and operational endpoints.

This is the direction I am validating in 0.1.0 — not a claim that every cache must work this way. Embedded designs are fine for many workloads. Squirix targets a different shape: the application stays a client; the server owns the data lifecycle.

The problem with "just a cache library"

A cache library is simple until you ask who owns what. When cache logic runs inside your app process, state, memory pressure, and persistence share the app's lifecycle. That works for local acceleration (IMemoryCache), but gets ambiguous when you need shared state across instances, durability across restarts, independent health/metrics, or cluster routing.

At that point you often have an implicit server with unclear boundaries. Squirix makes the split explicit from day one.

Embedded mode vs client/server mode

Embedded — cache logic in or tightly coupled to the app process: in-memory caches, libraries that hide remote I/O, or a co-located server called via direct references. Low friction; you rarely expect separate health probes or journal compaction on "just a dependency."

Client/server — the app holds no authoritative state. It connects over a wire contract; the server owns placement, mutations, durability, recovery, and admin/metrics endpoints. Redis and most production distributed caches follow this shape.

Squirix 0.1.0 is client/server first:

application -> Squirix client SDK -> Squirix.Server node(s)

Two packages, enforced at build time: squirix (client) and squirix.server (runtime). The server does not reference the client assembly.

You can embed the server in ASP.NET Core via AddSquirixServer / MapSquirixServer, but application access still goes through SquirixClient.ConnectAsync(...) — even in the same process. That is hosting convenience, not embedded cache semantics in the app layer.

Embedded mode is not bad — it optimizes for different goals. Squirix targets shared remote state, server-owned durability, and operability as infrastructure.

Why Squirix chooses client/server first

Reasoning behind the 0.1.0 shape:

Operational boundary — deploy, probe, and upgrade cache nodes independently. Health (/health/live, /health/ready), admin (/admin/whoami, /admin/ring), and Prometheus at /metrics live on the server.
Server-owned lifecycle — WAL journal, snapshots, compaction, and recovery stay in squirix.server, not in every application.
Lighter client package — run the server as its own process or container; apps only reference the client SDK.
Clustering path — static consistent-hash routing in 0.1.0 is early, but routing and failover can evolve server-side without rewriting ICache<T>.
Durability isolation — recovery and compaction failures stay out of application request threads while semantics harden.

How this affects the public API

You connect via SquirixClient.ConnectAsync(...) and work with typed ICache<T> and explicit read results (CacheValueResult<T>):

using System.Threading;
using Squirix;

var cancellationToken = CancellationToken.None;

await using var client = await SquirixClient.ConnectAsync(
    "http://localhost:5001",
    cancellationToken);

var cache = await client.GetCacheAsync<string>("demo", cancellationToken);
await cache.SetAsync("greeting", "hello", cancellationToken: cancellationToken);

var lookup = await cache.GetValueAsync("greeting", cancellationToken);
if (lookup.Found)
    Console.WriteLine(lookup.Value);

Production server hosting uses AddSquirixServer / MapSquirixServer. Transport is gRPC (SquirixCache.proto); cache operations, health, and admin routes are also available over HTTP/2 REST.

What exists in Squirix 0.1.0

0.1.0-preview.1, .NET 10 only:

Client/server split (squirix + squirix.server)
Typed ICache<T> — basic KV + expiration
gRPC + HTTP/2 REST cache endpoints
Per-node journal, snapshots, compaction, recovery
Health, readiness, admin routes; Prometheus metrics; OpenTelemetry journal tracing
Static consistent-hash single-owner routing

For local h2c dev: $env:DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP2UNENCRYPTEDSUPPORT = "1"

What is still experimental

Early preview — not production-ready. API, wire format, and on-disk layouts may change during 0.x.

DEV Community: Alex E

From one owner to a replica set: synchronous replication in Squirix preview.8

From one owner to a replica set: synchronous replication in Squirix preview.8

The failure mode to avoid

From single owner to replica set

The write path: majority before success

Write-ahead intent and the honest unknown

Catch-up: logs first, snapshots when behind

What preview.8 changes

What preview.8 does not promise

Closing thought

Links

When the disk fills up: pressure and tenant isolation in Squirix preview.7

The failure mode we wanted to avoid

A hard cap with soft observability

Why readiness stays healthy under quota

Backpressure is not memory pressure

Isolating callers by principal

What preview.7 changes

What preview.7 does not promise

Closing thought

Links

What happens after a write? Reworking Squirix's WAL in preview.6

Follow one write

Moving file I/O off request threads

Why the journal is binary

Sharing an fsync with group commit

The response can still get lost

Recovery starts by distrusting the files

A torn tail does not erase the valid prefix

Compaction must preserve retry semantics

What preview.6 changes

What preview.6 does not promise

Closing thought

Links

Why Squirix uses a strict client/server architecture for a .NET distributed cache

The problem with "just a cache library"

Embedded mode vs client/server mode

Why Squirix chooses client/server first

How this affects the public API

What exists in Squirix 0.1.0

What is still experimental

Links

Sharing an `fsync` with group commit