DEV Community: Alex E

When the disk fills up: pressure and tenant isolation in Squirix preview.7

Alex E — Sun, 26 Jul 2026 15:03:06 +0000

In my preview.6 article, I wrote about what happens after a write is accepted: binary journal frames, group commit, crash recovery, and durable retries. That work made the durability boundaries easier to name.

It does not answer the next uncomfortable question: what happens when the node should stop accepting more durable work?

A journal that never refuses appends will eventually fill the disk. A shared concurrency gate that never distinguishes callers will let one noisy tenant starve everyone else. From the outside, both failures can look like "the cache is down," even while the process is still alive.

Those questions shaped 0.1.0-preview.7: journal disk quotas that reject durable writes with a stable error, and per-principal backpressure keyed by JWT subject (or connection id when there is no subject).

Squirix is still an experimental preview, not a production-ready cache. Its APIs and storage formats may change during 0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.

The failure mode we wanted to avoid

Imagine a node that keeps appending journal segments until the volume is full. Depending on the OS and I/O path, the next write might throw, hang, or leave a torn tail. Operators then see process crashes, opaque I/O exceptions, and readiness flaps that take the node out of load balancers — exactly when health and metrics are most useful.

The contract we want instead is boring and explicit:

journal on-disk size approaches configured cap
                |
                v
durable append that would exceed the cap is rejected
                |
                v
client gets JOURNAL_DISK_QUOTA
(gRPC ResourceExhausted)
                |
                v
/health/ready stays healthy
/health/ready/details exposes journalDisk pressure

Rejection is definitive for that attempt. The process stays up. Readiness stays scrapeable so operators can reclaim space or raise the limit without first fighting a dead host.

That is different from the "commit unknown" case in preview.6. There, durable bytes may already exist and the client only saw a timeout. Here, the server refused the append before it crossed the durability boundary. Clients should not invent success; they should back off until operators reclaim headroom or raise the cap.

A hard cap with soft observability

Pipelined journal settings already expose JournalMaxTotalBytesMb — a hard on-disk size for journal segments, not snapshots or manifests. Preview.7 maps that cap into controlled write rejection. Segment-count and per-segment size caps can also reject with the same JOURNAL_DISK_QUOTA code.

The soft high-water for readiness details is fixed at 80% of the hard limit. Soft high-water is observability only: writes that still fit under the hard cap continue. Hard limit (critical) is the observed state when usedBytes >= maxBytes.

0% -------- 80% (high) -------- 100% (critical / hard cap)
|              |                      |
|   writes OK  |  writes OK if fit;   |  writeRejectionActive=true
|              |  oversize append     |  (usage already at cap)
|              |  still rejected      |
|              |  with JOURNAL_DISK_  |
|              |  QUOTA               |

The important timing detail: rejection uses usedBytes + appendBytes > maxBytes. An append can therefore be rejected while state is still high and writeRejectionActive is still false. writeRejectionActive becomes true only once usage has already reached the hard cap.

A few details matter in practice:

The quota covers journal segments only.
Rejected durable writes do not crash the process.
/health/ready remains healthy so the node can still expose pressure state.
/health/ready/details includes journalDisk with state, maxBytes, usedBytes, highWaterBytes, and writeRejectionActive.

The operator path is intentionally unexciting: confirm pressure, trigger or wait for snapshot plus compaction/retention, raise JournalMaxTotalBytesMb only after confirming disk capacity, and treat JOURNAL_DISK_QUOTA like other capacity rejections.

Why readiness stays healthy under quota

It is tempting to mark a node not-ready when the journal is full. That can remove it from traffic, but it also removes the cheapest way to inspect why durable writes stopped.

Preview.7 keeps /health/ready healthy and puts pressure into details:

load balancer / orchestrator
        |
        | GET /health/ready           --> 200 (still ready)
        |
        | GET /health/ready/details
        v
  journalDisk.state = critical
  writeRejectionActive = true

Cache clients already receive gRPC ResourceExhausted with public code JOURNAL_DISK_QUOTA. That is the write-path signal. Readiness is the operator-path signal. Mixing them turns every capacity event into a membership event.

HTTP on the primary listener is limited to health and metrics in v0.1; there is no public REST cache mutation surface. Shared error helpers can project the same logical error as HTTP 429 in tests and internal adapters, but application clients should key off the gRPC status and JOURNAL_DISK_QUOTA code.

Backpressure is not memory pressure

Squirix already separates several admission policies. Preview.7 does not merge them; it makes one of them fairer under multi-tenant load.

Transport limits still protect auth, payload size, and deadlines. Memory pressure still admits work based on estimated cache size. Journal disk quota still protects on-disk journal growth. Runtime backpressure protects concurrent cache operations, queues, slowdown, and optional rate limits.

Runtime backpressure sits after validation and before memory admission. A rejection there happens before logical cache operations enter memory admission or clustered/local paths, so rejected work does not append journal records, mutate local memory, update memory accounting, or record idempotency outcomes.

That is the same "fail closed before durability" idea as memory admission — applied to concurrency and rate, not to estimated bytes.

Isolating callers by principal

Global in-flight and queue limits protect the node. They do not stop one authenticated client from consuming the whole budget.

Preview.7 resolves a backpressure client id for each cache operation and applies per-client concurrency and rate limits against that id:

request
  |
  +--> HttpContext missing? ----------- yes --> runtime
  |
  +--> authenticated subject? --------- yes --> jwt:{subject}
  |
  +--> connection id present? --------- yes --> conn:{connectionId}
  |
  +-------------------------------------------> runtime

Authenticated external callers with a subject land in jwt:{subject} (NameIdentifier, then inbound sub).
Anonymous or subject-less HTTP callers land in conn:{connectionId}.
In-process callers without an HttpContext share one runtime bucket.

v0.1 external auth is JWT-only. Inter-node cluster forwarding uses mTLS on the internal listener and typically lands in the conn: or runtime bucket rather than a shared external JWT subject. That is deliberate: external tenant isolation should not treat owner-forwarded RPCs as one giant client.

Anonymous loopback clients on distinct TCP connections also do not share one JWT principal bucket. Without a principal, connection identity is the best stable key the host has.

What preview.7 changes

The pressure work in 0.1.0-preview.7 includes:

mapping the journal total-size hard cap to controlled durable write rejection (JOURNAL_DISK_QUOTA);
exposing soft high-water (80%) on readiness details without blocking writes that still fit;
keeping readiness healthy under quota so pressure remains observable;
keying per-client backpressure off JWT subject, then connection id, then the shared runtime bucket;
tightening path validation and symlink helpers, and clearing analyzer/Sonar debt on client and server hot paths.

Some of that is housekeeping. Most of it exists because awkward timing is not only about crashes. A full volume, a noisy neighbor, or a readiness flap at the wrong moment can erase the operational story as thoroughly as a torn journal frame.

What preview.7 does not promise

This work makes overload and disk exhaustion easier to reason about, but Squirix is still early software.

Quotas are per node and cover journal segments on that node; there is no cluster-wide disk accountant. In-process callers share runtime, and internal mTLS paths are not the same as external JWT tenants. Quota buys a clear error and time to act; it is not a substitute for capacity planning, compaction, backups, or recovery testing.

Preview.7 is stronger operator-facing pressure control on top of the preview.6 durability foundation, not a production-readiness declaration.

Closing thought

Durability work asks: what survives a crash? Pressure work asks: what happens when survival is no longer free?

A journal that always accepts appends will eventually lose the ability to flush cleanly. A concurrency gate that treats every caller as interchangeable will eventually amplify a noisy neighbor into a whole-node outage.

The difficult part is naming the refusal precisely:

rejected before durability (journal quota and backpressure both fail closed early);
rejected for this principal only when the signal is backpressure — journal disk quota remains node-wide;
still ready enough to inspect;
safe for clients to treat as definitive for that attempt.

Preview.7 makes those refusals more explicit. That gives operators a capacity policy instead of unexplained I/O chaos, and it gives future multi-tenant work a clearer place to hang fairness without rewriting the journal story.

If you operate multi-tenant .NET services or care about cache overload semantics, I would be glad to hear where this model still surprises you. Feedback on quota versus readiness, JWT versus connection isolation, and client reactions to JOURNAL_DISK_QUOTA is especially welcome.

Links

dotnet add package squirix --version 0.1.0-preview.7
dotnet add package squirix.server --version 0.1.0-preview.7

What happens after a write? Reworking Squirix's WAL in preview.6

Alex E — Sun, 19 Jul 2026 07:51:43 +0000

In my
first Squirix article,
I wrote about why Squirix keeps a strict boundary between the client and the server. Applications use a typed client
over gRPC; the server owns cache state, routing, persistence, recovery, and operational endpoints.

That boundary makes ownership clear. It does not answer the uncomfortable question underneath it: what exactly happens
between accepting a write and telling the client it succeeded?

Preview.6 goes one level deeper into that path. What if the process dies after writing to disk but before sending the
response? What survives a torn journal tail? How can a client retry without applying the same mutation twice?

Those questions shaped the release: a pipelined binary write-ahead log, tighter snapshot and journal recovery, and
idempotent mutation outcomes that survive a restart.

Squirix is still an experimental preview, not a production-ready cache. Its APIs and storage formats may change during
0.x. This article describes the current design and the reasoning behind it, not a compatibility promise.

Follow one write

The basic ordering rule is simple:

Append the mutation to the journal before applying it in memory, and do not return success before the required
durability boundary.

For a normal durable mutation, the path looks like this:

validate and admit the mutation
            |
            v
append a binary WAL record
            |
            v
cross the configured durability boundary
            |
            v
apply the mutation in memory
            |
            v
return success

The order is deliberate. If the server changed memory first and appended to the journal afterward, a crash in between
could make an acknowledged write disappear after restart.

Idempotent RPCs take one deliberate variation. Squirix appends the mutation, applies it in memory to produce the
response, appends that response as an idempotency outcome, then waits for a single durability boundary covering both
records. The response is not returned before that wait completes.

The mutation and its retry result share one durability commit, without pretending the network response is atomic with
the disk write.

Journal-first ordering avoids acknowledged-but-unrecoverable writes, but it exposes a different case. Suppose the
journal becomes durable and the process dies before the response reaches the client. Recovery may replay the mutation
even though the client observed only a timeout or broken connection.

That outcome is not an ordinary failure. It is commit unknown: the operation may have committed, but the client did
not receive a definitive answer.

This is different from a rejection before the journal. If the memory-admission gate rejects a growing entry, Squirix
has not appended anything and has not changed cache state. That rejection is definitive. Once durable bytes may exist,
the answer has to be more careful.

Moving file I/O off request threads

Preview.6 replaces the previous journal path with a pipelined binary WAL backend. The goal was not simply to make file
writes faster. It was to give journal work one owner and make backpressure visible.

Request threads do not independently open files, encode records, and call fsync. They submit append work to a bounded
ring. A dedicated background thread named squirix-journal-io owns the journal event loop:

request threads
   |    |    |
   v    v    v
+-------------------+
| bounded WAL ring  |
+-------------------+
          |
          v
+------------------------+
| single journal thread  |
+------------------------+
          |
          v
   binary WAL segments

The journal thread drains accepted work, encodes and batches frames, writes segment bytes, services durability
deadlines, performs flushes, and coordinates segment rolls.

The queue is bounded because storage can always fall behind incoming traffic. An unbounded queue would turn that
mismatch into unbounded memory growth; the ring turns it into visible backpressure instead.

The single writer also gives segment state one clear owner. Offsets, batches, durability state, and roll decisions do
not need to be coordinated among arbitrary request threads.

Why the journal is binary

The new WAL writes binary frames rather than passing persistence records through JSON text.

That gives the storage path predictable record sizes, explicit field validation, CRC32C checksums, and span-based
encoding for the hot parts of the format. It also makes versioning a property of the storage format rather than an
accidental consequence of a general-purpose serializer.

This is partly a performance choice, but the main benefit is control. Recovery needs to distinguish a complete record
from arbitrary bytes at the end of a file. A framed binary format can say exactly how long a record is and whether its
checksum is valid.

The event loop also coalesces encoded records into write batches. Writing bytes and declaring those bytes durable are
separate steps, which is where group commit enters the design.

Sharing an `fsync` with group commit

Flushing every mutation separately gives straightforward semantics, but it also makes each request pay the full
durability cost.

With group commit enabled, several appends can share one flush:

append A ─┐
append B ─┼── write batch ── fsync ── complete A, B, and C
append C ─┘

Each waiter is completed only after a durability flush covers the relevant appends. The journal thread flushes the
current write batch, performs the storage flush, and then completes the captured batch of waiters. If the flush fails,
every waiter in that batch fails; no caller is told that partial durability was enough.

The group is bounded by configuration—a maximum wait and a maximum batch size—trading a small, controlled delay for
fewer durability flushes. Workloads that prefer the strictest per-write latency can disable group commit.

The important contract is not merely that an fsync happened somewhere. It is that a completed waiter is covered by
that durability flush.

The response can still get lost

Even with correct WAL ordering, the server cannot make the network response atomic with the disk write.

The failure window is easiest to see as a short sequence:

1. append operation A
2. flush operation A
3. crash before the response reaches the client
4. recover operation A from the WAL
5. client retries operation A

Without a stable request identity, step 5 could apply the same logical mutation twice. The risk is greatest for
mutations whose effect is not naturally idempotent.

Squirix mutation RPCs therefore carry an opaque operation_id. The server associates the ID with a request fingerprint
and the serialized response. A retry with the same ID and fingerprint can return the original result. Reusing the ID
for a different request is rejected.

In preview.6, that identity survives restart. Recovery rebuilds successful durable outcomes from operation metadata in
the journal and from retained idempotency records in snapshots. The operation ID is also preserved when a receiving
node forwards a mutation to the key owner.

client
  |
  | operation_id = 4f...
  v
receiving node
  |
  | same operation_id
  v
owner node
  |
  +--> durable mutation record
  |
  +--> durable outcome metadata

This is not a distributed transaction, and I will not present it as one. It gives a supported mutation a stable identity
across forwarding, timeouts, retries, and restarts—exactly the situations where commit unknown becomes operationally
important.

Recovery starts by distrusting the files

A snapshot is not valid merely because a file with the right name exists. Squirix reads and validates the complete
snapshot into a temporary load result before applying its entries. The startup gate remains closed until snapshot
restore and journal replay have both completed.

The current recovery flow is:

Read the manifest.
Locate the referenced snapshot.
Decode cache entries and retained idempotency records into a temporary load result.
Validate the complete snapshot.
Apply the validated snapshot state.
Replay journal records after the snapshot watermark.
Open the startup gate.

manifest
   |
   v
snapshot ── validate all frames ── apply recovered state
   |
   v
last applied sequence N
   |
   v
replay WAL records where sequence > N

The watermark matters. A snapshot represents state through a particular journal sequence. Replaying records at or
before that sequence would apply mutations twice; skipping later records would lose committed state.

Preview.6 freezes the metadata that describes the snapshot cut so that the entries and their sequence boundary refer
to the same logical point.

Snapshot publication follows the usual temporary-file pattern: write and flush a temporary file, publish the completed
snapshot, and only then update the manifest reference. A crash during the temporary write must not make an incomplete
snapshot authoritative.

If the referenced snapshot is missing, unreadable, truncated, or fails checksum validation, Squirix discards all
snapshot-derived state. It falls back to journal-only recovery only when the required journal history is still
available. Otherwise recovery fails instead of guessing across a gap. Preview.6 also fixes the valid fallback path so
replay starts from the earliest available segment when there is no snapshot watermark.

A torn tail does not erase the valid prefix

A process can stop halfway through the final WAL frame. Recovery must not interpret those bytes as a complete mutation,
but it should not discard every earlier record either.

[valid frame][valid frame][valid frame][partial bytes...]
                                      ^
                               recovery stops here

Journal replay proceeds through complete, validated records in segment order. During startup, Squirix scans the active
segment to the last valid frame boundary and truncates an incomplete tail before reopening it for writes.

This is intentionally conservative:

never invent a record from partial bytes;
never replay beyond an invalid boundary;
preserve the valid committed prefix.

The same principle applies to snapshots: do not start applying entries until the complete file has been decoded and
validated. A clean journal fallback is easier to reason about than trusting a partly decoded snapshot.

Compaction must preserve retry semantics

Compaction cannot keep only the latest key/value state.

Imagine that operation A committed, its result was retained for deduplication, and old journal segments were then
compacted away. If the snapshot contained cache values but not the idempotency record, a delayed retry of A could lose
its original identity and execute again.

Snapshots therefore include retained idempotency outcomes alongside cache entries. After compaction and restart,
Squirix needs to recover both:

cache state
+
retry identity and durable outcomes

Durability is not only the ability to reconstruct values. It also means reconstructing enough protocol state to handle
uncertain retries safely.

What preview.6 changes

The durability work in 0.1.0-preview.6 includes:

a pipelined binary WAL;
a dedicated single-writer journal event loop;
bounded append backpressure;
configurable group commit;
journal-only recovery from the earliest available segment when the journal topology is valid;
a stable snapshot cut and replay watermark;
durable operation outcomes across restart;
operation-ID propagation between nodes;
retained idempotency records in snapshots;
compaction cancellation during shutdown;
lower allocation pressure on WAL hot paths.

Some changes improve throughput. Most exist because of awkward timing: a crash after a journal flush, a lost response,
a truncated final frame, compaction followed by an old retry, or shutdown while persistence work is active.

That is where durability engineering lives. The happy path is necessary, but it is rarely the part that keeps me
thinking after the code is written.

What preview.6 does not promise

This work makes the durability model easier to reason about, but Squirix is still early software.

The on-disk format may change during 0.x. Durability is per node; preview.6 does not add replication or automatic
failover. Operations spanning multiple owners are not globally atomic transactions. Serializer compatibility and
restart behavior still need to be validated for the mutation paths a workload depends on.

A WAL is not a substitute for replication, backups, or recovery testing. Preview.6 is a stronger durability foundation,
not a production-readiness declaration.

Closing thought

Appending bytes is the easy part of a write-ahead log. Deciding what those bytes mean after a crash is the real work.

The difficult part is naming every boundary precisely:

accepted;
appended;
flushed;
applied in memory;
visible to the client;
recoverable after restart;
safe to retry.

Preview.6 makes those boundaries more explicit. That gives future work—replication, failover, and more advanced cluster
behavior—a durability model it can build on instead of work around.

If you work on storage engines or distributed .NET systems, I would be glad to hear where you think the remaining sharp
edges are. The project is open source, and feedback on the WAL, recovery semantics, and retry model is especially
welcome.

Why Squirix uses a strict client/server architecture for a .NET distributed cache

Alex E — Sun, 07 Jun 2026 07:12:52 +0000

Squirix 0.1.0 is an early preview of a .NET distributed cache. A typed client SDK talks to a remote server over gRPC; the server owns state, routing, durability, and operational endpoints.

This is the direction I am validating in 0.1.0 — not a claim that every cache must work this way. Embedded designs are fine for many workloads. Squirix targets a different shape: the application stays a client; the server owns the data lifecycle.

The problem with "just a cache library"

A cache library is simple until you ask who owns what. When cache logic runs inside your app process, state, memory pressure, and persistence share the app's lifecycle. That works for local acceleration (IMemoryCache), but gets ambiguous when you need shared state across instances, durability across restarts, independent health/metrics, or cluster routing.

At that point you often have an implicit server with unclear boundaries. Squirix makes the split explicit from day one.

Embedded mode vs client/server mode

Embedded — cache logic in or tightly coupled to the app process: in-memory caches, libraries that hide remote I/O, or a co-located server called via direct references. Low friction; you rarely expect separate health probes or journal compaction on "just a dependency."

Client/server — the app holds no authoritative state. It connects over a wire contract; the server owns placement, mutations, durability, recovery, and admin/metrics endpoints. Redis and most production distributed caches follow this shape.

Squirix 0.1.0 is client/server first:

application -> Squirix client SDK -> Squirix.Server node(s)

Two packages, enforced at build time: squirix (client) and squirix.server (runtime). The server does not reference the client assembly.

You can embed the server in ASP.NET Core via AddSquirixServer / MapSquirixServer, but application access still goes through SquirixClient.ConnectAsync(...) — even in the same process. That is hosting convenience, not embedded cache semantics in the app layer.

Embedded mode is not bad — it optimizes for different goals. Squirix targets shared remote state, server-owned durability, and operability as infrastructure.

Why Squirix chooses client/server first

Reasoning behind the 0.1.0 shape:

Operational boundary — deploy, probe, and upgrade cache nodes independently. Health (/health/live, /health/ready), admin (/admin/whoami, /admin/ring), and Prometheus at /metrics live on the server.
Server-owned lifecycle — WAL journal, snapshots, compaction, and recovery stay in squirix.server, not in every application.
Lighter client package — run the server as its own process or container; apps only reference the client SDK.
Clustering path — static consistent-hash routing in 0.1.0 is early, but routing and failover can evolve server-side without rewriting ICache<T>.
Durability isolation — recovery and compaction failures stay out of application request threads while semantics harden.

How this affects the public API

You connect via SquirixClient.ConnectAsync(...) and work with typed ICache<T> and explicit read results (CacheValueResult<T>):

using System.Threading;
using Squirix;

var cancellationToken = CancellationToken.None;

await using var client = await SquirixClient.ConnectAsync(
    "http://localhost:5001",
    cancellationToken);

var cache = await client.GetCacheAsync<string>("demo", cancellationToken);
await cache.SetAsync("greeting", "hello", cancellationToken: cancellationToken);

var lookup = await cache.GetValueAsync("greeting", cancellationToken);
if (lookup.Found)
    Console.WriteLine(lookup.Value);

Production server hosting uses AddSquirixServer / MapSquirixServer. Transport is gRPC (SquirixCache.proto); cache operations, health, and admin routes are also available over HTTP/2 REST.

What exists in Squirix 0.1.0

0.1.0-preview.1, .NET 10 only:

Client/server split (squirix + squirix.server)
Typed ICache<T> — basic KV + expiration
gRPC + HTTP/2 REST cache endpoints
Per-node journal, snapshots, compaction, recovery
Health, readiness, admin routes; Prometheus metrics; OpenTelemetry journal tracing
Static consistent-hash single-owner routing

For local h2c dev: $env:DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP2UNENCRYPTEDSUPPORT = "1"

What is still experimental

Early preview — not production-ready. API, wire format, and on-disk layouts may change during 0.x.

DEV Community: Alex E

When the disk fills up: pressure and tenant isolation in Squirix preview.7

The failure mode we wanted to avoid

A hard cap with soft observability

Why readiness stays healthy under quota

Backpressure is not memory pressure

Isolating callers by principal

What preview.7 changes

What preview.7 does not promise

Closing thought

Links

What happens after a write? Reworking Squirix's WAL in preview.6

Follow one write

Moving file I/O off request threads

Why the journal is binary

Sharing an fsync with group commit

The response can still get lost

Recovery starts by distrusting the files

A torn tail does not erase the valid prefix

Compaction must preserve retry semantics

What preview.6 changes

What preview.6 does not promise

Closing thought

Links

Why Squirix uses a strict client/server architecture for a .NET distributed cache

The problem with "just a cache library"

Embedded mode vs client/server mode

Why Squirix chooses client/server first

How this affects the public API

What exists in Squirix 0.1.0

What is still experimental

Links

Sharing an `fsync` with group commit