How durability works in a storage engine we wrote ourselves

Durability is the most overloaded word on a database landing page. Sometimes it means "we take nightly backups." Sometimes it means "we replicate." Occasionally it means "we have not lost anything yet."

None of those are the same guarantee, and the difference between them is what you need to know before putting state on someone else's infrastructure.

We wrote the storage engine, so we can be precise. ScalixNova speaks the PostgreSQL wire protocol, but underneath it is our own write ahead log rather than a managed wrapper around an upstream distribution. This post walks the write path and says what "durable" means at each boundary it crosses.

A write moves through four states. The engineering that matters lives in the transitions.

1. Accepted by the log's current term holder
2. Flushed and fsync'd into a WAL segment on local disk
3. Archived as a closed segment in object storage
4. Restorable to a point in time

1. Accepted: term fencing on a single writer

The log has exactly one writer at a time. That writer holds a term, a monotonically increasing number. Every append carries the proposer's term, and the acceptor compares it against the term it currently honours:

• Proposer term lower than current: reject with a stale proposer error. Nothing touches the disk.
• Proposer term higher: adopt the new term, then accept.
• Equal: accept.

The obvious objection is that a single writer cannot split-brain by definition. It can, and that is the failure fencing exists for. The problem is not two writers running concurrently by design. It is one writer that has already been replaced, does not know it, and comes back holding a stale view of the log. A long garbage collection pause, a partitioned network, a slow restart after an operator action: any of those produce a process that still believes it is the writer.

Without fencing, that process appends after its successor already did, and you get one log with two divergent futures at the same LSN. Nothing in the byte stream tells you which branch is real. With fencing, its very first append is rejected on the term comparison, before a single byte is written.

Term is not just an in-memory counter. It is persisted alongside flush LSN, commit LSN and remote consistent LSN in the timeline metadata, written through a crash safe path: write a temp file, fsync it, rename over the target, then fsync the parent directory. After a crash the acceptor comes back knowing which term it last honoured. A stale proposer cannot win simply by outliving a restart.

2. Flushed: fsync, then acknowledge

An append computes the segment number and offset from the LSN, opens the segment file, seeks, writes, and calls sync_all() before returning. The flush LSN advances only after that fsync returns. If a record spans a segment boundary, each segment is fsync'd in turn.

There is a second rule in the append path that does more work than it looks like. Appends must be contiguous. If the incoming start LSN is not exactly the current flush LSN, the append is rejected with a gap error rather than written at the requested offset. A WAL with a hole in it is not a WAL. Refusing the gap at the append boundary means the recovery path never has to reason about what might be missing in the middle.

This is the layer where "durable" means what an SRE means by it. The bytes are on the device, and the acknowledgement is emitted after the fsync rather than before it.

3. Archived: local disk is one machine's opinion

An fsync'd segment survives a process crash. It does not survive the machine. Durability against machine loss means the log has to leave the machine, so two things go to object storage: base backups, and WAL segments as they close.

Which makes the interesting question not "do you archive" but "how far behind is the archive allowed to fall." Three bounds answer that:

• While the database is active , the log is forced to switch segments on a five minute interval. A segment closes and ships whether or not write volume happened to fill it. Low traffic databases do not get a worse recovery point than busy ones.
• When a database suspends , the open segment ships as part of the suspend path. Scale to zero is normal operation here, so the tail of the log cannot sit stranded on a machine waiting for a database that is not going to wake up on its own.
• When the compute layer shuts down , it takes a base backup of every active worker before exiting, under a bounded deadline. A deploy is a planned event, and a planned event should not cost you a recovery point.

Together those put the most recent restorable point within roughly five minutes of your latest write while the database is active.

Alongside the segments we write a recovery catalog: points carrying LSN, timestamp and the segments needed to reach them, plus the earliest and latest recoverable LSN and time. Recovery has to know which segments it needs before it starts pulling them.

4. Restorable: what recovery actually offers

Two operations, kept deliberately distinct because they answer different questions:

• Restore to a timestamp. POST /api/v1/tenants/{tenant_id}/pitr/restore restores by wall clock time, or to the latest recoverable point.
• Branch at an LSN. POST /api/v1/tenants/{tenant_id}/timelines/{timeline_id}/branch with a branch_lsn creates an independent timeline at a specific position in the log.

Both run the same machinery: pull the base backup, replay archived WAL forward to the target, bring up a fresh compute worker pointed at the result. A branch is a point in time restore you keep, addressed by LSN instead of by clock. It gets its own worker and its own backup cycle, and writes on it never reach the parent. Deleting one purges its base backups and archived WAL and reports the object count reclaimed.

The failure mode we thought hardest about is asking to restore to a moment after the last archived point. Quietly handing back older data is the worst available answer, because the restore reports success and the gap only surfaces later, in your data. That request returns a 409 with guidance instead.

Retention defaults to seven days and is configurable per tenant. Separately from the continuous path, nightly encrypted backups are replicated off site to EU located storage. Routine restore drills for that off site path are still being operationalized, and we would rather say so than imply a rehearsed process we do not have yet.

What we do not have yet

Precision cuts both ways, so here is the other half.

Multi-node quorum is not wired. Term fencing, shard placement and the quorum commit LSN calculation are written and tested. The commit function sorts acceptor flush LSNs and returns the highest LSN acknowledged by at least a quorum. It is not running across nodes, and the reason is worth stating plainly: the gRPC accept path returns unimplemented on purpose. The protobuf message for WAL data carries no term field, so accepting there would mean inventing a term, and an invented term either wrongly rejects a legitimate proposer after recovery or silently overwrites a higher one. That is the exact failure mode fencing prevents, so refusing beats faking success. Single node writes go over the binary protocol, which carries a real term. When quorum lands, it lands on fencing that is already there.

One region. One EU region today. Losing it is a restore from off-site backup, not a failover. We would rather say that than imply a topology we do not run.

Restorability introspection is incomplete. The endpoint that would tell you how far back you can currently restore returns a 501 saying it needs the WAL archive catalog. It could return a plausible looking window instead. A recovery bound you cannot verify is worse than no answer, because someone will plan against it.

Durability is scoped to the project timeline. Backups, branches and point in time recovery operate on the project's timeline, which is how the console labels them. Per database granularity within a project is roadmap.

Support is the two of us. We are two engineers running our own European infrastructure. The people who wrote the WAL path are the people who answer when you ask about it.

Every one of those is a real limit, and every one is easier to close than a durability model that was assumed rather than designed. That is the trade we made when we built the platform instead of assembling it.

The recovery semantics above are documented at docs.scalix.world/database/backups. To run a restore yourself, you can start without a card on a one time trial credit at scalix.world. If you would rather argue with the design first, we are on Discord, and that is a conversation we want.