Fencing a node that doesn't know it's dead: pgrac build log #2

pgrac is an open attempt to rebuild Oracle RAC's core machinery (shared-everything storage, multiple active nodes all writing one database, a cluster-wide change number) on top of PostgreSQL 16. Build log #1 laid out the four problems that fight back. This one is about the problem that turns a node failure into silent data corruption, and the first, deliberately modest, layer pgrac ships against it.

The failure mode

In a shared-nothing cluster an evicted node is mostly harmless: it owns its own disks, so the cluster routes around it. In a shared-everything cluster the same event is dangerous, because every node writes the same storage. Picture the classic split: node 2 misses heartbeats, the cluster declares it dead and remasters its work elsewhere, but node 2 is not actually dead. It is frozen on a long GC pause, or its interconnect NIC flaked, and it is about to wake up and finish the write it started. Now two nodes believe they own the same blocks, and shared storage will accept both writes. That is not a crash. It is corruption you find three days later.

Oracle RAC's answer is I/O fencing: before remastering a dead node's resources, you make certain it can no longer touch the storage, with STONITH, SCSI-3 persistent reservations, or a hardware watchdog. The node is fenced at a layer below its own software, because the whole point is that you cannot trust the dead node's software to behave.

That hardware layer is real work, and it is not what pgrac built first. What it built first is the layer above it: an in-process cooperative write-fence, now default-ON. The rest of this is precise about what that does and does not buy you, because "we have fencing" is the kind of claim that is worth less than nothing if it is overstated.

A fence needs an authority everyone can agree on

You cannot fence on local opinion, because the whole problem is that the dead node disagrees about being dead. Authority has to live on durable, shared, quorum-backed storage.

pgrac writes a small fence marker into the cluster's voting disks, the same disks used for membership. The marker is a compact tuple: a monotonic fence_epoch, a fence_generation tie-break, a fenced_dead_bitmap naming which nodes are declared dead, and the issuer's id, CRC-protected inside the voting slot. Reconfiguration bumps the epoch and submits the new marker. It is written to a quorum-majority of voting disks, and the reconfiguration proceeds only if a majority acknowledges. If the majority write fails, it fails closed: no event is published and no remastering starts. A minority of disks cannot amplify itself into an authority, because each disk preserves only its own marker, so a single stale disk cannot out-vote the quorum.

Every node runs a poller (qvotec) that reads all voting disks every couple of seconds, keeps only CRC-valid markers, orders them by (fence_epoch DESC, fence_generation DESC), and recognizes an authority only when it sees the same marker on a majority of disks. That majority requirement is the split-brain guard. It is what stops two partitions from each minting their own truth.

The hot path: check a token before touching storage

Consulting voting disks on every write would be absurd, so the poller distills the durable marker into a small local token in shared memory: the epoch this node is authorized for, a lease expiry, a self_fenced bit, and a copy of the dead bitmap. The write path consults only that token, through a pure, lock-free, allocation-free judge that is safe to call inside a critical section. A write is allowed only when:

enforcement_on
  && region_attached
  && epoch == authorized_epoch     // exact ==, not >=
  && now < lease_expire
  && !self_fenced

The == rather than >= is deliberate: a node holding a stale epoch is as dangerous as one explicitly fenced, so anything other than the exact current epoch is refused. Six storage entry points (create, unlink, extend, zero-extend, write, truncate) call this judge before writing shared storage. A fenced or stale node fails closed: ERROR 53R51 on the normal path, PANIC if it is caught in a critical section where it cannot safely back out. The lease makes silence safe: if a node is cut off from the voting disks and cannot refresh its token, the lease expires and the node fences itself. The safe default, when you have lost contact with the authority, is to stop writing.

Why default-ON was hard

The hard part was not fencing a dead node. It was not fencing a healthy one. Fail-closed-on-silence is correct, but it has a corollary: a healthy cluster, idle with nobody fenced, has no fence marker on the voting disks. No marker means the poller never refreshes the token, the lease expires, and every node fences itself, so the whole healthy cluster cannot write. The safety mechanism starves the happy path. That is why enforcement could not just be flipped on in the first cut.

The fix is a steady-state baseline marker: even with nobody fenced, the cluster continuously maintains a current, majority-agreed "all alive at epoch N" marker, so the token lease always renews and a healthy cluster writes normally, while a real fence still supersedes it the instant it is issued. Getting the priority right between baseline and real-fence markers is itself a split-brain hazard, since a stale baseline author must never overwrite a higher-epoch fence. That is what two hardening passes after the initial flip were about: a baseline author may not regress the epoch, and a node must engage its bring-up latch before it ever publishes a self-fencing token. Only after that does cluster.write_fence_enforcement default to on for a multi-node shared-FS cluster. A single-node instance, or one with no voting disks, auto-degrades to a no-op and opens cleanly. You do not pay for cluster fencing when there is no cluster.

What is demonstrated, and what is not

Demonstrated, in CI, on shared-FS clusters:

• On a 2-node and a 3-node cluster with enforcement default-ON, when a node is declared dead and self-fences, all six storage entry points reject the write with 53R51, verified end-to-end (t/271, t/272), with a pre-injection baseline write proving the test is not falsely green.
• The reconfiguration, marker-write, and fence path is exercised by a synthetic dead-node injection that proves the mechanism runs to completion. A counter delta proves the code path executed, rather than being dead code. This is the recovery acceptance mechanism-completion gate.

Deliberately not claimed yet:

• This is cooperative, not hardware, fencing. It fences a node that is still running the gate. It does nothing about a node whose software is wedged in a way that bypasses the gate. That is what STONITH, SCSI-3 PR, and cloud soft-fence are for, and that external fence plane is future work, not in this code.
• Faithful-crash auto-recovery is not proven in CI. The synthetic injection proves the mechanism; a real SIGKILL of a node followed by deterministic apply-through recovery is SKIP-with-limitation in the single-machine test harness, and labeled as such. That is not true N-node auto-recovery until it runs against a real multi-machine backend.
• No true 4-node CI demo. The 4-node recovery demo is a manual script that takes an external cluster connection string; with no such backend wired up it SKIPs with a reason rather than faking a pass.
• Being fenced is terminal in this stage. A fenced node comes back up in a non-serving mode: it refuses shared writes and waits for offline or cold-admin recovery. Online rejoin, a fenced node automatically returning to active service, is Stage 5 and later.
• Steady 2-node OLTP throughput and cross-node Cache Fusion block transfer remain Stage 5 problems. Nothing here changes that.

The honest one-liner: pgrac now has a default-on, quorum-backed, fail-closed cooperative write-fence, demonstrated to stop a self-fenced node from writing shared storage on 2- and 3-node clusters. It is the first of several fencing layers, with the hardware layer and faithful-crash recovery still ahead. That is a real step, and it is one layer of a problem that has several. I would rather undersell it than have a RAC veteran catch me overselling.

Code is in the open: cluster_write_fence.{c,h}, the storage gates in cluster_smgr.c, the durable marker and poller in cluster_qvotec.c, and the tests under src/test (t/271, t/272). https://github.com/sqlrush/pgrac · status board: https://pgrac.dev/features

If you have built I/O fencing for a shared-storage system and think the cooperative-first ordering is wrong, I want to hear it.