Mirror Failover: When Your "Redundant" Setup Isn't Actually Redundant

I've spent the last decade running storage systems for small-to-mid-sized businesses, and if there's one lesson I've learned the hard way, it's this: a mirror without a tested failover path is just a more expensive way to lose data. You'd think that having two disks, two controllers, or two nodes means you're safe. Then the day comes when your primary silently degrades, and you discover your "automatic" failover was never wired to the thing that actually broke.

Let me walk you through what mirror failover really looks like in practice—the commands, the failure modes, and the edge cases that documentation glosses over.

The Difference Between "Mirrored" and "Highly Available"

First, a distinction that matters. A RAID-1 mirror is not the same as a failover cluster. RAID-1 gives you redundancy at the block level—if one disk dies, the other still has your data. But your operating system, your file system, and your application are still running on whatever controller was active. If that controller hangs, you've got a paperweight with a healthy disk inside.

True mirror failover means that when the active path goes away, a standby path takes over without human intervention and without data loss. That requires three things working together:

1. A detection mechanism (heartbeat, SCSI reservation, or network ping)
2. A promotion mechanism (the standby takes the active role)
3. A quorum or fencing rule (so you don't get split-brain—two nodes both thinking they're primary)

I've seen more split-brain incidents than I care to count. The classic symptom: both nodes write to the same LUN, and now you have two divergent versions of your database. That's not failover; that's corruption with extra steps.

A Practical Failover Workflow (Linux + DRBD)

Let me show you a real setup I use for a small file server. It's DRBD (Distributed Replicated Block Device) over two nodes, with a floating IP and a simple heartbeat script. Nothing fancy, but it works.

Here's the DRBD resource config, /etc/drbd.d/r0.res:

resource r0 {
    protocol C;
    on node1 {
        device    /dev/drbd0;
        disk      /dev/sdb1;
        address   192.168.10.11:7788;
        meta-disk internal;
    }
    on node2 {
        device    /dev/drbd0;
        disk      /dev/sdb1;
        address   192.168.10.12:7788;
        meta-disk internal;
    }
}

After you've initialized both sides with drbdadm create-md r0 and brought them up, the primary is set manually. But for failover, you want a script that does this on node2 when node1 goes silent:

#!/bin/bash
# /usr/local/bin/failover.sh - run on the standby node

DRBD_RESOURCE="r0"
FLOATING_IP="192.168.10.20"
PEER_IP="192.168.10.11"
PEER_PORT="7788"

# Check if peer is alive
if ! nc -z -w 3 $PEER_IP $PEER_PORT 2>/dev/null; then
    logger "Peer $PEER_IP unreachable, promoting DRBD resource"

    # Promote to primary
    drbdadm primary $DRBD_RESOURCE

    # Mount the filesystem
    mount /dev/drbd0 /mnt/data

    # Bring up the floating IP
    ip addr add $FLOATING_IP/24 dev eth0
    arping -c 3 -A -I eth0 $FLOATING_IP

    logger "Failover complete, floating IP $FLOATING_IP is now local"
fi

Now, here's the catch. That nc -z check is a single point of failure. What if node1 is alive but its network path to node2 is down? Then node2 promotes, and node1 is still running—now you have two primaries. That's split-brain. The fix is a fencing mechanism: a STONITH device (Shoot The Other Node In The Head), or at minimum, a shared quorum disk that only one node can hold a reservation on.

I learned this the hard way when a switch port died, and both nodes decided they were primary. The DRBD metadata got confused, and I had to manually resolve the conflict with drbdadm disconnect r0 and then drbdadm -- --force primary r0 on the side I trusted. Not fun at 2 AM.

Failure Modes You Won't Find in the Brochure

Let me list the ones that actually bite:

Silent I/O errors. The disk is still spinning, but it returns read errors. DRBD marks the peer as "Outdated" only after a certain number of failures. If your application doesn't check for read errors, you might be serving corrupt data for hours before anyone notices.

Stale heartbeat. Your failover script checks a port, but the service behind that port is hung. The port is open, the process is dead. Your script says "all good," but your users see timeouts. You need to check application-level health, not just TCP connectivity.

Partial network partition. Node1 can talk to node2, but node2 can't talk to node1. Both see each other as down. This is where quorum rules matter. With two nodes, you have no majority. I recommend a tie-breaker: a third node that acts as a witness, or a shared disk that only one node can lock.

Failover during a write. This is the nastiest one. If node1 crashes mid-write, and node2 promotes, node2's DRBD may have the data in its write-ahead log but not yet flushed to disk. Protocol C (synchronous) guarantees that a write is acknowledged only after both nodes have it in memory. But if node1 dies before acknowledging, node2 might have the data but not the acknowledgment. In that case, you must decide: serve the write (and risk losing it if node2 also dies) or reject it. I always choose to serve it, because a partial write is better than a full outage—but you need to know your application can handle a torn write.

The PowerShell Side: Windows Storage Spaces and Failover Clustering

Not everyone is on Linux. On Windows, I've used Storage Spaces with a two-node failover cluster. The setup is different, but the principles are the same. Here's a snippet to check the health of a cluster role and force a move if needed:

# Check cluster health and force failover if needed
$cluster = Get-Cluster -Name "FileCluster01"
$role = Get-ClusterGroup -Cluster $cluster -Name "FileServer"

if (-not $role.IsStateful) {
    Write-Host "Role is not stateful, checking node status..."
    $nodes = Get-ClusterNode -Cluster $cluster
    foreach ($node in $nodes) {
        if ($node.State -ne "Up") {
            Write-Host "Node $($node.Name) is down. Attempting to move role."
            Move-ClusterGroup -Name "FileServer" -Node $node.Name -Force
            break
        }
    }
} else {
    Write-Host "Role is currently owned by: $($role.OwnerNode.Name)"
}

The edge case here is that Move-ClusterGroup -Force can cause a brief outage if the storage is not in a healthy state. I've seen it fail with "The cluster resource could not be brought online because the dependent resource is not available." That usually means the disk resource is still marked as "Pending" or "Failed" after the node came back. The fix is to manually reset the disk resource:

Get-ClusterResource -Name "Cluster Disk 1" | Set-ClusterResource -Status Offline
Get-ClusterResource -Name "Cluster Disk 1" | Set-ClusterResource -Status Online

But here's the thing: if the disk is actually corrupt, that command will just bring it online into a corrupted state. Always run chkdsk /f on the underlying volume first, before you re-add it to the cluster.

Edge Case: Failover When the Primary Is Not Dead

Here's a scenario that stumps most people. The primary node is alive, the network is fine, but the application has crashed. In a traditional mirror setup, your failover script only triggers on node loss. So your app is down, but your mirror is perfectly healthy. You're not redundant; you're just backed up.

The fix is a health-check loop that monitors the application, not just the node. On Linux, that might be a systemd timer that checks if your NFS export is responding. On Windows, you'd use a resource monitor in the cluster that runs a custom DLL or PowerShell script. The script should return non-zero if the app is unresponsive, and the cluster will then fail over the role.

I once had a customer whose PostgreSQL instance was hung due to a deadlock. The node was up, the network was up, but the database was unresponsive. Their failover script didn't check the database, so the mirror stayed passive. They lost six hours of availability because no one had thought to test "what happens when the app hangs, not the OS."

Testing Your Failover (Do It Now, Not at 3 AM)

You cannot test failover in production. You can, however, test it in a staging environment that mirrors your production topology. Here's my minimum test suite:

1. Kill the primary node (power off, not graceful shutdown). Verify the standby promotes within the expected time.
2. Unplug the network cable from the primary. Verify the standby promotes but does not cause split-brain.
3. Kill the application process on the primary, not the node. Verify the health check catches it and fails over.
4. Write a file to the primary, then immediately kill it. Verify the file is present on the standby after failover, and that its content is intact.

If any of these tests fail, you have a bug in your failover logic. Fix it before you need it.

Related resource

If you're dealing with Windows-based tools and need a reliable terminal emulator that handles multiple sessions cleanly—especially when you're juggling cluster nodes and storage consoles—check out ys ll uniterm 1 9 5 on GitVaults. It's a solid utility for keeping your failover sessions organized without the bloat of full IDE environments.

Caveat

I've written this from my own experience, and I'm sure I've missed some corner cases or made assumptions that don't hold in your environment. If you spot an error, or if your failover setup handles something differently, please correct me. The whole point of this article is to get you thinking about what can break—so if you've seen a failure mode I didn't cover, I'd genuinely like to hear about it.