Felicia Grace for BytesRack

Posted on Aug 27 Originally published at bytesrack.com

How to Build a Disaster Recovery Runbook for Bare-Metal Infrastructure (Step-by-Step Guide)

#devops #sysadmin #infrastructure #tutorial

Disaster recovery in the cloud has become almost push-button. Spin up a new instance in another AWS region, reattach a snapshot, update DNS, and you're back online in minutes. Bare-metal disaster recovery is a different discipline entirely.

When you own the hardware, there is no abstraction layer to fall back on. A bare-metal disaster recovery runbook has to account for physical provisioning, firmware and driver compatibility, network re-configuration, and manual verification steps that simply don't exist in a virtualized environment.

Skipping any of these steps turns a planned recovery into hours of improvisation during an outage.

This guide walks through building a practical, execution-ready runbook for recovering physical servers after hardware failure, data corruption, or a security incident—the kind of document your team can follow line-by-line at 3 a.m. without guessing.

Why Bare-Metal Disaster Recovery Is Different (And Harder)

Cloud DR and bare-metal DR solve the same problem—restoring service after failure—but the mechanics diverge sharply.

Physical hardware dependencies: A cloud VM can be recreated anywhere in seconds because the underlying hardware is abstracted away. A physical server recovery, by contrast, depends on actual components being racked, cabled, and powered before any restore can begin.
MAC addresses and licensing: Some software licenses, network configurations, and even certain OS activation schemes are tied to MAC addresses or hardware IDs. Replacement hardware can trigger licensing conflicts or require manual re-binding.
Network switch and VLAN configuration: In the cloud, virtual networking is redefined with an API call. On bare metal, someone has to physically or manually reconfigure switch ports, VLAN tagging, and routing to bring a replacement server into the network correctly and without colliding with production.
Hardware-agnostic restores: If the original server model is no longer available, you may be restoring an image built for one chipset and RAID controller onto entirely different hardware. This is where bare-metal restore (BMR) software has to inject the correct drivers during the recovery process, or the restored OS won't boot.

Because of this complexity, RTO/RPO for bare-metal environments tends to be measured in hours, not minutes—which is exactly why a documented, rehearsed runbook matters so much.

Pre-Requisites: What You Need Before Executing the Runbook

A runbook only works if the supporting infrastructure and access already exist before disaster strikes. Confirm the following are in place:

Recovery site tier: Define whether you're recovering to a cold site (empty facility, hardware provisioned on demand), a warm site (hardware present but not actively running production), or a hot site (fully mirrored, ready for near-instant failover).
Reliable bare-metal backup software: Tools such as Veeam, Acronis, or Datto need to support full image-based backups with driver injection for dissimilar hardware restores.
Comprehensive network documentation: IP address ranges, VLAN configurations, routing tables, and firewall rules must be documented and version-controlled.
Emergency access credentials: Out-of-band management credentials (iLO, iDRAC, IPMI) and physical access keys should be stored in a secure, offline-accessible vault.

The Step-by-Step Bare-Metal DR Runbook

This is the operational core of the plan. Each step should be assigned an owner and a target completion time.

Step 1: Incident Declaration and Initial Triage

The first action is confirming that what you're facing qualifies as a disaster. Not every outage warrants a full DR invocation.

Validate the failure through monitoring alerts or on-site inspection.
Formally declare the incident according to your response policy.
Notify the DR team and key stakeholders.
Open an incident ticket to log every action taken from this point forward.

Step 2: Hardware Provisioning & Network Isolation

Once the incident is declared, physical recovery begins.

Rack the replacement server(s) at the designated recovery site.
Connect power and network cabling per your documented rack diagram.
Power on and confirm the server responds via out-of-band management.
Isolate the recovery environment on a dedicated VLAN before doing anything else.

🚨 Critical Warning: Restoring a server with its original IP address while still cabled into the production network can cause IP conflicts, ARP table corruption, or routing loops.

Example of assigning a recovery VLAN on a managed switch:

interface GigabitEthernet1/0/12
 switchport mode access
 switchport access vlan 999
 description DR-RECOVERY-ISOLATED

Step 3: Booting the Recovery Environment (ISO/PXE Boot)

With the hardware isolated, boot into the recovery environment using one of the following methods:

Bootable USB media with your BMR vendor's recovery ISO.
Out-of-band virtual media via iLO/iDRAC, mounting the recovery ISO remotely.
PXE boot over the network from a dedicated DR PXE server.

Example of mounting a recovery ISO remotely via iDRAC racadm:

racadm remoteimage -c -l //10.0.99.5/iso/bmr-recovery.iso
racadm serveraction powercycle

Step 4: The Bare-Metal Restore (BMR) Process

This is where the actual data and OS restoration happens.

Launch the BMR software's recovery wizard.
Select the correct backup point.
Restore the OS, volume structure, and RAID configuration.
Inject hardware-agnostic drivers: If restoring to dissimilar hardware, inject the network, storage controller, and chipset drivers during the restore process.
Reboot into the restored OS and confirm it boots cleanly.

Step 5: Application & Data Verification

A server that boots is not the same as a server that's ready for production traffic.

Run database consistency checks (e.g., CHECKDB for SQL Server, pg_dump --schema-only for PostgreSQL).
Start application services in the isolated VLAN and test functionality.
Compare restored data against your last known-good state.

Step 6: Network Re-Routing and Go-Live

The final step brings the recovered server back into production.

Remove the server from the isolated recovery VLAN.
Assign it the correct production IP address(es).
Update DNS records and confirm propagation.
Route live traffic to the recovered server.

Best Practices for Maintaining Your Plan

A runbook that isn't maintained is a liability disguised as a safety net.

Test regularly: Run quarterly tabletop exercises and at least one full physical restore test annually.
Update documentation immediately: Every hardware refresh or VLAN change should trigger an update to the runbook.
Maintain air-gapped backups: Keep at least one backup copy physically or logically isolated from the network to protect against ransomware.

Conclusion

Bare-metal disaster recovery will never be as automated as cloud DR—but it can still be fast, predictable, and stress-free if the runbook is detailed, current, and rehearsed.

Don't wait for an outage to find out your documentation is outdated. Bookmark this step-by-step plan, or contact our infrastructure team to review your current disaster recovery strategy.

DEV Community