A database disaster recovery plan can look perfect on paper and still fail when a real incident occurs.
Why?
Because database recovery depends on more than backups and database configuration.
The network connecting the primary database to the recovery environment can become the actual bottleneck.
A recovery database network combines connectivity, database replication, standby infrastructure, and failover orchestration to move workloads from a primary database to a recovery database during an outage.
This article explains how to think about recovery database networks and, more importantly, how to test them properly.
What Is a Recovery Database Network?
A recovery database network is the infrastructure that connects a primary database environment with a standby or disaster recovery environment.
A simplified architecture looks like this:
Primary Application
|
v
Primary Database
|
| Replication
v
Recovery Network
|
v
Standby Database
|
v
Failover / Recovery
The architecture can include:
Site-to-site VPN
Dedicated leased lines
MPLS
Cloud-provider backbone connectivity
Synchronous replication
Asynchronous replication
Transaction-log shipping
Failover orchestration
The exact implementation depends on the database engine, workload, RPO, RTO, and infrastructure environment.
Why the Network Matters
It's easy to focus on the database itself.
You might test:
Database backup
↓
Restore
↓
Failover
↓
Application reconnect
But what happens if the replication network becomes slow?
Consider:
Normal network
18 ms latency
↓
Replication keeps up
Degraded network
340+ ms latency
↓
Replication falls behind
↓
Replication lag increases
↓
RPO target is missed
This is exactly the kind of scenario that can expose weaknesses in a DR architecture. ZProStudio describes a DR drill where network latency increased dramatically and asynchronous replication accumulated a roughly 40-minute backlog even though the documented RPO was under five minutes.
RPO vs. RTO
Before designing the network, understand two important recovery metrics.
RPO
Recovery Point Objective describes the amount of data loss the organization can tolerate.
Example:
RPO = 5 minutes
The recovery process should ideally ensure that no more than approximately five minutes of data is lost.
RTO
Recovery Time Objective describes how quickly the service needs to be restored.
Example:
RTO = 30 minutes
The system should ideally become operational again within approximately 30 minutes.
These metrics should influence the network design.
Choosing the Network Connection
There are several possible approaches.
- Site-to-Site VPN
A VPN over the public internet is generally cheaper and can be sufficient for systems where an RPO measured in hours is acceptable.
Advantages:
Lower cost
Flexible
Easy to deploy
Suitable for many non-critical workloads
Limitations:
Variable latency
Internet dependency
Shared network conditions
Less predictable throughput
- Dedicated Connectivity
Dedicated leased lines or similar connectivity can provide more predictable network performance.
They're more appropriate for workloads where:
Low latency matters
Consistent throughput matters
RPO requirements are strict
The workload is business-critical
The downside is cost.
- Cloud Cross-Region Replication
For workloads already running in a major cloud provider, native cross-region replication can be an attractive option.
Cloud providers operate their own backbone networks, which can provide more predictable connectivity than the public internet.
However, cloud infrastructure doesn't eliminate the need for network testing.
You still need to measure:
Cross-region latency
Throughput
Replication lag
Recovery time
ZProStudio notes that cloud-native replication can be a strong default for cloud workloads, but actual regional performance should still be verified rather than assumed.
Don't Confuse Advertised Bandwidth With Real Throughput
Suppose your network provider advertises:
1 Gbps
It is tempting to assume database replication can continuously use 1 Gbps.
That's not necessarily true.
Real throughput can be affected by:
Shared carrier infrastructure
Other network traffic
Encryption
Routing
Congestion
Packet loss
Network latency
ZProStudio describes a case where a supposedly 1 Gbps VPN connection delivered closer to 240 Mbps of sustained throughput during a regional outage after shared traffic was considered.
The lesson:
Measure the network under realistic load.
Synchronous vs. Asynchronous Replication
Replication strategy also affects network requirements.
Synchronous Replication
The primary and standby systems coordinate writes before considering the operation complete.
This can provide very low data-loss potential, but it requires reliable, low-latency connectivity.
High latency can directly affect application performance.
Asynchronous Replication
The primary system doesn't always wait for the recovery database to receive every change.
This can provide better performance across long distances, but replication lag can occur.
For example:
Primary
|
| Transaction log
v
Network
|
X High latency
|
v
Standby
Replication Lag ↑
If the lag becomes larger than the organization's RPO, the recovery plan isn't meeting its target.
The Most Important Test: Network Degradation
A complete network outage is easy to simulate.
But real incidents aren't always clean outages.
You may experience:
300 ms latency
Packet loss
Jitter
Reduced bandwidth
Intermittent connectivity
Network congestion
Therefore, DR testing should include degraded network conditions.
A useful Linux-based approach is to use tc to introduce controlled latency or packet loss during a test.
For example, conceptually:
Normal Network
↓
Start Replication
↓
Introduce Latency
↓
Monitor Replication Lag
↓
Measure RPO
↓
Restore Normal Conditions
↓
Measure Recovery
Always perform these experiments in an approved test environment with appropriate safeguards.
What Should You Measure?
A DR drill shouldn't simply produce:
FAILOVER: SUCCESS
You need actual measurements.
Track:
Metric Why It Matters
Latency Affects replication speed
Throughput Determines how much data can move
Packet loss Can disrupt replication
Replication lag Indicates potential data loss
RPO Measures tolerated data loss
RTO Measures recovery duration
Failover time Shows orchestration performance
These metrics tell you whether the recovery architecture actually meets business requirements.
Quarterly DR Testing
A useful baseline is to test the recovery environment at least quarterly.
Each test should ideally include more than a simple failover.
For example:
Quarterly DR Test
|
+-- Database failover
|
+-- Network latency test
|
+-- Packet-loss test
|
+-- Replication monitoring
|
+-- RPO measurement
|
+-- RTO measurement
|
+-- Application validation
This approach makes your DR plan measurable instead of theoretical.
Recovery Database Network Checklist
Use this checklist when reviewing an existing DR environment:
[ ] Primary database identified
[ ] Recovery database identified
[ ] Replication method documented
[ ] Network path documented
[ ] RPO defined
[ ] RTO defined
[ ] Actual throughput measured
[ ] Latency measured
[ ] Packet loss tested
[ ] Replication lag monitored
[ ] Failover tested
[ ] Application recovery tested
[ ] Backup restoration tested
[ ] Network degradation tested
[ ] Quarterly DR test scheduled
Common Mistakes
Mistake 1: Designing Around RPO Without Checking Network Capacity
You can't simply declare:
RPO = 1 minute
and assume the infrastructure can support it.
The database size, transaction volume, replication method, and network capacity must be capable of achieving that target.
Mistake 2: Testing Only a Complete Outage
A slow network can be more difficult than a completely dead network.
Test degradation.
Mistake 3: Trusting Provider Bandwidth Numbers
Contracted bandwidth doesn't necessarily equal sustained replication throughput.
Measure it.
Mistake 4: Ignoring Replication Lag
A successful failover doesn't automatically mean the RPO was achieved.
Always check how far behind the recovery database was before failover.
Conclusion
A recovery database network is a critical part of disaster recovery architecture.
The database engine may be perfectly configured, backups may be available, and failover automation may work correctly—but if the network can't move replication data quickly enough, your recovery objectives can still fail.
The best approach is:
Define RPO/RTO
↓
Choose Replication Strategy
↓
Design Network
↓
Measure Real Performance
↓
Test Under Degradation
↓
Measure Replication Lag
↓
Validate RPO/RTO
↓
Repeat Regularly
The key takeaway is simple:
A recovery network isn't proven until it has been tested under realistic failure conditions.
Don't wait for a real disaster to discover that your 1 Gbps connection doesn't actually deliver the performance your database needs.
Test the network before you need it.
Top comments (0)