Replication Lag Is a Network Decision: Keeping Cross-Cloud Databases Honest

Replication Lag Is a Network Decision: Keeping Cross-Cloud Databases Honest

A database replica in another cloud can report healthy while falling steadily behind. The link is up, bandwidth looks ample, no packet loss alarms are firing, and yet the lag counter climbs from two seconds to twenty to two hundred. When a failover is needed, that replica is not a copy. It is an older version of the truth, and the gap was created as much by network choices as by the database itself.

Replication across clouds rewards teams that design for lag explicitly, in seconds and in business terms, rather than treating it as a database setting that will look after itself.

1. Define lag in time and in consequences, not in bytes

Bytes behind is a misleading number. A megabyte of small updates can take longer to apply than a gigabyte of bulk inserts, depending on indexes, constraints, and row by row work on the replica. What matters to the application is time: how old is the newest transaction visible on the replica, and what breaks if that age grows?

Set three thresholds before any link is ordered. A normal threshold, where replica reads are safe. A degraded threshold, where replica reads are stopped or labelled stale and writes on the primary are watched closely. And a failover threshold, beyond which promoting the replica means accepting named data loss, and a named person must approve it. Writing those thresholds down while everyone is calm prevents the worst conversation in incident response: discovering during a failure that nobody agreed what lag was acceptable.

Measure lag at the replica, from the replica point of view, and graph it continuously. Averages hide the problem. The 95th percentile and the maximum in each hour are where the risk lives.

2. Bandwidth is rarely the bottleneck people expect

Teams often size a cross-cloud link by total replication volume per day, then discover peak behaviour is different. Replication is bursty. A batch job, a backfill, a checkpoint, or a deployment that touches many rows can send an hour of changes in five minutes. If the link, a VPN gateway, or a firewall in the path is sized for the average, the burst queues, latency rises, and the queue itself becomes lag that drains slowly long after the burst ends.

Size for the burst you cannot reschedule, and measure the whole path, not just the circuit. Gateways, stateful firewalls, and NAT devices in the path each have throughput and session limits that behave like bandwidth when they saturate. A link diagram that stops at the circuit edge misses the devices that actually throttle replication in production.

Latency matters in a quieter way. Synchronous or semi synchronous replication pays the round trip on every commit. Moving a primary and its replica further apart can add milliseconds that look trivial in a network report and are decisive for transaction throughput. For asynchronous replication, latency still sets how quickly a problem is even visible on the other side.

3. Give replication its own path, or at least its own protection

Replication traffic sharing a path with backups, analytics exports, and general internet egress will lose at the worst moment, because those other workloads are also bursty and often larger. Where a dedicated path is not justified, give replication explicit protection: a reserved share of bandwidth, a separate queue, or a traffic class that is not starved when bulk transfers run.

Separation also helps diagnosis. If replication, backup, and user traffic share one counter, a lag spike has three possible causes and no quick answer. Separate counters per traffic class turn an argument into a reading. The same applies to routing: know exactly which gateways and firewalls replication traverses, keep that list short, and alert when the path changes. A silent reroute through a smaller gateway can halve throughput without any device reporting down.

4. Match the write pattern to the link you actually have

Some lag is created on the primary. Long transactions hold changes invisible to replication until they commit, then release them all at once. Frequent small checkpoints, large unindexed updates on the replica side, and schema changes applied without a replication plan all show up first as network shaped symptoms.

Practical steps that help without heroics: keep long running transactions visible with an age alert, not just a count. Apply schema changes in a tested order that replication can follow. Pause or throttle known bulk jobs when lag crosses the degraded threshold, automatically if possible, rather than hoping the replica catches up while the job continues. And test the replica apply rate separately from the link rate. If the replica can only apply changes at half the speed the primary generates them at peak, no network upgrade will close the gap.

5. Decide in advance what the system does when lag grows

A replica falling behind should trigger behaviour, not just a notification. Useful responses, in order of increasing cost, include stopping stale reads, shedding non essential writes, throttling batch work, and preparing users for a possible failover with data loss. Each response should be tied to the thresholds from the first section, with an owner and a way to reverse it when lag recovers.

Recovery needs design too. When a link returns after an outage, the replica will try to catch up at full speed. That catch up burst can saturate the same path production traffic needs, and can overload the replica disk. Rate limit catch up where the database allows it, watch primary retention carefully, and make sure transaction logs cannot expire before the replica has consumed them. The most expensive version of this failure is a replica that fell so far behind during an outage that it must be rebuilt from scratch, extending the window with no usable copy at all.

6. Test lag under fault, not just under load

Load tests show whether replication keeps up on a good day. Fault tests show whether it degrades predictably: add latency, drop the primary path and force the backup, pause the link for thirty minutes, then restore it and time the catch up. Record three numbers from each test: peak lag, time to return below the normal threshold, and whether any automatic protection actually fired.

Those numbers belong in the failover runbook, next to the thresholds. A team that knows catch up after a thirty minute outage takes two hours can make an honest decision at minute ten. A team guessing will either wait too long or promote a stale replica too early.

Teams that run private multi-cloud connectivity as a managed estate tend to treat replication paths, thresholds, and catch up behaviour as part of the network design itself, not as an afterthought left to the database team alone. That is the approach taken by CloudVerg, where VergMesh handles multi-cloud networking and VergForge helps keep the surrounding platform consistent, so database copies in different clouds stay copies in practice, not just in the architecture diagram.