A cross-cloud link can pass every idle test and still fail the workload. Ping it at night and latency is flat and reassuring. Run the same test at 10 a.m., when replication, backups, and interactive traffic all share the path, and latency has doubled or tripled, jitter has widened, and the applications that suffer most are the small, chatty ones nobody thought to protect.
Nothing is broken in that picture. The link is doing exactly what queues do. The delay is not propagation and it is not bandwidth in the simple sense. It is time spent waiting in line.
Propagation is fixed, queueing is not
The latency a team usually designs for is propagation: the time light and electronics need to move a packet between two regions. That number is stable, measurable, and mostly out of anyone's control. Queueing delay is the other component, and it behaves in the opposite way. It is near zero on an idle link, and it grows sharply as utilisation climbs, especially past the point where bursts start arriving faster than the link can drain them.
That growth is not linear, and that is why it surprises people. A link at 40 percent average utilisation and a link at 85 percent average utilisation do not deliver latency that differs by a factor of two. They can deliver latency that differs by an order of magnitude during bursts, because averages hide the seconds when the queue is actually full. A one minute average that looks comfortable can contain ten seconds of saturation, and applications remember those ten seconds.
How a shared queue hurts the wrong traffic
On a shared interconnect, every flow waits behind every other flow in the same queue. A large replication stream does not mean to harm an interactive request, but in a single first in, first out queue, harm does not require intent. It only requires volume ahead in line.
Three patterns show up repeatedly:
- Bulk transfers fill the queue. Replication, backup, and object movement send large, sustained bursts. While those bursts drain, small requests sit behind megabytes of data that are in no hurry.
- Synchronised senders deepen the queue. Jobs that start on the hour, checkpoints that land together, and retry storms all arrive at once. Each sender is behaving reasonably on its own. Together they create a standing queue that takes minutes to clear.
- Retries add load to a full queue. When latency rises, clients time out and retry, and the retry traffic joins the same queue that caused the timeout. The link then carries both the original work and its duplicates, and the queue stays full longer than the original burst justified.
The result is head of line blocking at the network layer. The traffic with the tightest latency budget waits behind the traffic with the loosest one, and the monitoring often blames the application, because the application is where the timeout is visible.
Measure latency at load, not beside it
The standard mistake is measuring latency and throughput in separate tests. A throughput test saturates the link and reports a good number. A latency test runs on an idle link and reports a good number. Neither test reproduces production, where the question is what latency looks like while throughput is high.
A more honest test does both at once:
- Load the link with a realistic bulk flow, at the volume and burst pattern production actually produces, not a flat synthetic stream.
- While that load runs, measure the latency, jitter, and loss of small probe packets that mimic interactive traffic, in both directions.
- Record percentiles, not averages. The p50 will look fine while the p99 is where users live during a busy window.
- Repeat at the times of day when scheduled jobs collide, because a test run in a quiet window certifies a quiet window.
If latency at load is far worse than latency at idle, the problem is queueing, and adding bandwidth is only one of several possible answers, and often the most expensive one.
Separate traffic that cannot wait from traffic that can
The practical fix is to stop treating the link as one queue.
First, classify flows by latency sensitivity, not by team or by application name. Interactive requests, control plane calls, and health signalling belong in one class. Replication and backup belong in another. Bulk movement that is valuable but patient belongs in a third. Keep the classes few and the criteria written down, so a new service inherits a class instead of negotiating one during an incident.
Second, shape the patient traffic. Rate limit or schedule bulk transfers so they cannot consume the entire link in a single burst. A replication stream capped slightly below line rate often finishes at nearly the same time, because it stops inducing the timeouts and retries that were inflating its own workload, while leaving room for interactive traffic to pass without waiting behind it.
Third, give the sensitive class a protected share. Quality of service markings only matter if every device on the path honours them, so test that the markings survive tunnels, overlays, and provider boundaries. A marking that is stripped at the first encapsulation is a comment in a design document, not a control.
Fourth, size buffers with care. Larger buffers reduce drops and increase delay. Smaller buffers reduce delay and increase drops and retransmissions. There is no universally correct size, only a trade off that should be chosen deliberately for the traffic mix, and revisited when that mix changes.
Watch the queue, not just the link
Most link monitoring reports utilisation and errors. Neither shows queueing directly. Add signals that do:
- Latency percentiles for synthetic probes that run continuously across the path, alerted on sustained p99 movement rather than single spikes.
- Queue depth or discard counters where the platform exposes them, correlated with the latency record.
- A calendar view of scheduled bulk work, so a latency rise at a known job start is explained in seconds instead of investigated for an hour.
When those signals sit together, a familiar pattern becomes readable: the queue fills when three patient jobs overlap, the p99 rises, retries add load, and the incident that looked like an application failure resolves when one job is rescheduled by twenty minutes.
The design point
Cross-cloud latency problems are often queue management problems wearing a distance costume. Before paying to move workloads closer together or to widen a link, measure what the link does under its real load, separate the traffic that cannot wait from the traffic that can, and shape the patient flows so they stop taxing the impatient ones.
That separation of traffic classes, consistent shaping, and measurement across AWS, Azure, Google Cloud, and Oracle estates is part of the multi-cloud networking work described at CloudVerg, where VergMesh handles the mesh and routing layer and VergRail handles high volume transfer, alongside zero-trust security and SOC 2 readiness support.
A link that is busy is not a failing link. A link where everything waits behind everything else is a design that has not yet decided what matters most.
Top comments (0)