The Story
Back in Article 1, I said we'd get back to this: once you've decided what goes in the cloud and who manages it, there's a third question that decides whether any of it actually works — how does your data get there?
A telecom customer I worked with was pushing sustained real-time Kafka streams past 850 Mbps between on-prem and the cloud, with big unpredictable spikes on top, over an AWS Site-to-Site VPN. A single tunnel is rated up to 1.25 Gbps. On paper they had headroom.
They still hit a wall. It wasn't AWS's fault.
The Problem
The wall was the on-prem VPN appliance's CPU — specifically, it couldn't keep up with IPsec encryption for every packet, made worse by two tunnels that weren't splitting traffic evenly. I confirmed this from CloudWatch tunnel metrics and CLI inspection on the appliance itself, not from guessing based on symptoms.
This is the trap for any platform team running high-throughput streaming over a public-internet VPN: the bottleneck almost never shows up where the bandwidth numbers say it should. Think of a VPN over the public internet like a public road — cheap, open to everyone, fine most days. But there's a single-lane toll booth at the entrance where every car gets checked. On a normal day, minor delay. On a heavy day, that booth is the road, no matter how wide the highway gets afterward. The toll booth here was IPsec — every packet individually encrypted and decrypted, with real CPU cost.
Why the Obvious Fixes Didn't Work
Two fixes made sense on paper. Both failed, for different reasons.
Scaling up meant an emergency maintenance window to go from 4 vCPUs to 8, plus more RAM. No help — IPsec/IKE encryption for a given tunnel is bound to a single worker thread, and that thread doesn't spread across cores just because more exist. The new cores sat idle. The one thread doing the work stayed pegged at 98%.
Scaling out meant adding two more VPN tunnels, on the theory that more tunnels meant more hash buckets and distributed crypto load. Also failed. ECMP hashes traffic by flow — source/destination IP, source/destination port, protocol — and this Kafka stream was one sustained flow. It didn't matter how many tunnels existed; that flow kept landing on the same path. More lanes feeding the same toll booth. Retries got worse, not better.
Direct Connect and ExpressRoute are the private road built just for you — no public traffic, no toll booth. They cost more and take time to build, but they behave predictably once they're in place. Most teams start on the public road because it's what's available on day one. This is what happens once your traffic outgrows it.
The Architecture
The diagram lays out the "two private roads, no shared map" problem: the on-prem VPN appliance with its two IPsec tunnels to AWS; an HAProxy layer using 16 secondary IPs to feed ECMP; Direct Connect (AWS) and ExpressRoute (Azure) running as parallel dedicated circuits, each with dual-location redundancy; independent BGP autonomous systems per cloud with zero shared visibility into each other's routing; and the asymmetric-path failure itself — a request leaving via Direct Connect, its response coming back over ExpressRoute, hitting a stateful firewall that drops it as an unmatched session.
Direct Connect took six weeks to provision. A single-provider option was ruled out almost immediately — this customer was already deliberately multi-cloud for capacity, cost, and reliability reasons that had nothing to do with this problem. The design became Direct Connect into AWS and ExpressRoute into Azure, each with redundant physical locations, driven by a real contractual requirement: 99.99% uptime.
Here's where this specific engagement took an unexpected turn. In week two of running both circuits live, a new failure mode showed up: an asymmetric BGP routing loop that broke stateful firewalls. AWS and Azure each run independent BGP autonomous systems with no visibility into each other's routing decisions. Each cloud picked its own "best" path back to the same on-prem address block — unaware the other had picked differently. A request could leave over Direct Connect while the response came back over ExpressRoute, and to a stateful firewall expecting a matched pair, that looks like a session that was never opened. Dropped.
This is exactly why the failure was subtle: the data plane — the actual Kafka traffic — looked completely healthy right up until the control plane's routing asymmetry collided with a stateful security device.
Fixing It
Prerequisites: two cloud providers with existing dedicated-circuit relationships (Direct Connect for AWS, ExpressRoute for Azure), BGP peering already established on both circuits, a stateful firewall in the on-prem path, and TLS termination capability at the compute tier rather than just at a central gateway.
Step 1 — buy time while Direct Connect provisions
- Pulled non-critical workloads off the IPsec tunnels to free capacity for the critical stream
- Deployed local HAProxy bound to 16 separate secondary IPs on the on-prem appliance
- Why it worked: ECMP hashes partly on source/destination IP, so splitting one flow across 16 source IPs made it look like sixteen distinct flows — forcing real distribution across both tunnels instead of one path absorbing everything
- Not off-the-shelf load balancing — HAProxy's only job here was manufacturing enough distinct source IPs to break ECMP's per-flow hash
- Rollback: trivial. Pull HAProxy out of the path, nothing left to unwind
Step 2 — fix the asymmetric routing
- Rejected the easy fix: loosening firewall statefulness to just allow asymmetric traffic — would have weakened a required security posture
- Real fix: BGP community tagging, MED tuning, and AS-path prepending on the Azure side, plus subnet-specific routing, to force one deterministic, symmetric path per address block
- Rollback: reversible by withdrawing the community tags and MED values — but only cleanly if you documented the pre-change route table first. Untangling which of several manual tweaks caused a new asymmetry after the fact is much harder than reverting to a known-good baseline
Step 3 — get encryption out of the choke point
Rather than reintroduce a central encryption bottleneck — recreating the exact problem that started this story — we moved TLS to the application layer, handled independently by each service instance. Spreading crypto across the compute tier kept continuous pod-to-pod encryption without a single-threaded chokepoint.
Security Considerations
A private circuit isn't an encryption exemption. This was telecom data under SOC 2, which requires encryption in transit regardless of whether the path is public or private — "it's on a dedicated circuit" doesn't satisfy that requirement by itself.
And the obvious fix for the BGP asymmetry — loosening firewall statefulness — was a security regression we explicitly rejected. It would have widened the firewall's tolerance for traffic patterns that look identical to spoofing and session-hijacking attempts. We took the slower, harder BGP-policy route instead of the fast one that degraded posture.
Tradeoffs
Dedicated circuits gave us predictable throughput and latency, and a real path to 99.99% uptime through dual-cloud, dual-location redundancy. In exchange: six weeks of lead time per circuit, and an entirely new failure mode — BGP asymmetry across independent cloud ASNs — that a single-provider VPN never has to deal with.
Moving TLS to the application layer removed the single-threaded bottleneck entirely, but it cost us centralized visibility. One gateway doing crypto means one place to audit; per-service TLS means per-service certificate management and rotation discipline — more operational surface area, not less.
What I'd Do Differently
Design application traffic to use multiple distinguishable connections from the start — one giant flow is always a point of contention, VPN or dedicated circuit alike. And put BGP routing policy templates in place before the first circuit goes live, not after an outage forces the question. Both failures in this story were, in hindsight, predictable the moment more than one path existed.
Has any of this actually been tested? Partially. Automatic BGP failover from the primary Direct Connect circuit to the secondary has been proven for real — a production fiber cut triggered it, and it worked. Full failover down to the standby VPN has only run in a controlled drill, never a real outage. Worth naming rather than assuming away.
And this design has a ceiling. It holds for two clouds and a handful of regions, but it doesn't scale by just repeating the pattern — add enough regions and individually-advertised address blocks and you hit a hard limit on how many routes a provider's edge router will accept. Past that limit it doesn't degrade gracefully; it silently drops excess routes or drops the BGP session entirely, taking every route with it. That blackholes active streams — the exact failure this architecture was built to prevent. The fix at that scale isn't more hand-tuned BGP policy per circuit. It's a managed transit gateway: a centralized hub that handles routing between clouds and regions so no circuit is negotiating its own path in isolation.
What's Next
Next in Cloud Without the Chaos: Article 4 — Data Sovereignty in Practice: What "It's in an EU Data Centre" Actually Covers (and Doesn't) — going deeper on the compliance angle this article touched on with SOC 2.
Top comments (0)