Sandeep G

Posted on Oct 3

Azure ExpressRoute Gateway

#architecture #azure #networking #expressroute

Your ExpressRoute circuit isn't the bottleneck. Your gateway is.

What the Azure ExpressRoute gateway does, how it probably works inside, and the limits that matter for private endpoints and FastPath.

Audience: engineers who design or operate Azure networking over ExpressRoute. You should already know VNets, BGP basics, and what a private endpoint is.
Reading time: about 20 minutes.

TL;DR

Your end-to-end throughput is the smaller of the circuit and the gateway. The gateway is a set of VMs with SKU limits on bandwidth, packets per second, flows, and reachable VMs.
Traffic to private endpoints gets up to half the gateway's normal capacity, is stateful, and can drop during routine maintenance and SKU upgrades.
FastPath sends traffic straight from Microsoft's edge to your VMs, bypassing the gateway, but only for supported traffic types.
FastPath does not cover private endpoints on provider circuits. On ExpressRoute Direct it's limited GA, needs enrollment, and covers only some services and regions. For most teams, all private endpoint traffic goes through the gateway.
To survive maintenance, set client TCP timeouts to 15–30 seconds, size for one fewer instance, use fixed scale units (not autoscale) when you have Private Link traffic, and alert on the gateway's metrics, not just the circuit's.

Example Network flow from on-premises to Azure vNet.

The problem

There are several limitations from ExpressRoute virtual network
gateway: the throughput cap from its SKU, and the resets from how it handles stateful private endpoint traffic during maintenance. knowing them in advance helps avoiding potential problems

This post covers:

What the gateway does: its control plane and data path.
How it does it, including a reasoned look at what probably runs inside.
Its limitations, especially during maintenance.
How FastPath changes the picture, and where it doesn't.

1. What the gateway does

Microsoft describes the gateway as having two purposes: exchanging IP routes and routing trafficbetween your networks. In practice, that splits into a control plane and a data path.

Control plane: routes

Runs BGP with the Microsoft Enterprise Edge routers (MSEEs). The gateway uses ASN 65515 and the MSEEs use 12076. Each gateway instance runs its own BGP sessions.
Learns on-premises prefixes. The limit is 4,000 routes on Standard/ErGw1Az and 9,500 on every other SKU.
Advertises VNet prefixes. It advertises the hub and peered spokes that use this gateway, up to 1,000 IPv4 prefixes. In a large hub-and-spoke you can stay under that limit with summarizedGatewayPrefixes.
Programs routes into your VNets. Every NIC in the hub, and in spokes that use the remote gateway, gets your on-premises prefixes with the gateway as next hop. That's what you see in a VM's effective routes.
Spreads traffic across circuits. A gateway can connect 4, 8, or 16 circuits depending on SKU, but no more than 4 from the same peering location. You steer between them with connection weights and BGP attributes such as AS-path prepend.

Data path: packets

Inbound: forwards traffic from the MSEE to VMs, private endpoints, NVAs, and VNet-injected PaaS services.
Outbound: forwards traffic from those resources back to the MSEE.
Tracks flows: the gateway keeps per-flow state and publishes Active flows and Max flows created per second as metrics. Private endpoint connections are explicitly stateful.

The control plane does little per-packet work. The data path is where the capacity limits come from: every packet, every new flow, and every reachable resource costs the gateway something.

2. How it does it

It's a set of VMs, and that explains a lot

When you create the gateway, Azure deploys gateway VMs into GatewaySubnet.

Speculation. Microsoft hasn't published the gateway's internal design.
This model is inferred from public limits and metrics.

The gateway is just a standard Azure VM wearing a "router" costume. The actual heavy lifting doesn't happen inside the VM; it happens underneath it on the hypervisor host, using the exact same networking stack as every other VM in Azure.

You can break it down into simple parts:

1. Control Plane inside the VM

Inside a legacy Windows-based VM architecture (GatewayTenantWork), there is a standard BGP process. Its primary job is not to move packets, but to talk to the Microsoft Enterprise Edge routers (MSEEs). It learns the routes from your on-premises network and passes them up to Azure’s Regional Network Controller. The controller then programs those routes into the Virtual Filtering Platform (VFP) across your VNet.

2. Data Plane outside the VM

The actual packet routing, encapsulation (VXLAN), and flow-tracking happens on the physical Hyper-V host running the gateway VM, handled by Azure's Virtual Filtering Platform (VFP). VFP is the programmable vSwitch on every Azure host. When a packet hits the gateway, VFP does the work to figure out which physical host the destination VNet IP lives on, encapsulates it, and sends it on.

A packet's journey (Speculation)

On-premises to an Azure VM, without FastPath:

Your router sends the packet on the circuit's VLAN to the MSEE.
The MSEE removes the VLAN tag and looks up the destination. It learned the VNet prefix from every gateway instance, so it has several equal next hops and picks one by hash (ECMP).
The MSEE encapsulates the packet, probably in VXLAN, and sends it over the Microsoft backbone to the host running the chosen gateway instance.
Virtual Filtering Platform (VFP) on that host decapsulates the packet and hands it to the gateway VM.
The gateway routes it to the destination in the VNet. On the way out, VFP finds the physical host that owns the destination IP, creates a flow entry, and encapsulates the packet to that host.
VFP on the destination host decapsulates it, applies NSGs, and delivers it to the VM.

The return path mirrors this. The VM's route table sends on-premises traffic to
the gateway, which sends it to the MSEE.

3. Limitations

Private endpoints get up to half the capacity

Microsoft states that throughput and control plane capacity to private endpoints might be reduced by half
compared to other resources.

What it means for sizing: plan for about 2× gateway capacity for private endpoint traffic. If you expect 4 Gbps into Storage through private endpoints, size the gateway closer to 8 Gbps.

Routine maintenance can interrupt traffic

Microsoft performs routine host and OS maintenance on the
gateway. Instances reboot one at a time. Here's what happens during the window:

Capacity drops. Microsoft states that control plane and data path capacity are reduced. The remaining instances carry the full load, so a gateway already running hot can drop packets.
Routes reconverge (inferred). The rebooting instance's BGP sessions go down, the MSEE removes it as a next hop, and traffic re-hashes onto the remaining instances. Expect a short blip on the flows that move.
Private endpoint connections break. Each private endpoint connection is routed through one back-end instance. When that instance reboots, the connection's state goes with it. Clients see hangs or resets until they time out and reconnect. Ordinary VM traffic usually recovers on its own. Private endpoint traffic doesn't.

The same intermittent private endpoint problems are documented during SKU upgrades. ErGwScale autoscaling operations can take up to 30 minutes, and Microsoft recommends fixed scale units instead of autoscaling when you have Private Link
traffic.

Asymmetric paths break private endpoint traffic

Microsoft requires that packets for a given 5-tuple use a single MSEE next hop except during maintenance. If your on-premises routers or firewalls keep moving a connection between the primary and secondary path, private endpoint connectivity fails. The instance holding the connection state never sees the other half of the traffic.

Other hard limits

No fragmentation. The maximum TCP/UDP packet size is 1,400 bytes, and the gateway drops fragmented packets. Lower the MSS/MTU on the client side, or use FastPath, which bypasses the gateway and supports fragmentation.
One ExpressRoute gateway per VNet. The ceiling is 40 Gbps with ErGwScale. Beyond that, split workloads across VNets, use ExpressRoute Direct with FastPath, or use Virtual WAN.
500 VNet peerings for a VNet that has an ExpressRoute gateway.
About 11,000 total routes across VNet address spaces, on-premises, and peering, with a maximum of 1,000 advertised by the gateway.
SKU changes aren't always in place. You can upgrade within a family (forexample ErGw1Az → ErGw3Az or → ErGwScale, which can take up to 2 hours without downtime). Moving from the legacy Standard/HighPerformance/UltraPerformance SKUs requires the migration tool. Anything else means delete and recreate, with downtime.

4. FastPath

FastPath
sends traffic from the MSEE directly to VMs in your VNet, bypassing the gateway. The gateway still runs BGP and exchanges routes. It just stops forwarding the packets.

Figure 2. FastPath removes the gateway from the data path, not from the control plane

You get lower latency (one less hop), throughput limited by the circuit rather than the gateway SKU, and support for fragmented packets.

How it probably works :

ExpressRoute FastPath works by moving the data-plane routing intelligence out of the virtual gateway and directly into the hardware of the Microsoft Enterprise Edge (MSEE) routers.

Direct Route Injection: Azure's Software Defined Networking (SDN) controller maps every IP address in your Virtual Network to its underlying physical Hyper-V host. It then injects this routing map directly into the hardware tables of the MSEE.

The Gateway Bypass: When a packet arrives from your on-premises network, the MSEE does not forward it to the VNet Gateway. Instead, the MSEE performs the network encapsulation itself and routes the packet directly to the physical server hosting your destination VM.

Hardware Memory Constraints: Because FastPath relies on physical ASIC memory on the MSEE rather than a software-based virtual router, it is bound by strict hardware capacity limits. This is why IP limits are tied to the physical circuit or port, not the gateway. On ExpressRoute Direct, you own the entire physical port, meaning all of its hardware routing memory is dedicated to your environment, allowing for vastly higher IP limits.

The VNet Gateway is still required to handle the BGP control plane (exchanging routes between Azure and your on-premises routers), but FastPath completely removes it from the data path, drastically reducing latency and bypassing the gateway's bandwidth bottlenecks.

When you hit the IP limit, FastPath stops programming new routes and traffic silently falls back to the gateway, where all the gateway limits apply again. Alert on the FastPathRoutesCount metric before you reach the limit.

What FastPath covers

Traffic	ExpressRoute Direct	Provider circuit
Hub VNet VMs, IPv4	✓	✓
Hub VNet VMs, IPv6	✓	✗
Spoke VNets via peering	✓ (same region only)	✗
User-defined routes (UDRs)	✓	✗
Private endpoints / Private Link	Limited GA, enrollment required	✗

Even where FastPath is enabled, these always go through the gateway: internal load balancers and PaaS services in spokes, Azure Firewall in spokes, DNS Private Resolver in spokes, and anything cross-region.

5. FastPath doesn't help your private endpoints

⚠️ On a provider circuit, private endpoint traffic never uses FastPath.
Every packet goes through the gateway VMs, with up to half the gateway's
capacity, pinned to a single instance, and exposed to maintenance
interruptions.

The only exception is ExpressRoute Direct, and it comes with conditions:

It's limited GA. You must enroll, and deployment takes 4–6 weeks after approval.
It covers only some services: Azure Storage, Cosmos DB, Key Vault, and third-party Private Link services. SQL, for example, isn't on the list.
It works only in specific regions and same-region only. Cross-region private endpoint traffic goes back through the gateway.
It supports up to 100 Gbps to a single availability zone on 100/400 Gbps ports.
Even then, Microsoft still warns of intermittent connectivity during gateway maintenance for private endpoints.

In practice: unless you're on ExpressRoute Direct and enrolled and using a supported service in a supported region, plan as if FastPath doesn't exist for private endpoints. The gateway SKU is your private endpoint throughput ceiling.

The worst case is a high-volume private endpoint workload, such as bulk Storage or database replication, over a provider circuit. That traffic gets half the gateway's capacity, can't use FastPath, and breaks during maintenance unless the clients are built to retry.

Key takeaways

The gateway is a set of VMs with SKU limits. Size it explicitly. It's usually the real bottleneck, not the circuit.
Private endpoints cost about twice as much per unit of gateway capacity and are stateful.
Maintenance is routine, and it interrupts private endpoint connections. Short TCP timeouts and retries are mandatory, not optional.
FastPath is great for VMs and essentially unavailable for private endpoints unless you're on ExpressRoute Direct with enrollment.
Monitor the gateway itself. Circuit dashboards won't tell you it's saturated.

DEV Community