Intermittent Timeouts Every Evening, Zero Deployments: The Azure SNAT Port Exhaustion Story

The Problem

An order-processing API on Azure App Service worked fine all day. Every evening between 7 and 9 PM, during peak traffic, users got random 500s and timeouts. CPU was at 40%, memory was fine, and the database showed no slow queries. A restart “fixed” it for an hour, which made it feel like a ghost.

The app calls a third-party payment API and Azure services (Azure SQL, Storage). The errors in Application Insights looked like this:

System.Net.Http.HttpRequestException: Cannot assign requested address
System.Net.Sockets.SocketException: An attempt was made to access a socket in a way forbidden...

Those messages are the clue.

What Is SNAT Port Exhaustion?

When your app makes an outbound call to a public IP address, Azure translates your private address to a shared public one using SNAT (Source Network Address Translation). Each outbound connection to the same destination IP and port needs one SNAT port, and the supply is limited per instance. App Service preallocates only a small number per instance (128 by default, per Azure docs; check the current limits).

If your code opens new connections faster than old ones are released (they linger until the idle timeout, about 4 minutes by default), you run out of ports. New outbound calls then fail or hang until ports free up. That’s why it only appeared at peak load and why a restart helped.

Step 1: Confirm It’s SNAT

1. In the App Service, open Diagnose and solve problems and search for SNAT Port Exhaustion. The detector shows failed connection attempts over time.
2. Query Application Insights to see the failure pattern:

kusto
dependencies
| where timestamp > ago(24h) and success == false
| summarize failures = count() by target, resultCode, bin(timestamp, 15m)
| order by timestamp desc

Failures clustered on one or two targets (the payment API, for example) during peak hours point to SNAT.

Step 2: Find the Root Cause in Code

The culprit was a classic anti-pattern in the payment client:

csharp

// BAD: creates a new connection every call
public async Task<string> ChargeAsync(Order order)
{
    using var client = new HttpClient();
    var response = await client.PostAsJsonAsync("https://api.payments.example.com/charge", order);
    return await response.Content.ReadAsStringAsync();
}

Disposing HttpClient doesn’t immediately free the socket, so under load each request burns a new SNAT port. With hundreds of requests per minute across a few instances, the port pool was gone in seconds.

Step 3: The Fix (Three Layers)

Layer 1: Reuse connections in code (the biggest win)

csharp

// Program.cs
builder.Services.AddHttpClient("payments", c =>
{
    c.BaseAddress = new Uri("https://api.payments.example.com/");
})
.ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
{
    PooledConnectionLifetime = TimeSpan.FromMinutes(2)
});

csharp

// Usage: inject IHttpClientFactory
var client = _factory.CreateClient("payments");

Connections are now pooled and reused, so far fewer SNAT ports are needed. The same principle applies to Python (requests.Session) and Node (keep-alive agents).

Layer 2: Add a NAT Gateway for much more capacity

With regional VNet integration, route outbound traffic through a NAT Gateway. Each public IP on a NAT Gateway provides about 64,000 SNAT ports, shared dynamically, instead of a small fixed slice per instance.

bash
# Public IP + NAT Gateway
az network public-ip create -g rg-orders -n pip-nat --sku Standard --allocation-method Static
az network nat gateway create -g rg-orders -n natgw-orders \
  --public-ip-addresses pip-nat --idle-timeout 10

# Attach to the App Service integration subnet
az network vnet subnet update -g rg-orders --vnet-name vnet-orders \
  -n snet-appsvc-integration --nat-gateway natgw-orders

# Route all outbound traffic through the VNet
az webapp config set -g rg-orders -n orders-api --generic-configurations '{"vnetRouteAllEnabled": true}'

Bonus: you now have a single, static outbound IP, which the payment provider can whitelist.

Layer 3: Keep Azure-to-Azure traffic off the internet path

Use Private Endpoints for Azure SQL, Storage, and Key Vault so that traffic uses private IPs and consumes no SNAT ports at all.

bash

az network private-endpoint create -g rg-orders -n pe-sql \
  --vnet-name vnet-orders --subnet snet-private-endpoints \
  --private-connection-resource-id <sql-server-resource-id> \
  --group-id sqlServer --connection-name sql-conn

Remember to link the matching Private DNS zone (privatelink.database.windows.net) to the VNet.

Step 4: Prevent a Repeat

1. - Create an Azure Monitor alert on the NAT Gateway metric SNAT Connection Count (filter on failed state), plus App Service connection metrics.
2. - Add a dependency-failure alert in Application Insights for any target exceeding a threshold.
3. - Add a code review checklist item: “No new HttpClient() inside request paths.”
4. - Include NAT Gateway and private endpoints in your Bicep/Terraform baseline modules.

Results

In this kind of scenario, connection reuse alone typically removes most of the failures, and the NAT Gateway adds headroom for growth. Evening 500s drop to near zero, and no more restarts are needed as a workaround.

Key Takeaways

• “Works after restart, fails at peak” is a signature of connection or port exhaustion.
• SNAT limits are per instance and easy to hit when clients aren’t reused.
• Fix it in layers: connection pooling in code, NAT Gateway for capacity, private endpoints to avoid SNAT entirely.
• Monitor SNAT metrics before users find the problem for you.