Home / Azure

AzureRunbookAdvanced

Sev1: App Service outbound calls time out under load. SNAT port exhaustion triage and fix

Outbound calls from App Service to databases or APIs start timing out at peak, then recover when load drops. Confirm SNAT port exhaustion, stop the bleeding by scaling out and fixing connection reuse, then move egress to a NAT gateway or private endpoints.

Deepak Bhardwaj4 min read

Applies toAzure App Service (Windows and Linux, Premium v3, Isolated v2), Azure Functions on App Service plans, VNet integration, NAT Gateway

Symptoms

  • Intermittent SocketException, connection timed out, An attempt was made to access a socket in a way forbidden by its access permissions or HTTP 5xx from calls to external APIs, Azure SQL, Redis or storage.
  • It gets worse at peak and clears when traffic drops.
  • CPU and memory look normal; the downstream service says it saw nothing.

Why it happens

Each outbound connection to the same destination IP and port uses a SNAT port on the shared front end. App Service gives each instance a limited pre-allocation (it varies; check current docs). Code that opens a new connection per request, or never closes them, burns through the pool. New connections then wait and time out.

Step 1: confirm (5 minutes)

Portal: App Service > Diagnose and solve problems > SNAT Port Exhaustion detector. It shows allocated and failed SNAT ports per instance.

Metrics and connections:

APP_ID=$(az webapp show -g rg-web -n web-prod --query id -o tsv)
az monitor metrics list --resource $APP_ID --metric "Connections" "Http5xx" \
  --interval PT1M --aggregation Maximum --offset 2h -o table

From the Kudu console (Windows: Debug console; Linux: SSH), count connections by remote endpoint:

# Linux container
netstat -ant | awk '$6=="ESTABLISHED" || $6=="TIME_WAIT"{print $5}' | sort | uniq -c | sort -rn | head

# Windows (Kudu PowerShell)
Get-NetTCPConnection | Group-Object RemoteAddress, RemotePort | Sort-Object Count -Descending | Select-Object -First 10 Count, Name

Hundreds of connections in TIME_WAIT to one endpoint is the fingerprint.

Step 2: stop the bleeding (minutes)

Scale out: each new instance brings its own SNAT allocation.

az appservice plan update -g rg-web -n plan-prod --number-of-workers 6

If one dependency is the hot spot and it supports private endpoints (Azure SQL, Storage, Redis, Key Vault), traffic through VNet integration to a private endpoint does not use App Service SNAT at all. That change takes longer than scaling but is often doable in the same day.

Tip

Restarting the app clears connections and looks like a fix. It is not; the pool fills again at the next peak. Use the time it buys you to do step 3.

Step 3: fix connection reuse (the real fix)

Stack Wrong Right
.NET new HttpClient() per request One HttpClient via IHttpClientFactory, or a static instance with SocketsHttpHandler.PooledConnectionLifetime
Node.js default agent without keep-alive (older versions) new https.Agent({ keepAlive: true, maxSockets: 50 })
Python requests.get() each time one requests.Session() reused
Java new client per call shared pooled client (Apache HttpClient pool, OkHttp)
Database open/close without pooling driver pooling on, sensible max pool size

.NET example:

builder.Services.AddHttpClient("payments", c => c.BaseAddress = new Uri("https://api.payments.example"))
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler {
        PooledConnectionLifetime = TimeSpan.FromMinutes(5),
        MaxConnectionsPerServer = 100
    });

Python:

import requests
session = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=20, pool_maxsize=50)
session.mount("https://", adapter)

Step 4: move egress off the shared SNAT

With regional VNet integration, route all outbound traffic through the VNet and attach a NAT gateway to the integration subnet. A NAT gateway gives 64,512 SNAT ports per public IP and you can attach several IPs.

az network public-ip create -g rg-web -n pip-nat-web --sku Standard
az network nat gateway create -g rg-web -n nat-web --public-ip-addresses pip-nat-web --idle-timeout 4
az network vnet subnet update -g rg-web --vnet-name vnet-web -n snet-appsvc-integration --nat-gateway nat-web
az webapp config set -g rg-web -n web-prod --generic-configurations '{"vnetRouteAllEnabled": true}'

On newer platform versions the setting is exposed as outbound VNet routing; check the current CLI help. Remember that the outbound IP changes to the NAT gateway's IP, so update partner allow-lists first.

Step 5: confirm

  • SNAT detector shows no failed allocations at the next peak.
  • Connection counts per endpoint stay stable instead of climbing.
  • Timeouts in Application Insights dependency telemetry drop to baseline:
dependencies
| where timestamp > ago(6h)
| summarize failures = countif(success == false), calls = count() by target, bin(timestamp, 5m)

Prevent it next time

  • Code review rule: no per-request HTTP or DB clients.
  • Private endpoints for Azure PaaS dependencies; NAT gateway for internet egress.
  • Load test at 2x peak and watch the SNAT detector before big launches.

Related fixes