Azure Load Balancer SNAT port exhaustion: prove it, then fix it with outbound rules or NAT Gateway

#Symptom

VMs behind a public Standard Load Balancer randomly fail on outbound calls: API timeouts, slow database connects, or Connection timed out during peak load. Inbound traffic stays healthy. Retries often work. The failures get worse as you add traffic, and sometimes get worse again when you add VMs.

#How the ports run out

Each outbound connection to the same destination IP and port needs its own SNAT port on the frontend IP. One frontend IP has about 64,000 ports to share among all the backend instances.

If you rely on the load-balancing rule for outbound access, Azure uses a default allocation that shrinks as the pool grows:

So scaling from 50 to 51 instances halves the ports on every VM. That's why scaling out can make this worse. After a connection closes, its port isn't reusable immediately, so short-lived connections use ports up faster than the connection count suggests.

#Step 1: confirm it with metrics

LB_ID=$(az network lb show -g rg-app -n lb-app --query id -o tsv)

az monitor metrics list --resource "$LB_ID" \
  --metric SnatConnectionCount \
  --filter "ConnectionState eq 'failed'" \
  --interval PT1M --aggregation Total -o table

az monitor metrics list --resource "$LB_ID" \
  --metric UsedSnatPorts AllocatedSnatPorts \
  --interval PT1M --aggregation Average Maximum -o table

Split the used-port metric by backend IP in the portal. One or two hot instances mean an application leak or a poor distribution rather than a shortage across the pool.

#Step 2: find who is using the ports

On Linux:

# Outbound connection states
ss -tan state established | wc -l
ss -tan state time-wait | wc -l

# Top remote destinations
ss -tan | awk 'NR>1 {print $1, $5}' | sort | uniq -c | sort -rn | head -20

On Windows:

Get-NetTCPConnection |
  Group-Object State, RemoteAddress, RemotePort |
  Sort-Object Count -Descending |
  Select-Object -First 20 Count, Name

Thousands of TIME_WAIT sockets to one endpoint usually mean the application opens a new connection per request. The classic cause is creating a new HttpClient per call in .NET, or opening a new database connection each time without a pool.

#Step 3: fix it in order

#1. Reuse connections

This is often the biggest win, and adding IPs doesn't replace it.

• Use one shared HTTP client with keep-alive enabled (IHttpClientFactory in .NET, requests.Session in Python).
• Use database connection pooling.
• Call Azure PaaS services through private endpoints or service endpoints, so that traffic doesn't use SNAT through the load balancer.

#2. Use explicit outbound rules

Move away from default allocation, so ports stay fixed as the pool grows. First stop the load-balancing rule from handing out SNAT:

az network lb rule update -g rg-app --lb-name lb-app -n rule-https --disable-outbound-snat true

az network lb outbound-rule create -g rg-app --lb-name lb-app \
  -n outbound-all \
  --address-pool be-app \
  --frontend-ip-configs fe-public \
  --protocol All \
  --outbound-ports 8000 \
  --idle-timeout 15 \
  --enable-tcp-reset true

Work out the sizing first: frontend IPs × about 64,000 ÷ max instances must be at least the per-instance port number you choose. Include instances added during autoscale and rolling upgrades.

Warning

Changing the port allocation can interrupt existing outbound flows, because allocations are recalculated. Do it in a maintenance window, and never set a per-instance value that leaves no room for scale-out.

#3. Use NAT Gateway for heavy outbound traffic

NAT Gateway attaches to the subnet and allocates ports dynamically from a shared pool. Once attached, it takes over outbound traffic from that subnet, ahead of load balancer outbound rules.

az network public-ip create -g rg-app -n pip-nat --sku Standard --allocation-method Static

az network nat gateway create -g rg-app -n nat-app \
  --public-ip-addresses pip-nat \
  --idle-timeout 10

az network vnet subnet update -g rg-app --vnet-name vnet-app -n snet-app \
  --nat-gateway nat-app

Caution

Outbound traffic now comes from the NAT Gateway's public IP. Before you attach it, ask partners and update any firewall allowlists that include the old address, or their connections will start failing.

Decision guide:

#Verify the fix

Repeat the Step 1 queries during peak load. Failed SNAT connections should be at zero, and used ports should stay well below allocated. With NAT Gateway, check the SNATConnectionCount metric on the NAT Gateway resource instead, because the flows no longer pass through the load balancer for outbound.

#Prevent it next time

• Alert on failed SnatConnectionCount and on used ports above about 75% of allocated.
• Add a connection-reuse check to code reviews for anything that calls external APIs.
• Plan outbound design before scale-out events, not during them.
• Related: App Service outbound SNAT exhaustion and VM default outbound access changes.