Home / Azure

AzureTroubleshootingAdvanced

Sev1: outbound connections failing through Azure Firewall — SNAT port exhaustion

Diagnose and fix intermittent outbound failures caused by Azure Firewall running out of SNAT ports. Read the SNAT utilisation metric, find the noisy workload, add capacity with more public IPs or a NAT gateway, and reduce connection churn.

Deepak Bhardwaj2 min read

Applies toAzure Firewall Standard and Premium in hub-spoke networks, Azure NAT Gateway

Symptoms

  • Intermittent outbound timeouts from many spokes at once, worse at peak hours.
  • Apps report connection timed out or socket errors to internet APIs and PaaS public endpoints.
  • Inbound and internal traffic look fine.

1. Confirm SNAT exhaustion

FW=$(az network firewall show -g rg-hub -n afw-hub --query id -o tsv)
az monitor metrics list --resource $FW --metric SNATPortUtilization --interval PT5M --offset 6h \
  --aggregation Maximum -o table
az network firewall show -g rg-hub -n afw-hub --query "ipConfigurations[].publicIPAddress.id" -o tsv

Utilisation near 100% during the incident confirms it. Each public IP adds a fixed number of SNAT ports per backend instance of the firewall; check current documentation for the figure and for how the firewall scales out.

2. Find who is opening the connections

AZFWNetworkRule
| where TimeGenerated > ago(1h)
| summarize conns = count() by SourceIp, DestinationIp, DestinationPort
| top 20 by conns desc

If you use legacy diagnostics, query AzureDiagnostics with Category == "AzureFirewallNetworkRule" instead. Typical offenders: a batch job creating a new HTTP client per request, a crawler, or many pods calling one external API.

3. Quick relief

Add public IPs to the firewall (no downtime):

az network public-ip create -g rg-hub -n pip-afw-02 --sku Standard --allocation-method Static
az network firewall ip-config create -g rg-hub -f afw-hub -n ipcfg-02 --public-ip-address pip-afw-02

Warning

New outbound IPs change your egress addresses. Partners that allow-list your IPs will block traffic from the new ones; tell them first or use a NAT gateway with a public IP prefix you have already shared.

4. Durable fix: NAT gateway on the firewall subnet

Attaching a NAT gateway to AzureFirewallSubnet gives far more SNAT capacity and allocates ports dynamically.

az network public-ip prefix create -g rg-hub -n ippre-egress --length 30
az network nat gateway create -g rg-hub -n ng-hub --public-ip-prefixes ippre-egress --idle-timeout 4
az network vnet subnet update -g rg-hub --vnet-name vnet-hub -n AzureFirewallSubnet --nat-gateway ng-hub

Once attached, outbound internet traffic from the firewall uses the NAT gateway's IPs.

5. Reduce connection churn

  • Reuse HTTP clients and enable keep-alive (HttpClientFactory in .NET, a shared requests.Session in Python).
  • Use private endpoints for Azure PaaS so that traffic never goes through SNAT.
  • Cache responses from chatty external APIs.

Prevent it next time

  • Alert on SNATPortUtilization above 80% for 10 minutes.
  • Keep a NAT gateway on the firewall subnet from day one in new hubs.
  • Include egress connection counts in load tests.

Related fixes