Home / Azure

AzureTroubleshootingAdvanced

Asymmetric routing through NVAs behind an Azure internal load balancer: dropped sessions, half-open TCP and how to make flows symmetric

Connections through a pair of third-party firewalls in Azure hang or reset at random. Prove the return path takes a different NVA, then fix UDRs, gateway routes, HA ports and SNAT so each flow stays on one appliance.

Deepak Bhardwaj5 min read

Applies toAzure Standard internal Load Balancer with HA ports, third-party NVAs (Palo Alto, Fortinet, Check Point, Linux), hub-and-spoke VNets, VPN/ExpressRoute gateways

Symptoms

  • Two NVAs (firewalls) sit behind a Standard internal load balancer (ILB) in the hub. Most traffic works, but some connections hang after the SYN, reset after a few seconds, or work only on retry.
  • The firewall logs show the SYN on NVA-1 and the SYN-ACK arriving on NVA-2, dropped as "no session" or "out of state".
  • Problems started after adding a second NVA, a new spoke, or an on-premises route.

Stateful firewalls must see both directions of a flow. If the request goes through NVA-1 and the reply through NVA-2, NVA-2 drops it. That is asymmetric routing.

How the path is decided

Every hop that can choose differently is a place where asymmetry starts:

Hop Decided by
Spoke VM -> hub UDR on the spoke subnet (next hop = ILB frontend IP)
ILB -> which NVA Load balancer hash of the flow (5-tuple)
NVA -> destination NVA's own routes and the hub subnet's UDRs
Gateway -> hub (return from on-prem) UDR on GatewaySubnet
Peered spoke -> spoke UDRs on both spokes

If one direction is sent to the ILB and the other goes directly to an NVA IP, or bypasses the NVAs entirely, the flow breaks.

Step 1: map the real path both ways

Use Network Watcher next hop from each side:

# Spoke VM to on-prem
az network watcher show-next-hop -g rg-spoke1 --vm vm-app01 \
  --source-ip 10.1.1.4 --dest-ip 192.168.10.20

# What a hub NVA does with the reply direction
az network watcher show-next-hop -g rg-hub --vm nva-01 \
  --source-ip 10.0.1.4 --dest-ip 10.1.1.4

Check effective routes on each relevant NIC:

az network nic show-effective-route-table -g rg-spoke1 -n vm-app01-nic -o table
az network nic show-effective-route-table -g rg-hub -n nva-01-trust-nic -o table

Look for:

  • A spoke UDR pointing to 10.0.1.4 (one NVA's IP) instead of the ILB frontend.
  • A missing or wrong UDR on GatewaySubnet, so traffic from on-prem goes straight to the spoke.
  • More specific system routes (peering, private endpoints) beating your UDR.

Step 2: confirm on the NVAs

Capture on both appliances at the same time, filtered to one client:

# Linux-based NVA example
tcpdump -ni eth1 host 10.1.1.4 and host 192.168.10.20

SYN on one box, SYN-ACK on the other = proven asymmetry.

Step 3: fix the routing

Route every direction to the ILB frontend, never to an NVA IP

az network route-table route create -g rg-spoke1 --route-table-name rt-spoke1 \
  -n to-onprem --address-prefix 192.168.0.0/16 \
  --next-hop-type VirtualAppliance --next-hop-ip-address 10.0.1.100   # ILB frontend

GatewaySubnet UDR for traffic coming from on-premises

az network route-table create -g rg-hub -n rt-gateway
az network route-table route create -g rg-hub --route-table-name rt-gateway \
  -n to-spoke1 --address-prefix 10.1.0.0/16 \
  --next-hop-type VirtualAppliance --next-hop-ip-address 10.0.1.100
az network vnet subnet update -g rg-hub --vnet-name vnet-hub -n GatewaySubnet --route-table rt-gateway

Warning

Don't disable gateway route propagation on the route table attached to GatewaySubnet, and don't add a 0.0.0.0/0 route there. Both can break the gateway. Add specific spoke prefixes only.

Use HA ports on the ILB

An HA ports rule (protocol All, port 0) balances every protocol and port, so all flows are handled the same way:

az network lb rule create -g rg-hub --lb-name ilb-nva -n ha-ports \
  --protocol All --frontend-port 0 --backend-port 0 \
  --frontend-ip-name fe-trust --backend-pool-name be-nva --probe-name probe-nva \
  --floating-ip false

For a flow that crosses the ILB in both directions through the same frontend, the load balancer keeps it on the same backend. Problems start when the two directions use different frontends or load balancers (for example a separate "untrust" ILB), because each makes its own choice.

Make the NVA forward

az network nic update -g rg-hub -n nva-01-trust-nic --ip-forwarding true

Also enable forwarding inside the OS or appliance.

Step 4: when symmetry can't be guaranteed, SNAT

If traffic enters through one load balancer and returns through another (common with separate public and internal ILBs, or east-west between spokes), configure the NVAs to SNAT traffic to their own interface IP. The reply then comes back to the NVA that owns that IP, whatever the routing.

Pattern Symmetric without SNAT?
Spoke ↔ on-prem through one ILB frontend, UDRs both sides Yes
Spoke ↔ spoke through the same ILB frontend Yes, if both spoke UDRs point to it
Internet inbound via public LB, return via internal LB No: SNAT on the NVA
Traffic to private endpoints Often not: SNAT recommended

SNAT hides the original client IP from the server; make sure logging on the NVA keeps it.

Step 5: health probes

If the probe sees an NVA as up while its data path is broken (or the reverse), flows land on a dead box. Probe a port that reflects the forwarding plane, not just the management plane, and check that the probe source (168.63.129.16) is allowed by the NVA.

az network lb show -g rg-hub -n ilb-nva --query "probes[].{name:name,port:port,proto:protocol,interval:intervalInSeconds}"

Prevent it next time

  • One route table design document: every subnet, every prefix, next hop always the ILB frontend.
  • Use Azure Policy or pipeline checks to block UDRs that point at individual NVA IPs.
  • Test new spokes with a flow both ways and a packet capture before handing them over.
  • Prefer SNAT for any path where two load balancers are involved.

Related fixes