Path control with two ExpressRoute circuits and a VPN backup: connection weight, AS path prepending and local preference

The goal: a VNet connected to two ExpressRoute circuits (for example a primary in one peering location and a secondary in another) plus a site-to-site VPN as last resort. Traffic in both directions should use the primary circuit, fail over to the secondary, and only use the VPN when both circuits are down.

Out of the box you usually get something else: ECMP across both circuits from Azure, a single preferred circuit on-premises, and return traffic that takes a different path from the request. Stateful firewalls on-premises drop the half-flows. The result is intermittent timeouts that look like an application problem.

Each direction is decided by a different router, so set path preference on both sides.

#How each side chooses

#Azure → on-premises (decided by the Azure gateway)

1. Longest prefix match first. A more-specific route wins regardless of where it came from.
2. For the same prefix, ExpressRoute is preferred over site-to-site VPN. This applies to VNets linked through private peering.
3. Between ExpressRoute connections for the same prefix, connection weight is considered before AS path length. Higher weight wins.
4. If weight and AS path are equal, the gateway load-balances across circuits (ECMP).

#On-premises → Azure (decided by your edge routers)

Azure advertises the VNet prefixes over every circuit and over the VPN BGP session. Your routers choose using their normal BGP rules. Local preference is the usual tool: the default is typically 100 and higher wins. Over ExpressRoute private peering the Microsoft side appears as AS 12076. Over the VPN you see the VPN gateway's ASN (65515 by default).

#Step 1: see what you have now

RG=rg-hub-net
GW=vgw-er-hub-weu

# Routes the ExpressRoute gateway has learned, with AS path and weight
az network vnet-gateway list-learned-routes -g $RG -n $GW \
  --query "value[].{prefix:network, nextHop:nextHop, asPath:asPath, weight:weight, peer:sourcePeer}" -o table

# Connection objects and their routing weight
az network vpn-connection list -g $RG \
  --query "[].{name:name, type:connectionType, weight:routingWeight}" -o table

# What a workload NIC actually uses
az network nic show-effective-route-table -g rg-app-prod -n nic-app-01 \
  --query "value[?nextHopType=='VirtualNetworkGateway'].{prefix:addressPrefix[0], nextHops:nextHopIpAddress}" -o table

Output shapes vary by CLI version. If a query returns nothing, run the command without --query and adjust the paths (for example drop the leading value). If a prefix shows two next-hop IPs on the NIC, Azure is using ECMP across circuits for it.

Check the circuit side, including what on-premises is advertising into each circuit:

az network express-route list-route-tables -g rg-er -n er-ams-primary \
  --peering-name AzurePrivatePeering --path primary -o table
az network express-route list-route-tables-summary -g rg-er -n er-ams-primary \
  --peering-name AzurePrivatePeering --path primary -o table

Run both commands for --path secondary as well, and for the second circuit. Every circuit should receive the same on-premises prefix set. If one circuit receives an extra more-specific prefix, it wins for that range whatever weights you set.

#Step 2: make Azure prefer the primary circuit

Option A is connection weight. It is simple and stays in Azure:

az network vpn-connection update -g $RG -n conn-er-ams-primary   --routing-weight 100
az network vpn-connection update -g $RG -n conn-er-lon-secondary --routing-weight 0

az network vpn-connection manages ExpressRoute connection objects as well as VPN ones. Confirm afterwards with the list command above that routingWeight changed on the ExpressRoute connections. In PowerShell, set RoutingWeight on the object from Get-AzVirtualNetworkGatewayConnection and save it with Set-AzVirtualNetworkGatewayConnection.

Option B is AS path prepending from on-premises. Use it when several VNets or regions share the circuits and you want one policy at the edge rather than per-connection settings in Azure:

! Cisco IOS-XE: advertise on-prem prefixes over the secondary circuit with a longer AS path
route-map TO-AZURE-SECONDARY permit 10
 match ip address prefix-list ONPREM-TO-AZURE
 set as-path prepend 65010 65010 65010
!
router bgp 65010
 address-family ipv4
  neighbor 192.168.20.2 route-map TO-AZURE-SECONDARY out

Weight is evaluated before AS path length. If both are in use, a weight set in Azure overrides your prepends. Choose one method per gateway and record which one it is.

#Step 3: make on-premises prefer the primary circuit, then the VPN last

! Cisco IOS-XE: rank routes received from Azure
route-map FROM-AZURE-PRIMARY   permit 10
 set local-preference 200
route-map FROM-AZURE-SECONDARY permit 10
 set local-preference 150
route-map FROM-AZURE-VPN       permit 10
 set local-preference 50
!
router bgp 65010
 address-family ipv4
  neighbor 192.168.10.2 route-map FROM-AZURE-PRIMARY in
  neighbor 192.168.20.2 route-map FROM-AZURE-SECONDARY in
  neighbor 10.250.0.30  route-map FROM-AZURE-VPN in

# Junos equivalent for the primary group
set policy-options policy-statement FROM-AZURE-PRIMARY then local-preference 200
set protocols bgp group azure-er-primary import FROM-AZURE-PRIMARY

Local preference only applies inside your AS. If several edge routers terminate the circuits, carry it over iBGP to every router that makes a forwarding decision.

#Step 4: keep the VPN from winning by accident

The VPN backup takes over silently when it carries a more-specific prefix than ExpressRoute, because longest prefix match is applied before the ExpressRoute-over-VPN preference.

Warning

Changing weights, prepends or local preference moves live flows. Long-lived TCP sessions through stateful firewalls will reset when their return path changes. Make the change in a window, one direction at a time, and check both ends between steps.

#Step 5: prove it and test failover

Checks after the change:

• list-learned-routes shows the primary circuit's routes with the higher weight (or shorter AS path), and the NIC effective routes show a single next hop per on-premises prefix.
• On-premises, show ip bgp <azure-prefix> lists the primary circuit as best with local preference 200.
• A traceroute or flow logs from both ends show the same circuit. VNet flow logs on the Azure side and firewall session logs on-premises are the quickest way to confirm symmetry.

Failover test: shut the BGP session to the primary circuit on your edge router (both primary and secondary links of that circuit), and confirm that:

1. Azure's learned routes now come only from the secondary circuit.
2. On-premises best paths move to the secondary.
3. Application sessions recover within your target.

Then shut the secondary as well and confirm traffic moves to the VPN. Restore in reverse order. Check prefix counts against the limit on each session. Private peering drops the BGP session if you advertise more prefixes than the circuit allows, and the limit depends on the SKU and the Premium add-on, so check the current limits.

#Prevent it next time

• Write the intended path for each direction into the network design: primary, secondary, last resort, and the mechanism (weight or prepend) for each.
• Keep identical prefix sets on every path, enforced by one prefix-list template on all edge routers.
• Alert on changes to learned-route counts and on ExpressRoute BGP availability for each circuit.
• Test failover at least twice a year, and after every change to edge router policy.
• Manage connection weights in IaC, so a rebuilt connection does not come back at the default weight.