Home / Azure

AzureRunbookAdvanced

Sev1: ExpressRoute down — circuit, BGP peering and gateway checks in order

Work through an ExpressRoute outage layer by layer: circuit and provider state, BGP sessions, ARP and route tables, the virtual network gateway and the connection, with VPN failover as the fallback.

Deepak Bhardwaj3 min read

Applies toAzure ExpressRoute (provider and Direct), ExpressRoute virtual network gateways, Azure CLI 2.50+

Symptoms

  • On-premises cannot reach Azure VNets (or the reverse); the site-to-site VPN backup may or may not have taken over.
  • Monitoring shows BGP peer down or zero bits in/out on the circuit.

1. Circuit and provider state

az network express-route show -g rg-hybrid -n er-prod \
  --query "{circuit:circuitProvisioningState, provider:serviceProviderProvisioningState, sku:sku.tier, bw:serviceProviderProperties.bandwidthInMbps}" -o table
Field Expected If not
circuitProvisioningState Enabled Azure-side issue; check Service Health, open a case
serviceProviderProvisioningState Provisioned Provider deprovisioned or changed it; call the provider

2. BGP and ARP

az network express-route list-arp-tables -g rg-hybrid -n er-prod --path primary --peering-name AzurePrivatePeering -o table
az network express-route list-arp-tables -g rg-hybrid -n er-prod --path secondary --peering-name AzurePrivatePeering -o table
az network express-route list-route-tables -g rg-hybrid -n er-prod --path primary --peering-name AzurePrivatePeering -o table
  • Empty ARP table → layer 2 problem between your edge and the Microsoft edge (provider, VLAN tag, physical link). Escalate to the provider with the circuit service key.
  • ARP present, no routes → BGP session down: check ASN, MD5 key, peer IPs (/30 subnets) on your router against az network express-route peering show.
  • Routes present but missing your prefixes → you stopped advertising them, or exceeded the prefix limit (private peering has a route limit; the session drops when you exceed it — check the current limit for your SKU).
az network express-route peering show -g rg-hybrid --circuit-name er-prod -n AzurePrivatePeering \
  --query "{asn:peerAsn, primary:primaryPeerAddressPrefix, secondary:secondaryPeerAddressPrefix, state:state}"
az network express-route get-stats -g rg-hybrid -n er-prod

3. Gateway and connection

az network vpn-connection list -g rg-hybrid -o table           # ExpressRoute connections appear here too
az network vnet-gateway list-bgp-peer-status -g rg-hybrid -n ergw-prod -o table
az network vnet-gateway list-learned-routes -g rg-hybrid -n ergw-prod -o table

If the circuit has routes but the gateway has not learned them, the connection between gateway and circuit is the problem (deleted connection, authorization key revoked, gateway in a failed state). Reset the gateway only as a last resort:

az network vnet-gateway reset -g rg-hybrid -n ergw-prod

Warning

A gateway reset interrupts all connections through it, including VPN connections on a coexisting VPN gateway path. Announce it first.

4. Fail over

  • If you have a second circuit in another peering location, make sure on-premises prefers it (AS path prepend or local preference) and check that it learns the Azure prefixes.
  • If you have a VPN backup, confirm on-premises routes now point at the VPN and that the Azure VPN gateway learns your prefixes. ExpressRoute routes are preferred over VPN for the same prefix when both are up.

5. After recovery

  • Compare advertised prefix counts with the limit.
  • Check Connection Monitor tests and the BgpAvailability and ArpAvailability circuit metrics for the outage window.

Prevent it next time

  • Use both primary and secondary BGP sessions actively; a single-session design halves your resilience.
  • Alert on BgpAvailability below 100 for either path.
  • Keep a tested VPN or second-circuit path and run a failover drill at least yearly.

Related fixes