What happened
Azure's post-incident review (tracking ID 7Q30-010) describes a gateway service degradation from 20:30 UTC on 30 September to 02:15 UTC on 1 October 2026:
- A rolling OS update for gateways resumed after a pause and sent a surge of requests to a regional control plane. An earlier change had already made configuration operations slower, so the control plane was overwhelmed.
- Management operations failed for ExpressRoute gateways, Application Gateway, Azure Firewall, Bastion, Virtual WAN and VPN Gateway.
- Some ExpressRoute gateways lost data-plane connectivity. Restarting instances reported healthy before their configuration was restored, which made the outage longer. Azure VMware Solution customers could also lose connectivity.
- 33 regions were affected, including Central US, West Europe, North Europe, UK South, East Asia and Southeast Asia.
The review says customers couldn't have avoided the impact. What you control is whether your design has another path and whether you find out quickly.
1. Know which kind of failure you can survive
| Failure | Survives with |
|---|---|
| One ExpressRoute circuit or provider edge down | Two connections on the circuit (primary/secondary) - the default |
| Whole peering location down | A second circuit at a different peering location |
| ExpressRoute gateway in a region unhealthy (this incident) | A path that doesn't use that gateway: a gateway in another region, or a site-to-site VPN |
| Region-wide control plane problem | Workloads that don't need configuration changes to keep running; changes frozen during the incident |
Most designs cover the first row. This incident hit the third.
2. Add a backup path
Site-to-site VPN as ExpressRoute backup. A VPN gateway in the same hub can carry traffic when ExpressRoute routes are withdrawn. Azure prefers ExpressRoute routes over VPN routes for the same prefix, so the VPN is used only when ExpressRoute isn't advertising.
Points to check:
- The on-premises side must also prefer ExpressRoute (for example with BGP local preference or AS path prepending on the VPN) and fail over when ExpressRoute routes disappear.
- VPN bandwidth is lower. Decide which traffic must keep working and size for that.
- Test it. A backup path that has never carried traffic usually has a firewall rule or route missing.
Cross-region gateway. For critical spokes, connect a second ExpressRoute circuit (or the same circuit, if your SKU allows) to a gateway in a paired or nearby region, and peer the spokes to both hubs. With Virtual WAN, hubs in two regions give a similar result.
3. Detect it quickly
Data-plane metrics on the gateway and connection:
# list metric definitions for the ExpressRoute gateway
az monitor metrics list-definitions --resource <expressroute-gateway-resource-id> -o table
Alert on (names can differ slightly by SKU; check the list above):
- BGP availability and ARP availability on the circuit
- Bits in/out per second on the connection dropping to near zero during business hours
- Count of routes learned from on-premises falling sharply
Add an active probe from both sides (a Connection Monitor test from an Azure VM to an on-premises host, and the reverse from your monitoring). Metrics can look fine while traffic doesn't pass, which is what the "healthy before configured" defect caused here.
Then add Service Health alerts for ExpressRoute, VPN Gateway and Virtual WAN in your regions, so the incident notice reaches the on-call channel rather than an inbox.
4. During an incident like this
- Check Azure status and Service Health first. If it's a platform incident, don't start reconfiguring gateways. Management operations were failing, and half-applied changes make recovery harder.
- Confirm which path traffic is using (
az network nic show-effective-route-tableon a spoke VM) and whether the backup took over. - If the backup didn't take over automatically, use the pre-written manual step (for example, raising the VPN's preference on-premises) rather than improvising.
- Record times and symptoms for your own review and for any support case.
5. Practise
Run a failover test each quarter: disable the ExpressRoute connection (or withdraw routes on-premises) in a maintenance window and confirm that critical flows move to the backup path and back.
Checklist
- Backup path that doesn't depend on a single regional ExpressRoute gateway
- On-premises routing prefers ExpressRoute and fails over to the backup
- Connection Monitor probes in both directions
- Metric alerts on BGP/ARP availability and traffic drop
- Service Health alerts for networking services in your regions
- Change freeze rule during platform incidents
- Quarterly failover test with results recorded