Sev1: production VMs unreachable after an NSG or route table change — find the rule in minutes

#Symptoms

• An app tier can't reach its database, or users can't reach a VM, right after a change window.
• Ping/RDP/SSH or port checks time out; nothing changed on the VM itself.

#1. What changed?

az monitor activity-log list -g rg-network --offset 6h \
  --query "[?contains(operationName.value,'networkSecurityGroups') || contains(operationName.value,'routeTables')].{time:eventTimestamp, op:operationName.value, who:caller, status:status.value}" \
  -o table

If you find the change, the quickest safe fix is often to revert it. Still run the checks below to confirm.

#2. Ask Network Watcher if the packet is allowed

az network watcher test-ip-flow -g rg-app --vm vm-app01 \
  --direction Outbound --protocol TCP \
  --local 10.20.1.4:50000 --remote 10.30.2.5:1433

The output names the rule that allowed or denied the packet, for example securityRules/deny-all-outbound. Run it for both directions and for both ends: outbound on the source NIC and inbound on the destination NIC.

#3. See effective rules, not just the NSG you think applies

A NIC can have an NSG on the NIC and on the subnet. Both must allow the traffic.

az network nic list-effective-nsg -g rg-app -n vm-app01-nic \
  --query "value[].effectiveSecurityRules[?access=='Deny'].{name:name, prio:priority, dir:direction, dst:destinationAddressPrefix, ports:destinationPortRange}" \
  -o table

#4. Check the route the packet takes

az network watcher show-next-hop -g rg-app --vm vm-app01 \
  --source-ip 10.20.1.4 --dest-ip 10.30.2.5

az network nic show-effective-route-table -g rg-app -n vm-app01-nic -o table

If next hop is VirtualAppliance (a firewall) and the firewall has no rule or no return route, traffic is dropped there. If it is None, a UDR is black-holing the prefix.

Warning

Asymmetric routing is a classic after adding a UDR on only one side: the request goes through the firewall but the reply goes direct, and the stateful firewall drops it. Apply matching routes on both subnets.

#5. Fix it

Revert a rule:

az network nsg rule delete -g rg-network --nsg-name nsg-app --name deny-sql-temp
# or restore allow
az network nsg rule create -g rg-network --nsg-name nsg-app --name allow-sql-to-db \
  --priority 200 --direction Outbound --access Allow --protocol Tcp \
  --destination-address-prefixes 10.30.2.0/24 --destination-port-ranges 1433

Detach a bad route table from a subnet:

az network vnet subnet update -g rg-network --vnet-name vnet-spoke1 -n snet-app --route-table ""

Then rerun test-ip-flow and show-next-hop to confirm before telling the app team.

#6. Use connection troubleshoot for end-to-end proof

az network watcher test-connectivity -g rg-app --source-resource vm-app01 \
  --dest-address 10.30.2.5 --dest-port 1433

It reports hops and where the connection failed.

#Prevent it next time

• Turn on VNet flow logs and Traffic Analytics so you can see denied flows.
• Manage NSGs and route tables in IaC with peer review; avoid portal edits in production.
• Run test-ip-flow checks for key flows as a post-change test.
• Leave gaps in priorities (100, 200, 300) so emergency rules can be slotted in cleanly.