Home / Azure

AzureTroubleshootingIntermediate

Sev1: Azure Load Balancer marks every backend down — health probe and data path fixes

Fix an outage where Azure Load Balancer stops sending traffic because all backends fail the health probe. Read the probe and data path metrics, check NSGs for the AzureLoadBalancer tag, and fix the probe endpoint.

Deepak Bhardwaj3 min read

Applies toAzure Standard Load Balancer (public and internal), VMs and VM scale sets

Symptoms

  • Clients time out on the load balancer frontend IP; backends are running and respond locally.
  • The Health Probe Status metric drops to 0 for all backends.
  • It started right after an NSG change, an OS firewall change, an app deployment or a probe edit.

1. Read the two metrics

Metric (portal name) Metric ID Low value means
Health Probe Status DipAvailability Backends fail the probe
Data Path Availability VipAvailability Frontend path broken (platform or config) 
LB=$(az network lb show -g rg-app -n lb-app --query id -o tsv)
az monitor metrics list --resource $LB --metric DipAvailability --interval PT1M --offset 1h \
  --dimension BackendIPAddress -o table

If data path is healthy and probe status is 0, it is a probe problem: continue below.

2. What exactly is the probe checking?

az network lb probe list -g rg-app --lb-name lb-app -o table
az network lb rule list -g rg-app --lb-name lb-app --query "[].{rule:name, probe:probe.id, fe:frontendPort, be:backendPort}" -o table

Test from a backend itself:

curl -sv http://localhost:8080/healthz      # HTTP probe: must return 200
ss -ltnp | grep 8080                         # TCP probe: must be listening
Finding Fix
Health path returns 301/302/401/500 Probe needs a 200; point it at an anonymous health endpoint
App listens on 127.0.0.1 only Bind to 0.0.0.0 or the NIC IP
Port changed in the release Update the probe and rule, or the app config
Health endpoint checks a down dependency Make the probe check the instance, not the database

3. NSG and OS firewall

Probes come from 168.63.129.16, which NSGs represent with the AzureLoadBalancer service tag. The default AllowAzureLoadBalancerInBound rule is easy to override with a higher-priority deny.

az network nsg rule list -g rg-app --nsg-name nsg-app --include-default -o table \
  --query "sort_by([], &priority)[].{p:priority, name:name, access:access, src:sourceAddressPrefix, port:destinationPortRange}"

On Windows, check the firewall allows the probe port; on Linux check iptables/nftables rules.

az network nsg rule create -g rg-app --nsg-name nsg-app -n Allow-LB-Probe --priority 100 \
  --source-address-prefixes AzureLoadBalancer --destination-port-ranges 8080 --access Allow --protocol Tcp

4. Backend pool membership

az network lb address-pool show -g rg-app --lb-name lb-app -n bepool \
  --query "backendIPConfigurations[].id" -o tsv

A VMSS model update or NIC change can drop instances from the pool. Re-add them, then run az vmss update-instances if the scale set model was changed.

Prevent it next time

  • Probe a dedicated /healthz that checks the instance only and needs no authentication.
  • Never deny AzureLoadBalancer above the default rule; enforce it with Azure Policy.
  • Alert on DipAvailability below 100 for more than five minutes.

Related fixes