Azure VM extension provisioning failed: a troubleshooting guide

The symptom is usually one of these: a deployment fails with VMExtensionProvisioningError, the portal shows an extension as Transitioning for an hour, or a later change (resize, Run Command, a new extension) fails because an earlier extension never finished. Extensions run through the VM guest agent, so the fault is either in the agent, in the network path the agent uses, or in the extension's own script or installer.

#Step 1: read the status the agent reported

Start with the instance view. It shows what the agent itself last reported, which is more useful than the ARM provisioning state.

# Agent health: version, status and last message
az vm get-instance-view -g rg-prod -n vm-app-01 \
  --query "instanceView.vmAgent.{version:vmAgentVersion, status:statuses[0].displayStatus, message:statuses[0].message}" -o json

# Every extension: handler status and the message from the extension
az vm get-instance-view -g rg-prod -n vm-app-01 \
  --query "instanceView.extensions[].{name:name, type:type, status:statuses[0].displayStatus, message:statuses[0].message}" -o json

How to read it:

#Step 2: fix the guest agent

If the agent isn't Ready, no extension will progress. The agent talks to the host through the Azure platform address 168.63.129.16 on ports 80 and 32526. An NSG, a guest firewall, a proxy setting or a custom route that sends this traffic elsewhere will break it.

On Windows (through the serial console or an RDP session):

Get-Service WindowsAzureGuestAgent, RdAgent | Select-Object Name, Status, StartType
Restart-Service WindowsAzureGuestAgent

# Can the VM reach the wire server?
Test-NetConnection 168.63.129.16 -Port 80
Test-NetConnection 168.63.129.16 -Port 32526

# Agent log
Get-Content C:\WindowsAzure\Logs\WaAppAgent.log -Tail 50

On Linux (the service is walinuxagent on Ubuntu/Debian and waagent on RHEL, SUSE and most others):

sudo systemctl status walinuxagent || sudo systemctl status waagent
sudo systemctl restart walinuxagent || sudo systemctl restart waagent
sudo tail -n 100 /var/log/waagent.log
curl -s -o /dev/null -w "%{http_code}\n" http://168.63.129.16/?comp=versions

Tip

If the agent can't reach 168.63.129.16, check for a proxy configured system-wide. The wire server must be reached directly; add it to the proxy bypass list (NO_PROXY on Linux, the WinHTTP/WinINET bypass list on Windows).

If the agent is missing or badly out of date, reinstall it from the Microsoft packages for your OS. On Linux, the distribution's package (walinuxagent or WALinuxAgent) is the supported route.

#Step 3: read the extension's own logs

Once the agent is healthy, the extension's logs tell you why it failed. Folder names include the publisher, type and handler version, so list the directory first.

For the Custom Script Extension, the script's output is what you actually need:

# Linux: stdout and stderr of the last run
sudo cat /var/lib/waagent/custom-script/download/0/stdout
sudo cat /var/lib/waagent/custom-script/download/0/stderr
sudo tail -n 50 /var/log/azure/custom-script/handler.log

# Windows: handler log and the downloaded script
Get-ChildItem C:\WindowsAzure\Logs\Plugins\Microsoft.Compute.CustomScriptExtension -Recurse -Filter *.log |
  Sort-Object LastWriteTime | Select-Object -Last 1 | Get-Content -Tail 50
Get-ChildItem C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension -Recurse -Directory -Filter Downloads

Common Custom Script causes:

For agent-style extensions (Azure Monitor Agent, Defender, Dependency agent), the cause is usually outbound access to the service endpoints, a conflicting older agent already installed, or an unsupported OS version. The handler log names the failing step.

#Step 4: remove and redeploy the extension

Once the cause is fixed, clear the failed state. Re-running a deployment with the same settings often does nothing, because the agent sees no change.

# See what's installed
az vm extension list -g rg-prod --vm-name vm-app-01 \
  --query "[].{name:name, publisher:publisher, type:typePropertiesType, version:typeHandlerVersion, state:provisioningState}" -o table

# Option A: force the same settings to run again
az vm extension set -g rg-prod --vm-name vm-app-01 \
  --name CustomScriptExtension --publisher Microsoft.Compute \
  --settings '{"commandToExecute":"powershell -ExecutionPolicy Bypass -File configure.ps1"}' \
  --force-update

# Option B: remove it, then deploy again from your template or pipeline
az vm extension delete -g rg-prod --vm-name vm-app-01 -n CustomScriptExtension

Warning

Deleting an agent extension such as Azure Monitor Agent or a security agent stops monitoring or protection on that VM until it's reinstalled. Do it in a change window and redeploy straight away.

A VM can only have one instance of each extension type. If a pipeline tries to add a second Custom Script Extension under a different name, it fails. Reuse the existing name or delete the old one first.

If an extension is stuck in Transitioning and blocks every other operation, and the agent is healthy, a reapply asks the platform to reconcile the VM's state:

az vm reapply -g rg-prod -n vm-app-01

#Find failed extensions across the fleet

Resource Graph shows the ARM provisioning state for every extension you can read:

az graph query -q "
resources
| where type == 'microsoft.compute/virtualmachines/extensions'
| extend state = tostring(properties.provisioningState)
| where state != 'Succeeded'
| project vm = tostring(split(id, '/')[8]), extension = name,
          publisher = tostring(properties.publisher), state, resourceGroup, subscriptionId
| order by vm asc" --first 1000 -o table

This needs the resource-graph CLI extension (az extension add --name resource-graph). The ARM state can lag what the agent reports, so confirm individual VMs with the instance view from Step 1.

#Prevent it next time

• Allow 168.63.129.16 in guest firewalls and proxy bypass lists in your base image, and don't block it in NSGs or route it through an NVA.
• Keep the guest agent's auto-update on. Old agents are a frequent cause of extension failures.
• Make Custom Script scripts idempotent, non-interactive and fast. Use them to bootstrap, not to run long installs.
• Pull scripts with a managed identity instead of SAS tokens that expire.
• Alert on extensions that aren't Succeeded with the Resource Graph query above, run on a schedule, so failures are found before the next deployment trips over them.