Home / Azure

AzureTroubleshootingAdvanced

Sev1: AKS upgrade failed or stuck — node pool in Failed state, nodes won't drain

Recover an AKS cluster or node pool left in a Failed provisioning state after an upgrade. Find the blocking PodDisruptionBudget, quota or subnet issue, reconcile the cluster and finish the upgrade safely.

Deepak Bhardwaj3 min read

Applies toAzure Kubernetes Service, Kubernetes 1.28+, Azure CLI 2.50+

Symptoms

  • az aks upgrade or az aks nodepool upgrade fails after a long wait.
  • provisioningState is Failed or stuck on Upgrading.
  • Some nodes are on the new version, some on the old; some are cordoned (SchedulingDisabled).
  • Errors mention Too many requests, eviction, QuotaExceeded or SubnetIsFull.

1. Read the actual error

az aks show -g rg-aks -n aks-prod --query "{state:provisioningState, version:kubernetesVersion}" -o table
az aks nodepool list -g rg-aks --cluster-name aks-prod \
  --query "[].{pool:name, state:provisioningState, version:orchestratorVersion, count:count}" -o table

az monitor activity-log list -g rg-aks --offset 6h --status Failed \
  --query "[].{op:operationName.localizedValue, msg:properties.statusMessage}" -o json | head -60
Error contains Cause Fix
Eviction, PodDrainFailure, Too many requests A PDB allows zero disruptions Step 2
QuotaExceeded Not enough vCPU quota for surge nodes Step 3
SubnetIsFull, InsufficientSubnetSize No IPs for surge nodes Step 3
VMExtensionProvisioningError Node bootstrap failed (often egress blocked) Step 4

2. Blocking PodDisruptionBudgets

kubectl get pdb -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable
kubectl get nodes -o wide | grep -i SchedulingDisabled

Any PDB with ALLOWED = 0 blocks draining. Typical culprits: minAvailable: 1 on a one-replica deployment, or a deployment whose pods are all unhealthy.

Fix by scaling the workload up so a disruption is allowed, or temporarily relax the PDB:

kubectl scale deploy/payments -n prod --replicas=2
kubectl patch pdb payments-pdb -n prod --type merge -p '{"spec":{"minAvailable":null,"maxUnavailable":1}}'

Warning

Note the original PDB and put it back after the upgrade. Deleting PDBs to get an upgrade through is how you turn a maintenance into an outage.

3. Quota or subnet

Each surge node needs vCPU quota and, with Azure CNI (non-overlay), IPs for the node and its pods.

az vm list-usage -l westeurope --query "[?contains(name.value,'Family')].{name:name.localizedValue, used:currentValue, limit:limit}" -o table
az aks nodepool update -g rg-aks --cluster-name aks-prod -n userpool --max-surge 1

Lower max-surge uses fewer extra resources but upgrades slower. Request a quota increase or fix subnet space for a permanent fix.

4. Reconcile the cluster

Once the blocker is fixed, run an update with no changes. This makes AKS re-apply the desired state and usually clears Failed:

az aks update -g rg-aks -n aks-prod
az aks nodepool upgrade -g rg-aks --cluster-name aks-prod -n userpool --kubernetes-version <target>

Uncordon any node left cordoned by the failed attempt only if it is on a supported version and healthy:

kubectl uncordon aks-userpool-12345678-vmss000003

5. Verify

kubectl get nodes -o wide
kubectl get pods -A --field-selector status.phase!=Running

Check the application's own health endpoint, not just pod status.

Prevent it next time

  • Gate upgrades with a pre-check: every PDB must have disruptionsAllowed >= 1.
  • Run critical deployments with at least two replicas across zones.
  • Keep spare quota and subnet space for surge nodes, or use Azure CNI Overlay.
  • Upgrade a non-production cluster on the same version first, and use planned maintenance windows.

Related fixes