AKS cluster autoscaler not scaling up: pods stuck Pending while the node pool stays at the same size

#Symptoms

• kubectl get pods shows pods in Pending for minutes.
• kubectl describe pod shows FailedScheduling ... Insufficient cpu (or memory, or nvidia.com/gpu).
• The node pool node count doesn't change, or it changes and the new node never becomes Ready.

#Step 1: Ask the autoscaler what it thinks

The cluster autoscaler writes its status to a ConfigMap:

kubectl -n kube-system get configmap cluster-autoscaler-status -o yaml

Look at each node group:

• Health: Healthy and ScaleUp: NoActivity with pending pods → it decided the pods won't fit on a new node (see Step 4).
• ScaleUp: InProgress that never finishes → Azure failed to create the VM (Step 3).
• cloudProviderTarget higher than registered → nodes were requested but didn't join.

Pod events are the other half:

kubectl describe pod <pod> | sed -n '/Events/,$p'

You can also read autoscaler logs if you enabled control plane diagnostics (category cluster-autoscaler) to Log Analytics:

AKSControlPlane
| where Category == "cluster-autoscaler"
| where TimeGenerated > ago(1h)
| project TimeGenerated, Message
| order by TimeGenerated desc

(Table name depends on whether you use resource-specific mode; older setups use AzureDiagnostics.)

#Step 2: Check the pool limits

az aks nodepool list -g rg-aks -n aks-prod \
  --query "[].{pool:name, auto:enableAutoScaling, min:minCount, max:maxCount, count:count, vm:vmSize, mode:mode}" -o table

If count == maxCount, raise it:

az aks nodepool update -g rg-aks --cluster-name aks-prod -n userpool \
  --update-cluster-autoscaler --min-count 3 --max-count 20

#Step 3: Azure couldn't create the VM

Check Activity Log for the node resource group (MC_*):

NODE_RG=$(az aks show -g rg-aks -n aks-prod --query nodeResourceGroup -o tsv)
az monitor activity-log list -g $NODE_RG --offset 2h --status Failed \
  --query "[].{op:operationName.localizedValue, msg:properties.statusMessage}" -o table

Common failures:

Check quota:

az vm list-usage --location centralindia \
  --query "[?contains(name.value,'standardDSv5Family')]" -o table

Check subnet free IPs:

SUBNET_ID=$(az aks nodepool show -g rg-aks --cluster-name aks-prod -n userpool --query vnetSubnetId -o tsv)
az network vnet subnet show --ids $SUBNET_ID --query "{prefix:addressPrefix, used:length(ipConfigurations)}"

#Step 4: The pod would not fit a new node anyway

The autoscaler simulates scheduling on a template node. If the pod still wouldn't fit, it doesn't scale.

• Requests larger than the node's allocatable: a pod asking for 16 GiB on a pool of 16 GiB VMs won't fit, because allocatable is lower after system reservations. Check with kubectl describe node <node> | grep -A6 Allocatable.
• nodeSelector / affinity pointing at a label no autoscaling pool has.
• Taints: GPU or spot pools are tainted; pods need matching tolerations.
• PVC zone: a pod with an Azure Disk PV in zone 1 can only run on a node in zone 1. If the pool's zone 1 has no capacity, it won't scale elsewhere. Use ZRS disks or one pool per zone.
• Pod topology spread with whenUnsatisfiable: DoNotSchedule that can't be met.

kubectl get pod <pod> -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.tolerations}{"\n"}{.spec.containers[*].resources.requests}{"\n"}'

#Step 5: The new node joined but stays NotReady

Then it's not an autoscaler problem. Check the node:

kubectl describe node <new-node> | sed -n '/Conditions/,/Addresses/p'

Typical causes: egress blocked by a firewall/UDR (nodes can't pull images or reach the API server), custom DNS that can't resolve *.azmk8s.io, or CNI not ready because the subnet ran out of IPs.

#Tuning the autoscaler profile

If it scales, but too slowly for bursty load:

az aks update -g rg-aks -n aks-prod \
  --cluster-autoscaler-profile scan-interval=10s scale-down-delay-after-add=10m \
  max-node-provision-time=15m expander=least-waste

Warning

Very aggressive scale-down settings cause node churn and pod restarts. Change one parameter at a time and watch for a week.

#Prevent it next time

• Alert when any pool's node count reaches maxCount (Container Insights metric or a scheduled KQL query).
• Keep 20-30% vCPU quota headroom per VM family; review monthly.
• Size node subnets for maxCount × (maxPods + 1) IPs, or use Azure CNI Overlay.
• Run at least two pools with different VM SKUs for critical workloads so capacity shortages in one SKU don't block scaling.
• For latency-sensitive bursts, keep spare capacity with low-priority placeholder pods that get evicted first.