AKS node is NotReady: diagnosis and recovery

#1. What does the node report?

kubectl get nodes -o wide
kubectl describe node <node> | sed -n '/Conditions/,/Addresses/p'
kubectl get events -A --field-selector involvedObject.name=<node> --sort-by=.lastTimestamp | tail

#2. Is the VM itself healthy?

NODE_RG=$(az aks show -g rg-aks -n aks-prod --query nodeResourceGroup -o tsv)
az vmss list -g $NODE_RG -o table
az vmss list-instances -g $NODE_RG -n <vmss-name> --query "[].{id:instanceId, name:osProfile.computerName, state:provisioningState}" -o table
az vmss get-instance-view -g $NODE_RG -n <vmss-name> --instance-id <id> --query "statuses[].displayStatus"

Check Resource Health for the scale set too. Host failures show up there.

#3. Look inside the node

kubectl debug node/<node> -it --image=mcr.microsoft.com/cbl-mariner/busybox:2.0
# inside: chroot /host
chroot /host
systemctl status kubelet containerd --no-pager
journalctl -u kubelet --since "30 min ago" --no-pager | tail -50
df -h /var/lib/containerd

If kubectl debug can't schedule because the node is too broken, use VMSS run-command:

az vmss run-command invoke -g $NODE_RG -n <vmss-name> --instance-id <id> \
  --command-id RunShellScript --scripts "systemctl status kubelet --no-pager; df -h"

#4. Recover

In order of least disruption:

1. Fix the pressure. Delete the runaway pod, clean disk (crictl rmi --prune), or add limits.
2. Restart the kubelet with run-command: systemctl restart kubelet.
3. Reimage or replace the node. First drain workloads off it:

   kubectl cordon <node>
   kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=10m
   az vmss reimage -g $NODE_RG -n <vmss-name> --instance-ids <id>
   # or delete the instance and let the autoscaler / node count replace it
   az vmss delete-instances -g $NODE_RG -n <vmss-name> --instance-ids <id>
   

Warning

Don't modify resources in the node resource group (MC_*) directly, except for recovery actions like the ones above. Manual changes there are unsupported and can be overwritten by AKS operations.

#5. Many nodes NotReady at once

That points at something shared:

• The API server is unreachable: an egress firewall change, private DNS, or an expired client certificate after failed credential rotation.
• Outbound connectivity is broken: NAT gateway, firewall or a UDR change.
• A platform event: check AKS Resource Health and Azure Status.

#Prevent it next time

• Set resource requests and limits on every workload, with LimitRanges per namespace.
• Turn on node auto-repair (on by default) and the cluster autoscaler.
• Alert on kube_node_status_condition with Ready not true for more than 5 minutes (Container Insights or Prometheus).