Sev1: AKS API server unreachable — authorized IP ranges, private clusters and expired certificates

#Symptoms

• kubectl get nodes returns Unable to connect to the server: dial tcp ...: i/o timeout.
• x509: certificate has expired or is not yet valid.
• You must be logged in to the server (Unauthorized) for everyone at once.
• Every CI/CD deployment to the cluster fails at the same time.

#1. Is the control plane itself healthy?

az aks show -g rg-aks -n aks-prod --query "{state:provisioningState, power:powerState.code, fqdn:fqdn, private:apiServerAccessProfile.enablePrivateCluster, ranges:apiServerAccessProfile.authorizedIpRanges}" -o json

Check Azure Service Health for AKS in the region. If power is Stopped, someone ran az aks stop: start it with az aks start -g rg-aks -n aks-prod.

#2. Use command invoke as a lifeline

az aks command invoke runs kubectl from Azure's side, so it works even when your own network path does not.

az aks command invoke -g rg-aks -n aks-prod --command "kubectl get nodes -o wide"
az aks command invoke -g rg-aks -n aks-prod --command "kubectl get pods -A --field-selector status.phase!=Running"

If this works, the cluster is fine and the problem is the path between you and it.

#3. Authorized IP ranges

A changed egress IP (new NAT gateway, VPN change, new hosted build agents) is a very common cause of the timeout.

curl -s https://ifconfig.me
az aks update -g rg-aks -n aks-prod --api-server-authorized-ip-ranges 203.0.113.10/32,198.51.100.0/27

Warning

The update replaces the whole list. Include every range you still need, including your build agents' egress IPs.

#4. Private clusters

Peered hub VNets and on-premises DNS need the private zone linked, or a conditional forwarder to Azure DNS (168.63.129.16) through a DNS resolver.

#5. Expired cluster certificates

kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -enddate

Rotate:

az aks rotate-certs -g rg-aks -n aks-prod
az aks get-credentials -g rg-aks -n aks-prod --overwrite-existing

Caution

rotate-certs restarts nodes and can cause up to about 30 minutes of disruption. Every client needs fresh credentials afterwards. Do it in a change window unless you are already down.

#6. Unauthorized for everyone

With Entra ID integration, check that the admin group still exists and that kubelogin is current:

az aks show -g rg-aks -n aks-prod --query aadProfile
kubelogin convert-kubeconfig -l azurecli

Break-glass with az aks get-credentials --admin only works if local accounts are enabled (disableLocalAccounts is false). Decide in advance whether you keep that door open.

#Prevent it next time

• Keep build agents on fixed egress IPs and manage the authorized range list in IaC.
• Keep clusters on supported Kubernetes versions and track certificate expiry.
• Put the command invoke path in your on-call runbook before you need it.