Home / Azure

AzureTroubleshootingAdvanced

Sev1: AKS pods stuck in ImagePullBackOff — ACR 401/403, detached registry and private endpoint DNS

New and rescheduled pods can't pull images from Azure Container Registry and the service degrades as old pods die. Diagnose auth, network and tag problems and restore pulls fast.

Deepak Bhardwaj3 min read

Applies toAzure Kubernetes Service, Azure Container Registry (Basic/Standard/Premium), Azure CLI 2.50+

Symptoms

  • Pods show ErrImagePull or ImagePullBackOff; deployments can't roll out and node replacements make it worse.
  • kubectl describe pod shows 401 Unauthorized, 403 Forbidden, not found or i/o timeout.

1. Read the exact pull error

kubectl get pods -A | grep -E "ImagePull|ErrImage"
kubectl describe pod <pod> -n <ns> | sed -n '/Events/,$p'
Message Cause Go to
401 Unauthorized / unauthorized: authentication required Kubelet identity lost AcrPull, or wrong pull secret Step 2
403 Forbidden / client with IP ... is not allowed ACR firewall or public access disabled Step 3
dial tcp ... i/o timeout / no such host DNS or private endpoint path Step 3
manifest unknown / not found Tag deleted or retention policy purged it Step 4

2. Fix authentication

AKS pulls with the kubelet identity, which needs AcrPull on the registry.

az aks check-acr -g rg-aks -n aks-prod --acr acrprod.azurecr.io

This command runs a test pod and reports exactly what fails. If the role is missing, reattach:

az aks update -g rg-aks -n aks-prod --attach-acr acrprod

Or grant it directly to the kubelet identity:

KUBELET=$(az aks show -g rg-aks -n aks-prod --query identityProfile.kubeletidentity.objectId -o tsv)
ACR_ID=$(az acr show -n acrprod --query id -o tsv)
az role assignment create --assignee-object-id $KUBELET --assignee-principal-type ServicePrincipal \
  --role AcrPull --scope $ACR_ID

If the workload uses an imagePullSecrets secret with an ACR token or admin password, it may have been rotated. Recreate the secret:

kubectl create secret docker-registry acr-pull -n app \
  --docker-server=acrprod.azurecr.io --docker-username=<token-name> --docker-password='<password>' \
  --dry-run=client -o yaml | kubectl apply -f -

3. Fix network access

az acr show -n acrprod --query "{public:publicNetworkAccess, rules:networkRuleSet}"
az acr check-health -n acrprod --yes
  • If public access was disabled, the cluster must reach ACR through a private endpoint. Check both acrprod.azurecr.io and the data endpoint (acrprod.<region>.data.azurecr.io) resolve to private IPs from a node:
kubectl run dnscheck --rm -it --image=busybox:1.36 --restart=Never -- nslookup acrprod.azurecr.io
  • The privatelink.azurecr.io private DNS zone must be linked to the AKS VNet (or the DNS forwarders it uses).
  • If the cluster egresses through Azure Firewall, allow ACR FQDNs and the data endpoint.

Tip

If the registry you pull from is not reachable but a geo-replica is, Premium registries route to the nearest replica. Check replica status with az acr replication list -r acrprod -o table.

4. Fix missing tags

az acr repository show-tags -n acrprod --repository orders-api --orderby time_desc --top 10 -o table
az acr config retention show -r acrprod

If a purge task or retention policy removed the tag, rebuild or re-push from CI, or roll back the deployment to a tag that still exists:

kubectl rollout undo deployment/orders-api -n app

5. Unblock the rollout

After the fix, delete the stuck pods so they retry immediately instead of waiting for back-off:

kubectl delete pod -n app -l app=orders-api --field-selector=status.phase=Pending
kubectl rollout status deployment/orders-api -n app

Prevent it next time

  • Deploy by immutable digest or unique tags, and lock production tags (az acr repository update --write-enabled false).
  • Exclude production repositories from purge tasks.
  • Run az aks check-acr in the pipeline after cluster or registry changes.
  • Use Premium with geo-replication for multi-region clusters.

Related fixes