Home / Azure

AzureTroubleshootingIntermediate

AKS pods stuck in ImagePullBackOff from Azure Container Registry

Read the pull error, then check image names, the kubelet identity's AcrPull role, ACR networking and private DNS to get images pulling again.

Deepak Bhardwaj2 min read

Applies toAKS 1.29+, Azure Container Registry (Basic/Standard/Premium)

1. Read the actual error

kubectl describe pod <pod> -n <ns> | sed -n '/Events/,$p'
Event text Cause
manifest unknown / not found Wrong repository or tag
401 Unauthorized / authentication required The cluster isn't allowed to pull
403 Forbidden / denied: client with IP ... is not allowed access ACR firewall or private endpoint rule
dial tcp ... i/o timeout / no such host Network or DNS from the nodes to ACR
toomanyrequests Docker Hub rate limit, not ACR. Import the image into ACR.

2. Run the built-in check

az aks check-acr -g rg-aks -n aks-prod --acr myacr.azurecr.io

It tests authentication, DNS and connectivity from a node and reports what's broken.

3. Attach ACR (grants AcrPull to the kubelet identity)

az aks update -g rg-aks -n aks-prod --attach-acr myacr

To check it by hand:

KUBELET_ID=$(az aks show -g rg-aks -n aks-prod --query "identityProfile.kubeletidentity.objectId" -o tsv)
az role assignment list --assignee $KUBELET_ID --all --query "[?roleDefinitionName=='AcrPull'].scope" -o tsv

Note

The kubelet identity pulls images, not the cluster's control-plane identity. Granting AcrPull to the wrong one is a classic mistake.

4. The image name and tag

az acr repository show-tags -n myacr --repository payments/api --orderby time_desc --top 5 -o table
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.containers[*].image}'

Check the registry name in the image reference (myacr.azurecr.io/...). A missing domain makes the node try Docker Hub.

5. Networking (Premium with a private endpoint)

  • privatelink.azurecr.io must be linked to the AKS VNet, and both the registry endpoint and the regional data endpoint (myacr.<region>.data.azurecr.io) must resolve to private IPs.
  • If public access is disabled, nodes can only pull through the private endpoint.
  • An egress firewall must allow the ACR login server and data endpoints, plus mcr.microsoft.com for system images.
kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- nslookup myacr.azurecr.io

6. Pull secrets across tenants

When ACR is in another tenant and managed identity can't be used, create an imagePullSecret from a token or service principal, and reference it in the pod spec or on the service account.

Prevent it next time

  • Pin image digests or immutable tags in production. Rewriting :latest breaks rollbacks.
  • Import public base images into ACR (az acr import) to avoid Docker Hub limits and outages.

Related fixes