AKS Service stuck with EXTERNAL-IP pending: identity, subnet and annotation checks

#Symptom

$ kubectl get svc -n app web
NAME   TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
web    LoadBalancer   10.0.45.12    <pending>     80:31234/TCP   14m

A new load balancer rule usually appears within a minute or two. If it's been more than a few minutes, something is failing, and the cluster is probably telling you why.

#Step 1: read the events

kubectl describe svc web -n app
kubectl get events -n app --field-selector involvedObject.name=web --sort-by=.lastTimestamp

Look for SyncLoadBalancerFailed or EnsuringLoadBalancer. The message maps to a cause:

#Step 2: check the cluster identity's permissions

The AKS cluster identity, not the kubelet identity, changes load balancers and reads network resources. It already has rights in the node resource group. It needs extra rights for anything outside it: a VNet you brought yourself, or a public IP in another group.

Find the identity:

az aks show -g rg-aks -n aks-prod \
  --query "{type:identity.type, principalId:identity.principalId, userAssigned:identity.userAssignedIdentities, nodeRg:nodeResourceGroup, subnet:agentPoolProfiles[0].vnetSubnetId}"

For a user-assigned identity, use the principal ID shown in userAssignedIdentities. Then list what it has:

PRINCIPAL=<cluster identity principal id>
SUBNET_ID=<subnet resource id>

az role assignment list --assignee "$PRINCIPAL" --all \
  --query "[].{role:roleDefinitionName, scope:scope}" -o table

Grant access on the subnet if it's missing:

az role assignment create \
  --assignee-object-id "$PRINCIPAL" \
  --assignee-principal-type ServicePrincipal \
  --role "Network Contributor" \
  --scope "$SUBNET_ID"

Network Contributor is the usual choice. You can create a custom role instead with only the actions in the error message. The error names the exact action it wanted, such as Microsoft.Network/virtualNetworks/subnets/join/action, and the scope.

RBAC changes can take several minutes to apply. The controller retries with backoff, so wait, then re-read the events.

#Step 3: fix an internal Service

An internal load balancer takes its IP from a subnet, not the public internet:

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: app
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
    service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "snet-ilb"
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

Checks to make:

• The annotation value must be the literal string "true".
• azure-load-balancer-internal-subnet is the subnet name, and that subnet must be in the cluster's VNet.
• The subnet needs free addresses:

az network vnet subnet show -g rg-net --vnet-name vnet-aks -n snet-ilb \
  --query "{prefix:addressPrefix, used:length(ipConfigurations || \`[]\`)}"

If you request a fixed IP, make sure it's inside the subnet and not in use. Use the azure-load-balancer-ipv4 annotation, because spec.loadBalancerIP is deprecated:

    service.beta.kubernetes.io/azure-load-balancer-ipv4: "10.240.10.25"

#Step 4: fix a static public IP

The default is to look for the IP in the node resource group. If yours lives elsewhere, you must say so, and the identity needs rights there:

metadata:
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-resource-group: "rg-aks-ips"
    service.beta.kubernetes.io/azure-pip-name: "pip-web-prod"

Check the IP. It must be Standard SKU to match a Standard load balancer, and must not be in use elsewhere:

az network public-ip show -g rg-aks-ips -n pip-web-prod \
  --query "{sku:sku.name, ip:ipAddress, attachedTo:ipConfiguration.id}"

az role assignment create \
  --assignee-object-id "$PRINCIPAL" \
  --assignee-principal-type ServicePrincipal \
  --role "Network Contributor" \
  --scope "$(az group show -n rg-aks-ips --query id -o tsv)"

Caution

Don't change load balancer resources in the node resource group by hand. The cloud provider owns them and can overwrite your changes. Deleting a Service with a controller-created public IP also deletes that IP, and its address is gone. For a DNS name or allowlist that depends on it, use a static IP you own, in your own group.

#Step 5: check limits and policy

# Public IP usage in the region
az network list-usages --location centralindia \
  --query "[?contains(name.value, 'PublicIPAddresses')].{name:name.localizedValue, current:currentValue, limit:limit}" -o table

# Why was this denied?
az monitor activity-log list \
  --resource-group "$(az aks show -g rg-aks -n aks-prod --query nodeResourceGroup -o tsv)" \
  --offset 1h --status Failed \
  --query "[].{when:eventTimestamp, op:operationName.value, msg:properties.statusMessage}" -o json

Many organizations deny public IPs with Azure Policy. If that's what the event says, use an internal load balancer, or see RequestDisallowedByPolicy. There are also limits on load balancer rules. Check the current limits if you run many Services on one cluster.

#Step 6: no events, or still stuck

kubectl get svc web -n app -o yaml
az aks show -g rg-aks -n aks-prod --query "{state:provisioningState, power:powerState.code, lbSku:networkProfile.loadBalancerSku}"

Check for these:

• spec.loadBalancerClass is set. Another controller is expected to handle it, so Azure's won't.
• A cluster in a failed state. Fix that first, because reconciliation may not be running. See upgrade failed state.
• A replaced cluster identity. Check that the new identity has the same network roles as the old one.

#After it gets an IP but traffic fails

That's a different problem. Check the data path:

kubectl get endpointslices -n app -l kubernetes.io/service-name=web
kubectl get pods -n app -l app=web -o wide

An empty endpoint list means the Service selector doesn't match any ready pods. Also see load balancer health probes. Clusters can customize the probe with annotations such as azure-load-balancer-health-probe-request-path.

#Prevent it next time

• Grant the identity its network role when you create the cluster, in your infrastructure code.
• Own your static IPs in a separate resource group, and manage those role assignments as code.
• Alert on SyncLoadBalancerFailed events.
• Keep a spare range in internal load balancer subnets. For address planning, see CNI overlay vs pod subnet.