Home / Azure

AzureTroubleshootingAdvanced

High severity: subnet out of IP addresses (AKS scale-out, App Service, private endpoints failing)

Fix deployments and scale-outs that fail because an Azure subnet ran out of IP addresses. Covers finding the consumer, quick relief options, adding address space without downtime, and moving AKS to an overlay network model.

Deepak Bhardwaj3 min read

Applies toAzure Virtual Network with AKS (Azure CNI), App Service VNet integration, private endpoints, VMSS

Symptoms

  • AKS: nodes or pods fail with SubnetIsFull, InsufficientSubnetSize, pods stuck ContainerCreating with "failed to allocate IP".
  • SubnetIsFull or PrivateIPAddressAllocationFailed when creating NICs, private endpoints or scaling VMSS.
  • App Service VNet integration scale-out fails.

1. Find out who uses the space

az network vnet subnet show -g rg-net -n snet-aks --vnet-name vnet-spoke1 \
  --query "{prefix:addressPrefix, ipConfigs:length(ipConfigurations || \`[]\`)}"

az network vnet subnet list-available-ips -g rg-net -n snet-aks --vnet-name vnet-spoke1

Remember Azure reserves 5 IPs per subnet. With Azure CNI (classic), every pod gets a VNet IP: a 30-node cluster with max-pods 30 needs about 30 × (30 + 1) = 930 IPs, plus upgrade surge nodes.

2. Quick relief (minutes)

  • Lower AKS max surge during upgrades, or pause the scale-out.
  • Delete unused NICs, stale private endpoints, deallocated test VMs.
  • AKS: scale down non-critical node pools; set a lower maxPods on new node pools.

3. Add space properly

Add address space to the VNet (no downtime), then a new subnet:

az network vnet update -g rg-net -n vnet-spoke1 --address-prefixes 10.20.0.0/16 10.21.0.0/16
az network vnet subnet create -g rg-net --vnet-name vnet-spoke1 -n snet-aks-pool2 --address-prefixes 10.21.0.0/18

Peered VNets: after adding address space, sync the peering (az network vnet peering sync).

Resize a subnet: possible only if no resources use it (or by adding a non-overlapping range in some cases); usually it's easier to create a new, bigger subnet.

AKS: add a new node pool on the new subnet, move workloads (cordon and drain old nodes), delete the old pool.

az aks nodepool add -g rg-aks --cluster-name aks-prod -n pool2 --vnet-subnet-id <new-subnet-id> --node-count 3

4. Fix the design

Option Effect
Azure CNI Overlay Pods get IPs from a private overlay CIDR; only nodes use VNet IPs
Azure CNI with dynamic pod IP allocation (separate pod subnet) Pod subnet can be large and separate from nodes
Dedicated subnets per service (private endpoints, App Service integration) One consumer can't starve another

Existing clusters can be upgraded from Azure CNI to Overlay (az aks update --network-plugin-mode overlay --pod-cidr 192.168.0.0/16); test in non-production first and check prerequisites.

5. Prevent

  • Alert at 80% subnet utilisation (script or Azure Resource Graph query on a schedule).
  • IP plan with sized subnets per workload type, recorded in IPAM (Azure Virtual Network Manager IPAM or your own).
  • /24 is too small for most AKS subnets with classic CNI.

Warning

Don't overlap new address space with on-premises or other spokes "just to get going". It breaks routing later and is very hard to undo. Check the IP plan first.

Related fixes