EKS pods stuck in ContainerCreating - fixing pod IP exhaustion with VPC CNI prefix delegation and custom networking

#Symptoms

• Pods stay in ContainerCreating. kubectl describe pod shows failed to assign an IP address to container or add cmd: failed to assign an IP address.
• Nodes show plenty of CPU and memory but no new pods start.
• New nodes from the autoscaler fail to become Ready, or join and immediately fill up.

The Amazon VPC CNI gives each pod a real VPC IP from the node's subnet. Two limits apply:

1. Subnet free IPs. The subnet can run out of addresses.
2. Per-node limit. The number of ENIs × IPs per ENI for the instance type sets max-pods.

#1. Which limit did you hit?

kubectl get pods -A --field-selector status.phase=Pending -o wide | head
kubectl describe pod <pod> -n <ns> | sed -n '/Events/,$p'

# CNI logs on the node (via SSM or a debug pod)
kubectl -n kube-system logs -l k8s-app=aws-node -c aws-node --tail=100 | grep -i -E "insufficient|failed|no available"

Free IPs per subnet used by the nodes:

aws ec2 describe-subnets --filters Name=tag:kubernetes.io/cluster/prod,Values=shared,owned \
  --query "Subnets[].{id:SubnetId,az:AvailabilityZone,cidr:CidrBlock,free:AvailableIpAddressCount}" --output table

Pods per node compared with the limit:

kubectl get nodes -o custom-columns=NAME:.metadata.name,MAXPODS:.status.allocatable.pods,TYPE:.metadata.labels.node\\.kubernetes\\.io/instance-type
kubectl get pods -A -o wide --field-selector spec.nodeName=<node> | wc -l

#2. Reduce warm IP waste (quick relief)

By default the CNI keeps a whole spare ENI's worth of IPs warm on each node. On large node counts that wastes many addresses.

kubectl -n kube-system set env daemonset aws-node WARM_IP_TARGET=5 MINIMUM_IP_TARGET=20
kubectl -n kube-system get ds aws-node -o yaml | grep -A1 -E "WARM_|MINIMUM_"

This returns unused IPs to the subnet. The trade-off is more EC2 API calls when pods churn quickly.

#3. Prefix delegation (more pods per node, fewer ENIs)

With prefix delegation the CNI assigns /28 IPv4 prefixes (16 addresses each) to ENI slots instead of single IPs. This raises the pods-per-node ceiling substantially on Nitro instances.

kubectl -n kube-system set env daemonset aws-node ENABLE_PREFIX_DELEGATION=true WARM_PREFIX_TARGET=1

Then raise max-pods for new nodes. For managed node groups with the EKS-optimised AMI, set it through the launch template or bootstrap configuration. Calculate the recommended value with the script AWS publishes:

curl -O https://raw.githubusercontent.com/awslabs/amazon-eks-ami/main/templates/al2/runtime/max-pods-calculator.sh
chmod +x max-pods-calculator.sh
./max-pods-calculator.sh --instance-type m6i.xlarge --cni-version 1.18.0 --cni-prefix-delegation-enabled

Script paths differ between AMI families and versions. Check the EKS documentation for your node AMI.

Warning

Prefix delegation needs contiguous free /28 blocks in the subnet. A fragmented subnet can have free IPs but no free /28. Use subnet CIDR reservations (aws ec2 create-subnet-cidr-reservation --reservation-type prefix) or new subnets for nodes.

Existing nodes keep their old ENI layout. Roll the node groups to apply the change.

#4. Custom networking (pods in a secondary CIDR)

If the VPC itself is short of space, add a secondary CIDR (for example from 100.64.0.0/10) and put pods there while nodes stay in the original subnets.

aws ec2 associate-vpc-cidr-block --vpc-id vpc-0abc --cidr-block 100.64.0.0/16
# create one pod subnet per AZ in that range, then:
kubectl -n kube-system set env daemonset aws-node AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: eu-west-1a          # must match the zone label value
spec:
  subnet: subnet-0pods1a
  securityGroups: [sg-0nodes]

Create one ENIConfig per AZ, then replace the nodes. With custom networking the primary ENI isn't used for pods, so max-pods per node drops unless you also enable prefix delegation.

#5. Long-term options

#Checklist

• Confirmed whether subnet or per-node limit is the cause
• Warm IP targets tuned
• Prefix delegation enabled and max-pods updated, nodes rolled
• Subnets have contiguous /28 space (or reservations)
• Alert on subnet AvailableIpAddressCount below a threshold