Home / AWS

AWSTroubleshootingAdvanced

Sev1: EKS pods stuck in ContainerCreating — VPC CNI out of IP addresses

New pods won't start with failed to assign an IP address to container. Confirm subnet and ENI exhaustion, free addresses, and fix capacity with prefix delegation or more subnets.

Deepak Bhardwaj3 min read

Applies toAmazon EKS with Amazon VPC CNI plugin (aws-node), kubectl, AWS CLI v2

Symptoms

  • Pods stuck in ContainerCreating during a scale-out or deployment.
  • kubectl describe pod events show failed to assign an IP address to container or add cmd: failed to assign an IP address.
  • Cluster Autoscaler or Karpenter adds nodes, but pods still don't start.

1. Confirm IP exhaustion

kubectl get pods -A --field-selector=status.phase=Pending -o wide | head
kubectl describe pod <pod> -n <ns> | grep -iA3 "failed"
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50 | grep -iE "error|insufficient|ipamd"

Check subnet free addresses:

aws ec2 describe-subnets --filters Name=tag:kubernetes.io/cluster/eks-prod,Values=shared,owned \
  --query "Subnets[].{id:SubnetId, az:AvailabilityZone, cidr:CidrBlock, free:AvailableIpAddressCount}" --output table
Finding Meaning
Subnets show very few free IPs Subnet exhaustion; VPC CNI gives each pod a VPC IP
Plenty of subnet IPs, node at max pods ENI / IP-per-ENI limit of the instance type
aws-node pods crash-looping CNI itself broken (IAM, version)

2. Quick relief

  • Reduce warm IP waste: by default ipamd keeps a whole spare ENI's worth of IPs per node. Lower it:
kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=2 MINIMUM_IP_TARGET=10

Warning

Very low warm targets increase EC2 API calls and pod start latency, and can hit API throttling on large clusters. Use moderate values and revisit after the incident.

  • Scale down non-critical workloads to free IPs for critical pods.
  • Remove idle nodes holding warm IPs (cordon, drain, terminate) if autoscaling over-provisioned.

3. Fix node-level limits: prefix delegation

With prefix delegation, each ENI slot gets a /28 prefix (16 IPs), greatly raising pods per node on Nitro instances.

kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true WARM_PREFIX_TARGET=1

Existing nodes keep their old allocation; new nodes use prefixes. Also raise maxPods for new nodes (managed node groups compute it with the right settings; for custom AMIs set it in the bootstrap). Prefix delegation needs contiguous free /28 blocks in the subnet, which fragmented subnets may not have.

4. Fix subnet exhaustion: more address space

Add a secondary CIDR to the VPC and new subnets for pods with custom networking:

aws ec2 associate-vpc-cidr-block --vpc-id vpc-0abc --cidr-block 100.64.0.0/16
aws ec2 create-subnet --vpc-id vpc-0abc --cidr-block 100.64.0.0/19 --availability-zone eu-west-1a

Then enable custom networking and create an ENIConfig per AZ:

kubectl set env daemonset aws-node -n kube-system AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true \
  ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone

apiVersion: crd.k8s.amazonaws.com/v1alpha1
kind: ENIConfig
metadata:
  name: eu-west-1a
spec:
  subnet: subnet-0pods1a
  securityGroups: [sg-0nodes]

New nodes then place pods in the 100.64.x.x subnets. Replace nodes in a rolling way to move existing ones.

5. Verify

kubectl get pods -A --field-selector=status.phase=Pending
kubectl get nodes -o custom-columns=NAME:.metadata.name,PODS:.status.allocatable.pods

Prevent it next time

  • Size pod subnets for peak pods plus warm IPs (plan /19 or larger per AZ for busy clusters).
  • Turn on prefix delegation for new clusters on Nitro instances.
  • Alarm when AvailableIpAddressCount in cluster subnets drops below a threshold.
  • Keep the VPC CNI add-on on a supported version and managed as an EKS add-on.

Related fixes