VPC subnet out of IP addresses: find what's using them and add capacity

#Symptoms

Exact wording varies by service, but these all point at the same problem:

You can't resize a subnet after you create it. The only options are to free addresses or to move workloads to new subnets.

#1. Confirm which subnets are short

VPC=vpc-0abc1234
aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC \
  --query "sort_by(Subnets,&AvailableIpAddressCount)[].{Subnet:SubnetId,AZ:AvailabilityZone,Cidr:CidrBlock,Free:AvailableIpAddressCount,Name:Tags[?Key=='Name']|[0].Value}" \
  --output table

AWS reserves the first four addresses and the last address in every subnet, so a /24 holds 251 usable addresses, not 256.

Check Lambda directly, because a running function can look healthy while its updates fail:

aws lambda get-function-configuration --function-name my-function \
  --query "{State:State,Reason:StateReasonCode,LastUpdate:LastUpdateStatus,LastReason:LastUpdateStatusReasonCode}"

#2. Find what is holding the addresses

Every address in use belongs to a network interface. Summarise them by type and status:

SUBNET=subnet-0abc1234
aws ec2 describe-network-interfaces --filters Name=subnet-id,Values=$SUBNET --output json \
  | jq -r '.NetworkInterfaces | group_by(.InterfaceType + "/" + .Status)[]
      | "\(.[0].InterfaceType)\t\(.[0].Status)\tENIs=\(length)\tIPs=\(map(.PrivateIpAddresses|length)|add)\tPrefixes=\(map((.Ipv4Prefixes // [])|length)|add)"'

Each IPv4 prefix on an interface is a /28, which takes 16 addresses. A few nodes using prefix delegation can consume far more than the interface count suggests.

Then list the interfaces and look at their descriptions:

aws ec2 describe-network-interfaces --filters Name=subnet-id,Values=$SUBNET \
  --query 'NetworkInterfaces[].{Id:NetworkInterfaceId,Status:Status,IPs:length(PrivateIpAddresses),Managed:RequesterManaged,Instance:Attachment.InstanceId,Desc:Description}' \
  --output table

Descriptions usually identify the owner. Names vary, so treat this as a guide:

#3. Reclaim addresses safely

These are quick wins when you need a few addresses immediately.

#Detached interfaces

aws ec2 describe-network-interfaces \
  --filters Name=subnet-id,Values=$SUBNET Name=status,Values=available \
  --query 'NetworkInterfaces[].{Id:NetworkInterfaceId,Managed:RequesterManaged,Desc:Description,Tags:TagSet}' \
  --output json

Caution

Deleting an interface can't be undone, and any fixed private IP it held may be taken by something else. Only delete an interface when you know what created it and that nothing will reattach it. Leave interfaces with AWS service descriptions alone, including Lambda, EFS and load balancer interfaces. The owning service manages them.

aws ec2 delete-network-interface --network-interface-id eni-0abc1234

#Stopped instances

A stopped instance keeps its private IP on its primary interface:

aws ec2 describe-instances \
  --filters Name=subnet-id,Values=$SUBNET Name=instance-state-name,Values=stopped \
  --query 'Reservations[].Instances[].{Id:InstanceId,Name:Tags[?Key==`Name`]|[0].Value,Since:StateTransitionReason}' \
  --output table

Ask the owner before terminating anything. Termination is permanent and can delete attached volumes.

#EKS warm IP pools

The VPC CNI keeps spare addresses on each node by default. On a tight subnet, set an explicit warm target:

kubectl set env daemonset aws-node -n kube-system WARM_IP_TARGET=2 MINIMUM_IP_TARGET=10

Warning

This restarts the aws-node pods and means more EC2 API calls when pods churn. Pod start-up can slow during bursts. Choose values that fit your pod density, and test outside production first.

#4. Add capacity

If existing VPC ranges have free space, skip to the create-subnet step. Check first:

aws ec2 describe-vpcs --vpc-ids $VPC \
  --query 'Vpcs[].CidrBlockAssociationSet[].{Cidr:CidrBlock,State:CidrBlockState.State}' --output table
aws ec2 describe-subnets --filters Name=vpc-id,Values=$VPC \
  --query 'Subnets[].CidrBlock' --output text

#Add a secondary CIDR if the VPC is full

Check these rules before you pick a range:

• The block must be between /28 and /16 and can't overlap an existing VPC block.
• It can't be the same size as or larger than a destination in any VPC route table. A route to a /24, for example, blocks associating that /24 or anything containing it.
• Other RFC 1918 ranges are restricted. With a primary in 10.0.0.0/8, you can't add a block from 172.16.0.0/12 or 192.168.0.0/16. Blocks from 100.64.0.0/10 are allowed.
• Primary and secondary blocks count toward a per-VPC quota. Check the current limit in Service Quotas.
• The range must not overlap anything you route to, including peered VPCs, transit gateway networks and on-premises.

aws ec2 associate-vpc-cidr-block --vpc-id $VPC --cidr-block 100.64.0.0/20

Wait until the state shows associated. The local route is added to the VPC's route tables automatically.

#Create the subnets and copy the old subnet's settings

Create one subnet per AZ you need:

aws ec2 create-subnet --vpc-id $VPC --availability-zone eu-west-1a \
  --cidr-block 100.64.0.0/22 \
  --tag-specifications 'ResourceType=subnet,Tags=[{Key=Name,Value=app-a-2}]'

aws ec2 associate-route-table --route-table-id rtb-0abc1234 --subnet-id subnet-0new1234

A new subnet does not inherit the old subnet's settings. Before moving traffic, check:

• Route table: Without an explicit association, the subnet uses the VPC's main route table.
• Network ACL: New subnets join the default NACL, so re-associate a custom one if you use it.
• Return routes: Add the new range to peer VPC route tables, transit gateway routes and on-premises routing.
• Allow rules: Update security group rules, NACLs and firewalls that name the old CIDR.
• Subnet tags: Copy discovery tags, such as kubernetes.io/role/internal-elb and karpenter.sh/discovery.

#Move workloads

Warning

Each of these lists replaces the previous one, so include every subnet you want to keep. set-subnets on an ALB needs at least two AZs and can change its node IPs, so do it in a change window if any client caches them. NLB subnet changes have different rules, so check the NLB docs first.

Check Lambda's status after the update. It's complete when LastUpdateStatus is Successful.

#Prevent it next time

• Alert before you run out. VPC IPAM publishes SubnetIPUsage to the AWS/IPAM namespace, and AWS suggests alarming on its MAX statistic. It doesn't publish every subnet in large estates, so check the IPAM metric docs for coverage.
• Or publish your own metric from a scheduled job:

aws ec2 describe-subnets --query 'Subnets[].[SubnetId,AvailableIpAddressCount]' --output text |
while read -r id free; do
  aws cloudwatch put-metric-data --namespace Custom/VPC --metric-name FreeIPs \
    --dimensions SubnetId="$id" --value "$free"
done

• Separate the bursty consumers. Give load balancers, Lambda and pods their own subnets. AWS recommends at least a /27 with 8 free addresses per ALB subnet, and scaling a neighbour shouldn't starve your load balancer.
• Size for failure. During an AZ outage, the surviving subnets must absorb the extra instances, tasks and functions.
• Reserve ranges early. Allocate a spare block per VPC in your IP plan, so you aren't choosing a non-overlapping range during an incident.