Home / AWS

AWSTroubleshootingIntermediate

Sev1: private subnets lost outbound internet — NAT gateway, route tables and port allocation errors

Instances, Lambda functions and EKS pods in private subnets can't reach the internet or AWS APIs. Check routes, NAT gateway state, ErrorPortAllocation and NACLs, then restore egress.

Deepak Bhardwaj3 min read

Applies toAmazon VPC NAT gateways, route tables, network ACLs, AWS CLI v2

Symptoms

  • Package installs, API calls to SaaS services and calls to public AWS endpoints time out from private subnets.
  • EKS nodes fail to join, Lambda in a VPC times out, SSM agents go offline (if no VPC endpoints).
  • Public subnets still work.

1. Check the route from the affected subnet

SUBNET=subnet-0priv1
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=$SUBNET \
  --query "RouteTables[].Routes[?DestinationCidrBlock=='0.0.0.0/0']"

If no route table is explicitly associated, the subnet uses the VPC main route table:

aws ec2 describe-route-tables --filters Name=vpc-id,Values=vpc-0abc Name=association.main,Values=true
Route state Meaning
0.0.0.0/0 → nat-xxx, State: active Route OK, look at the NAT gateway
State: blackhole NAT gateway deleted or failed
No default route Route removed or subnet moved to another table

2. Check the NAT gateway

aws ec2 describe-nat-gateways --nat-gateway-ids nat-0abc \
  --query "NatGateways[].{state:State, subnet:SubnetId, eip:NatGatewayAddresses[0].PublicIp, fail:FailureMessage}"
  • deleted or failed → create a new one in a public subnet with an Elastic IP and update routes.
  • available but traffic fails → check the NAT's subnet has a route 0.0.0.0/0 → igw-....

Create a replacement:

EIP=$(aws ec2 allocate-address --domain vpc --query AllocationId --output text)
NAT=$(aws ec2 create-nat-gateway --subnet-id subnet-0pub1 --allocation-id $EIP \
  --query NatGateway.NatGatewayId --output text)
aws ec2 wait nat-gateway-available --nat-gateway-ids $NAT
aws ec2 replace-route --route-table-id rtb-0priv --destination-cidr-block 0.0.0.0/0 --nat-gateway-id $NAT

Warning

A new NAT gateway has a new public IP. Partners that allow-list your egress IP will block you. If the old Elastic IP still exists (released only when you release it), reuse that allocation ID.

3. Port exhaustion and packet drops

aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name ErrorPortAllocation \
  --dimensions Name=NatGatewayId,Value=nat-0abc --statistics Sum --period 60 \
  --start-time "$(date -u -d '-1 hour' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)"

Also check PacketsDropCount and ActiveConnectionCount.

ErrorPortAllocation > 0 means too many concurrent connections to the same destination IP and port. Fixes:

  • Add secondary Elastic IPs to the NAT gateway (each adds source ports):
aws ec2 associate-nat-gateway-address --nat-gateway-id nat-0abc --allocation-ids eipalloc-0new
  • Reuse connections (HTTP keep-alive, connection pools) in the app.
  • Move AWS service traffic (S3, DynamoDB, ECR, STS) to VPC endpoints so it bypasses NAT.

4. Network ACLs

NACLs are stateless. The private subnet NACL must allow outbound to 0.0.0.0/0 and inbound return traffic on ephemeral ports 1024-65535. The NAT's public subnet NACL needs the same.

aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=$SUBNET \
  --query "NetworkAcls[].Entries[?RuleAction=='deny']"

5. Verify from inside

aws ssm start-session --target i-0abc   # if SSM works through endpoints
curl -s -m 5 https://checkip.amazonaws.com   # should show the NAT's public IP

Prevent it next time

  • One NAT gateway per AZ, with each private subnet routed to the NAT in its own AZ.
  • Gateway endpoints for S3 and DynamoDB, interface endpoints for ECR, STS, SSM, Logs.
  • Alarms on ErrorPortAllocation, PacketsDropCount and route blackholes (AWS Config rule or custom check).
  • Protect NAT gateways and route tables with IaC and change review.

Related fixes