Home / AWS

AWSTroubleshootingIntermediate

Instances in private subnets lost internet access: NAT gateway troubleshooting

Check routes, the NAT gateway's own subnet, NACLs and port allocation errors when private instances can't reach the internet or AWS APIs.

Deepak Bhardwaj2 min read

Applies toAmazon VPC NAT gateways (public), EC2/EKS/Lambda in private subnets

Symptoms

  • yum/apt updates time out, and pulls from Docker Hub or GitHub fail.
  • AWS API calls from private instances hang (no VPC endpoints).
  • The problem is intermittent under load → suspect port allocation.

1. Route path

# Private subnet route table: 0.0.0.0/0 -> nat-...
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=subnet-0private \
  --query "RouteTables[].Routes[?DestinationCidrBlock=='0.0.0.0/0']"

# NAT gateway state and its subnet
aws ec2 describe-nat-gateways --nat-gateway-ids nat-0abc \
  --query "NatGateways[].{state:State,subnet:SubnetId,eip:NatGatewayAddresses[0].PublicIp,type:ConnectivityType}"
  • The NAT gateway is available and public (a private NAT gateway has no internet path).
  • The NAT gateway is in a public subnet whose route table has 0.0.0.0/0 → igw-....
  • The private subnet routes to a NAT gateway in the same AZ. Routing across AZs works, but costs more and fails if that AZ has problems.

2. NACLs

The NAT gateway's subnet NACL must allow:

  • Inbound from the private subnets (any port they use), and inbound ephemeral ports 1024–65535 from 0.0.0.0/0 for return traffic.
  • Outbound to 0.0.0.0/0 on the destination ports (80/443), and outbound ephemeral ports back to the private subnets.

Default NACLs allow everything. Custom ones are where this breaks.

3. Port allocation errors

aws cloudwatch get-metric-statistics --namespace AWS/NATGateway --metric-name ErrorPortAllocation \
  --dimensions Name=NatGatewayId,Value=nat-0abc --statistics Sum \
  --start-time "$(date -u -d '-6 hours' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" --period 300

A NAT gateway supports up to about 55,000 simultaneous connections to each unique destination (IP, port, protocol) per public IP. Heavy traffic to one endpoint (an S3 bucket, a single API) runs out of ports.

Fixes:

  • Add secondary public IPs to the NAT gateway. Each adds port capacity.
  • Use VPC endpoints for AWS services (S3 and DynamoDB gateway endpoints are free) to take that traffic off the NAT.
  • Reuse connections in the application (HTTP keep-alive, connection pools).

Also watch PacketsDropCount and ConnectionEstablishedCount.

4. Test from the instance

curl -sS -m 5 -o /dev/null -w "%{http_code} %{remote_ip}\n" https://aws.amazon.com
curl -sS -m 5 https://checkip.amazonaws.com    # should print the NAT gateway's EIP

If checkip shows a different IP, traffic is taking another path, such as a firewall appliance or a proxy.

5. Other egress paths

  • An AWS Network Firewall or third-party appliance between the subnet and the NAT may be dropping traffic. Check its logs.
  • A proxy configured in /etc/environment or in app settings that has gone down.

Tip

Alarm on ErrorPortAllocation > 0 and on NAT BytesOutToDestination trends. Rising NAT data processing usually means traffic that should go through a VPC endpoint.

Related fixes