Home / AWS

AWSTroubleshootingIntermediate

Sev1: traffic stopped after a security group or NACL change — prove it with Reachability Analyzer and revert

An app can no longer reach its database or users can't reach the service after a network change. Use CloudTrail, Reachability Analyzer and flow logs to find the blocking rule and restore traffic.

Deepak Bhardwaj3 min read

Applies toAmazon VPC security groups, network ACLs, VPC Reachability Analyzer, AWS CLI v2

Symptoms

  • Connection timeouts (not refusals) between tiers right after a change window.
  • Health checks fail, ALB targets go unhealthy, or RDS connections hang.

1. Find recent network changes

for ev in AuthorizeSecurityGroupIngress RevokeSecurityGroupIngress RevokeSecurityGroupEgress \
          ModifyNetworkInterfaceAttribute CreateNetworkAclEntry ReplaceNetworkAclEntry DeleteNetworkAclEntry \
          ReplaceNetworkAclAssociation; do
  aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=$ev \
    --max-results 5 --query "Events[].{t:EventTime,e:EventName,u:Username}" --output text
done

ModifyNetworkInterfaceAttribute catches someone swapping the security groups on an instance or ENI.

2. Prove the path with Reachability Analyzer

PATH_ID=$(aws ec2 create-network-insights-path --source i-0app --destination eni-0rdsabc \
  --protocol tcp --destination-port 5432 --query NetworkInsightsPath.NetworkInsightsPathId --output text)
AN=$(aws ec2 start-network-insights-analysis --network-insights-path-id $PATH_ID \
  --query NetworkInsightsAnalysis.NetworkInsightsAnalysisId --output text)
aws ec2 describe-network-insights-analyses --network-insights-analysis-ids $AN \
  --query "NetworkInsightsAnalyses[0].{reachable:NetworkPathFound, why:Explanations[0]}"

When unreachable, Explanations names the component, for example a security group with no matching rule or a NACL entry that denies.

3. Security group checks

Security groups are stateful and allow-only.

aws ec2 describe-instances --instance-ids i-0app \
  --query "Reservations[].Instances[].SecurityGroups"
aws ec2 describe-security-group-rules --filters Name=group-id,Values=sg-0db \
  --query "SecurityGroupRules[?IsEgress==\`false\`].{port:FromPort, src:CidrIpv4, srcSg:ReferencedGroupInfo.GroupId}" \
  --output table
Common mistake Fix
DB SG allowed the app SG, but the app instances were moved to a new SG Add a rule referencing the new SG
Outbound rules restricted to specific ports, app needs another Add the egress rule
Rule references a CIDR, app now in a new subnet Prefer SG references over CIDRs

Restore access:

aws ec2 authorize-security-group-ingress --group-id sg-0db \
  --ip-permissions "IpProtocol=tcp,FromPort=5432,ToPort=5432,UserIdGroupPairs=[{GroupId=sg-0app,Description=INC12345}]"

4. NACL checks

NACLs are stateless, evaluated in rule-number order, first match wins.

aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=subnet-0db \
  --query "NetworkAcls[].Entries[].{n:RuleNumber, egress:Egress, action:RuleAction, cidr:CidrBlock, ports:PortRange}" \
  --output table

Check both directions: inbound to the DB port and outbound ephemeral ports (1024-65535) for the replies. A new deny with a low rule number overrides allows below it.

aws ec2 delete-network-acl-entry --network-acl-id acl-0abc --rule-number 90 --ingress

Tip

Timeouts point to SGs, NACLs or routes. A fast "connection refused" means packets arrived and nothing listens on the port; look at the service, not the network.

5. Confirm with flow logs

fields @timestamp, srcAddr, dstAddr, dstPort, action
| filter dstPort = 5432 and action = "REJECT"
| sort @timestamp desc
| limit 50

Run this in CloudWatch Logs Insights against the VPC flow log group. After the fix, REJECTs for the flow should stop.

Prevent it next time

  • Reference security groups, not IPs, between tiers.
  • Manage SGs and NACLs in IaC; use AWS Config to flag drift.
  • Keep NACLs simple (or default allow) and do filtering in SGs.
  • Save Reachability Analyzer paths for critical flows and rerun them after every network change.

Related fixes