Transit Gateway routing problems: asymmetric flows through inspection VPCs, appliance mode and blackhole routes

#Symptoms

• VPC-to-VPC or VPC-to-on-prem traffic through a central inspection VPC works for some flows and hangs for others.
• Problems depend on which Availability Zone the client or server is in.
• After adding a VPC or VPN attachment, part of the network becomes unreachable with no error.

#How a packet crosses a Transit Gateway

1. The VPC route table of the source subnet sends traffic to the TGW.
2. The TGW looks up the TGW route table associated with the source attachment.
3. That table's route (static or propagated) chooses the destination attachment.
4. The destination VPC's route table for the TGW attachment subnet decides where it goes inside that VPC.

Each step must exist in both directions.

#Step 1: check associations and propagations

TGW_RT=tgw-rtb-0123456789abcdef0

aws ec2 get-transit-gateway-route-table-associations --transit-gateway-route-table-id $TGW_RT \
  --query "Associations[].{att:TransitGatewayAttachmentId,type:ResourceType,res:ResourceId}" --output table

aws ec2 get-transit-gateway-route-table-propagations --transit-gateway-route-table-id $TGW_RT \
  --query "TransitGatewayRouteTablePropagations[].{att:TransitGatewayAttachmentId,state:State}" --output table

An attachment is associated with exactly one TGW route table. In an inspection design, spoke attachments are associated with a "spoke" table whose default route points to the inspection VPC, and the inspection attachment is associated with a "firewall" table that has the spoke routes propagated.

#Step 2: find blackhole and missing routes

aws ec2 search-transit-gateway-routes --transit-gateway-route-table-id $TGW_RT \
  --filters Name=state,Values=blackhole --output table

aws ec2 search-transit-gateway-routes --transit-gateway-route-table-id $TGW_RT \
  --filters Name=route-search.longest-prefix-match,Values=10.20.5.10/32 \
  --query "Routes[].{cidr:DestinationCidrBlock,type:Type,state:State,att:TransitGatewayAttachments[0].TransitGatewayAttachmentId}"

#Step 3: the AZ asymmetry problem (appliance mode)

By default, the TGW tries to keep traffic in the same AZ it arrived from. In an inspection VPC with one firewall per AZ:

• Request: client in AZ-a → TGW → firewall in AZ-a → server in AZ-b.
• Reply: server in AZ-b → TGW → firewall in AZ-b → client.

Two different stateful firewalls see the two halves of one flow, and drop them.

Appliance mode on the inspection VPC attachment makes the TGW pick one AZ per flow (by flow hash) and keep both directions there:

aws ec2 modify-transit-gateway-vpc-attachment \
  --transit-gateway-attachment-id tgw-attach-0inspection \
  --options ApplianceModeSupport=enable

aws ec2 describe-transit-gateway-vpc-attachments \
  --transit-gateway-attachment-ids tgw-attach-0inspection \
  --query "TransitGatewayVpcAttachments[0].Options.ApplianceModeSupport"

Warning

Enabling appliance mode changes which AZ some existing flows use. Long-lived connections may reset. Enable it in a maintenance window, and only on the inspection VPC attachment.

#Step 4: check the VPC side

Inside the inspection VPC, the subnet that holds the TGW attachment ENIs must route to the firewall endpoint in the same AZ, and the firewall subnet must route back to the TGW:

aws ec2 describe-route-tables --filters Name=vpc-id,Values=vpc-0inspect \
  --query "RouteTables[].{rt:RouteTableId,subnets:Associations[].SubnetId,routes:Routes[].[DestinationCidrBlock,TransitGatewayId||VpcEndpointId||GatewayId]}"

Common mistakes:

• One route table for TGW subnets in all AZs, pointing to a single firewall endpoint (cross-AZ, and a single point of failure).
• Spoke VPC route tables missing the route to the TGW for the remote prefix (return traffic goes nowhere).

#Step 5: prove the path

Route Analyzer (AWS Network Manager, Global Network required) checks the TGW path between two IPs both ways:

aws networkmanager start-route-analysis --global-network-id global-network-0abc \
  --source TransitGatewayAttachmentArn=arn:aws:ec2:eu-west-1:111122223333:transit-gateway-attachment/tgw-attach-0spokeA,IpAddress=10.10.1.10 \
  --destination TransitGatewayAttachmentArn=arn:aws:ec2:eu-west-1:111122223333:transit-gateway-attachment/tgw-attach-0spokeB,IpAddress=10.20.5.10 \
  --include-return-path

TGW flow logs show what actually crossed the TGW, including the AZ:

aws ec2 create-flow-logs --resource-type TransitGateway \
  --resource-ids tgw-0123456789abcdef0 \
  --log-destination-type cloud-watch-logs --log-group-name /tgw/flowlogs \
  --deliver-logs-permission-arn arn:aws:iam::111122223333:role/flowlogs-role

On the firewall: AWS Network Firewall alert/flow logs, or the appliance's session table, show whether both directions arrive at the same instance.

#Decision table

#Prevent it next time

• Enable appliance mode on every inspection VPC attachment at creation.
• Disable default route table association and propagation on the TGW; set them explicitly per attachment.
• Keep TGW route tables in Terraform/CloudFormation and run Route Analyzer in your network change tests.
• Alert on new blackhole routes with a scheduled script or AWS Config custom rule.