AWS Network Firewall dropping or passing the wrong traffic: rule order, default actions, asymmetric routing and idle timeouts

#Symptom

One of these shows up after you put AWS Network Firewall in the path:

• A flow your rules clearly allow times out or gets reset.
• TCP connections that sit idle for a few minutes (database pools, SSH, long polls) hang the next time they send data.
• Traffic you wrote a drop rule for still gets through.
• Everything works in one Availability Zone and fails from another.

Almost every case comes down to one of four things: the policy's rule order, the stateful default actions, routing that skips the firewall in one direction, or the TCP idle timeout.

#Check 1: read the policy as the engine sees it

Find out how the engine is set up before you read any rules:

FW=egress-fw
POLICY_ARN=$(aws network-firewall describe-firewall --firewall-name "$FW" \
  --query "Firewall.FirewallPolicyArn" --output text)

aws network-firewall describe-firewall-policy --firewall-policy-arn "$POLICY_ARN" \
  --query "FirewallPolicy.{Stateless:StatelessDefaultActions,StatelessFrag:StatelessFragmentDefaultActions,StatefulDefault:StatefulDefaultActions,Engine:StatefulEngineOptions,Groups:StatefulRuleGroupReferences}"

#Action order vs strict order

Under action order (DEFAULT_ACTION_ORDER), Suricata evaluates all pass rules first, then drop, then reject, then alert. Priorities and the order you wrote the rules in don't apply. So a broad pass tcp any any -> any any in any rule group overrides a narrow drop somewhere else, which is the usual reason "blocked" traffic gets through.

Under strict order, rule groups run in the priority order set in the policy, and rules inside a group run in the order they're written. The first match that has an action wins. AWS recommends strict order with aws:drop_established and aws:alert_established as default actions.

Warning

You can only set rule order when you create the policy or rule group. You can't switch an existing object from action order to strict order. Build new strict-order rule groups and a new policy, then swap the policy on the firewall with associate-firewall-policy during a change window.

#drop_established vs drop_strict

aws:drop_strict drops any packet that no rule passes, and that includes the TCP handshake. Domain list rules (HTTP Host and TLS SNI) can't match until the handshake has finished and the first data packet arrives, so with drop_strict every domain-allow rule fails. aws:drop_established lets the handshake through, then drops established flows that no rule passed. That's the setting domain filtering needs.

#Check 2: find the rule that acted

Turn on alert and flow logs if they aren't already on:

aws network-firewall describe-logging-configuration --firewall-name "$FW"

aws network-firewall update-logging-configuration --firewall-name "$FW" \
  --logging-configuration '{"LogDestinationConfigs":[
    {"LogType":"ALERT","LogDestinationType":"CloudWatchLogs","LogDestination":{"logGroup":"/nfw/egress/alert"}},
    {"LogType":"FLOW","LogDestinationType":"CloudWatchLogs","LogDestination":{"logGroup":"/nfw/egress/flow"}}]}'

Each update-logging-configuration call can add or remove only one destination. If no logging is configured yet, send only the ALERT entry first, then run the command again with both entries.

Search the alert log for the client's address in CloudWatch Logs Insights:

fields @timestamp, event.src_ip, event.dest_ip, event.dest_port, event.alert.action, event.alert.signature_id, event.alert.signature
| filter event.src_ip = "10.20.4.17" or event.dest_ip = "10.20.4.17"
| sort @timestamp desc
| limit 50

A blocked action with a signature_id names the rule that dropped the flow. If the flow appears in the flow log and has no alert entry, the default action dropped it, or the traffic never came back through the firewall (see Check 3). Give every custom rule a unique sid and a msg so the alert log points at it. Rules without alert keywords only log when the default action is aws:alert_established or aws:alert_strict.

Check the per-engine counters to see whether the drops come from the stateless or the stateful engine:

aws cloudwatch get-metric-statistics --namespace AWS/NetworkFirewall \
  --metric-name DroppedPackets \
  --dimensions Name=FirewallName,Value=$FW Name=AvailabilityZone,Value=eu-west-1a Name=Engine,Value=Stateful \
  --start-time $(date -u -d '-1 hour' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Sum

#Check 3: asymmetric routing

The stateful engine has to see both directions of a flow through the same endpoint. When the reply skips the firewall, or comes back through an endpoint in a different AZ, the engine sees packets for a flow it has no record of. With drop_established it drops them, and the client gets a timeout.

The usual causes:

# Appliance mode on the inspection VPC attachment
aws ec2 describe-transit-gateway-vpc-attachments --transit-gateway-attachment-ids tgw-attach-0abc \
  --query "TransitGatewayVpcAttachments[].Options.ApplianceModeSupport"

aws ec2 modify-transit-gateway-vpc-attachment --transit-gateway-attachment-id tgw-attach-0abc \
  --options ApplianceModeSupport=enable

# Map each endpoint to its AZ
aws network-firewall describe-firewall --firewall-name "$FW" \
  --query "FirewallStatus.SyncStates"

Compare the SyncStates output (one VPC endpoint ID per AZ) with every route table that sends traffic to a vpce- target. Each route table should point at the endpoint in its own AZ. Routing across AZs works for stateless traffic, but stateful flows break whenever the return path is different.

#Check 4: idle connections dying

If a connection stays quiet for longer than TcpIdleTimeoutSeconds (350 seconds by default), the firewall forgets the flow. With drop_established, the next data packet gets dropped. The client then retransmits until its own timeout, which looks like a hang. This hits database connection pools, JDBC clients and long-lived gRPC streams.

You can fix it in two ways. Either is enough, and doing both is safest:

1. Have the client send TCP keepalives more often than the timeout, for example net.ipv4.tcp_keepalive_time = 300 on Linux, or the driver's keepalive setting.
2. Raise the timeout on the policy. The range is 60 to 6000 seconds.

TOKEN=$(aws network-firewall describe-firewall-policy --firewall-policy-arn "$POLICY_ARN" --query UpdateToken --output text)
aws network-firewall describe-firewall-policy --firewall-policy-arn "$POLICY_ARN" \
  --query FirewallPolicy > policy.json
# edit policy.json: "StatefulEngineOptions": {"RuleOrder":"STRICT_ORDER","StreamExceptionPolicy":"REJECT","FlowTimeouts":{"TcpIdleTimeoutSeconds":1800}}
aws network-firewall update-firewall-policy --firewall-policy-arn "$POLICY_ARN" \
  --update-token "$TOKEN" --firewall-policy file://policy.json

Caution

Changing StatefulEngineOptions can restart the stateful engine. Existing flows are then handled according to StreamExceptionPolicy: DROP and REJECT break every live connection, while CONTINUE keeps evaluating them without their earlier context. Make the change in a maintenance window. A new timeout only applies to connections opened after the change.

#Quick mapping

#Prevent it next time

• Build every policy and rule group with STRICT_ORDER, aws:drop_established and aws:alert_established so any dropped flow leaves a log entry.
• Keep alert logs on all the time, and alarm on DroppedPackets for each AZ.
• Test new rule groups by adding alert rules first. Change them to drop once the alert log shows they only match what you meant.
• Make appliance mode and per-AZ endpoint routing part of the inspection VPC module, not something people remember to do.
• Write the TCP idle timeout and the client keepalive standard into your connection pool guidelines.