Network Load Balancer connections time out or reset: client IP preservation, hairpinning, security groups and idle timeout

Clients connecting through a Network Load Balancer (NLB) get timeouts, intermittent failures or connection reset by peer. The NLB has no access logs for plain TCP listeners, so use target health, attributes and metrics to narrow it down.

The symptom usually points to the cause:

#Check target health first

TG_ARN=arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/app-tg/1234567890123456

aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
  --query 'TargetHealthDescriptions[].[Target.Id,Target.Port,TargetHealth.State,TargetHealth.Reason,TargetHealth.Description]' \
  --output table

Two behaviours can mislead you:

• Fail-open. If every registered target is unhealthy, the NLB routes to all of them. Traffic still reaches broken targets, so failures can look random instead of total.
• TCP health checks only test the handshake. A hung application whose port still accepts connections shows as healthy. Use an application check if you have an endpoint for one:

aws elbv2 modify-target-group --target-group-arn "$TG_ARN" \
  --health-check-protocol HTTP --health-check-path /healthz

Targets also get more health check requests than the interval suggests, because NLB health checks are distributed and use consensus. That's normal, so don't rate-limit them.

#Check client IP preservation

Most NLB timeouts that aren't health-related come back to this attribute.

aws elbv2 describe-target-group-attributes --target-group-arn "$TG_ARN" \
  --query "Attributes[?Key=='preserve_client_ip.enabled']"

UDP and QUIC target groups can't turn it off.

#What it changes on the target

AWS also lists preservation as unsupported in these cases:

• Traffic through a transit gateway.
• Traffic through a Gateway Load Balancer endpoint.
• Target groups containing PrivateLink network interfaces or another NLB's network interface.
• Older instance types such as M1, M3 and T1.

#Hairpinning: a target calling its own NLB

NAT loopback isn't supported with preservation on. When the NLB routes the request back to the instance that sent it, the connection times out. Other requests from that instance land on other targets and succeed, which is why the failure looks intermittent. The same applies to EKS pods calling an NLB when the backend is on the same worker node.

Run this from a target several times:

for i in $(seq 1 20); do
  curl -s -o /dev/null -m 3 -w "%{http_code} %{time_total}\n" http://my-nlb-1234.elb.eu-west-1.amazonaws.com/
done

If some calls time out here but none do from outside the target group, it's hairpinning. Either call the service directly, for example through localhost or a service address, or turn preservation off.

#Turning preservation off

Warning

With preservation off, the application logs and any IP allow lists see the NLB's private IP, not the client's. Before you change it, update target security groups to allow the NLB's IPs or security group, or targets will go unhealthy. The change only affects new TCP connections.

aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" \
  --attributes Key=preserve_client_ip.enabled,Value=false

If the application needs the real client IP, enable Proxy Protocol v2:

aws elbv2 modify-target-group-attributes --target-group-arn "$TG_ARN" \
  --attributes Key=proxy_protocol_v2.enabled,Value=true

Caution

An application that doesn't expect the Proxy Protocol header treats it as bad data and connections fail. Enable it on the application and the target group in the same change window, and test on a non-production target group first.

#Check security groups and NACLs

Find the private IPs of the NLB nodes:

aws ec2 describe-network-interfaces \
  --filters "Name=description,Values=ELB net/my-nlb/*" \
  --query 'NetworkInterfaces[].[AvailabilityZone,PrivateIpAddress,SubnetId]' --output table

Check whether the NLB has security groups:

aws elbv2 describe-load-balancers --names my-nlb \
  --query 'LoadBalancers[0].[SecurityGroups,Scheme]'

• NLB with security groups. Reference the NLB's security group in the target's inbound rules. This works with preservation on or off. The NLB's outbound rules must allow the traffic and health check ports, because health checks are subject to outbound rules but not inbound rules.
• NLB without security groups. Allow the NLB node IPs for health checks, then the client CIDRs if preservation is on.
• NACLs. With preservation on, the target subnet's NACL must allow client IPs in and ephemeral ports back out.

Important

You can't add security groups to an NLB created without them. To get them, you need a new NLB, which means a new DNS name to cut over.

If SecurityGroupBlockedFlowCount_Inbound_TCP is rising, the NLB's own security group is dropping the traffic. PrivateLink consumers can be exempt from the inbound rules by turning off enforcement:

aws elbv2 set-security-groups --load-balancer-arn "$NLB_ARN" \
  --security-groups sg-0123456789abcdef0 \
  --enforce-security-group-inbound-rules-on-private-link-traffic off

#Read the reset and port metrics

Metrics are in the AWS/NetworkELB namespace. The LoadBalancer dimension is the end of the ARN, in the form net/my-nlb/1234567890123456.

aws cloudwatch get-metric-statistics --namespace AWS/NetworkELB \
  --metric-name TCP_ELB_Reset_Count \
  --dimensions Name=LoadBalancer,Value=net/my-nlb/1234567890123456 \
  --start-time "$(date -u -d '1 hour ago' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
  --period 60 --statistics Sum --output table

If PortAllocationErrorCount is above zero, you're hitting the limit of about 55,000 connections per target IP and port. Add targets, register more ports, or re-enable preservation where your traffic path supports it.

#Idle timeout

After the timeout, the connection is closed. A client or target that sends data on it receives a TCP RST. This usually hits database and connection pools, where the first query after a quiet period fails.

The better fix is keepalives below the timeout. On Linux, the kernel default of 7,200 seconds is far too long, and the application must also enable SO_KEEPALIVE on its sockets:

sudo sysctl -w net.ipv4.tcp_keepalive_time=300
echo 'net.ipv4.tcp_keepalive_time=300' | sudo tee /etc/sysctl.d/90-keepalive.conf

Or raise the timeout on a TCP listener:

aws elbv2 modify-listener-attributes --listener-arn "$LISTENER_ARN" \
  --attributes Key=tcp.idle_timeout.seconds,Value=900

If you go above 350 seconds, the TcpEstablishedTimeout on the target's network interface should be equal to or greater than the NLB timeout.

#Intermittent failures from clients behind NAT

With preservation on, a NAT device can reuse the same source port across several NLB IPs. The target then sees two connections with the same client IP and port, and one fails. AWS lists these fixes:

• Disable cross-zone load balancing.
• Don't register the same IP and port in more than one target group.
• Disable client IP preservation.

Check the current setting:

aws elbv2 describe-load-balancer-attributes --load-balancer-arn "$NLB_ARN" \
  --query "Attributes[?Key=='load_balancing.cross_zone.enabled']"

#Prevent it next time

• Create new NLBs with a security group, and reference it from target security groups.
• Set preserve_client_ip.enabled explicitly in your infrastructure code. The default depends on target type and protocol.
• Don't let services call themselves through their own NLB.
• Alarm on UnHealthyHostCount, PortAllocationErrorCount and TCP_ELB_Reset_Count.
• Set TCP keepalives on connection pools during the build, not after the first incident.
• Remember that NLBs don't multiplex connections. Targets moved from an ALB can see many more TCP connections, so check max_connections and file descriptor limits before you migrate.