Home / OCI

OCITroubleshootingAdvanced

OCI Network Load Balancer with client IP preservation: connections that hang, hairpin traffic and asymmetric return paths

After enabling source IP preservation on an OCI Network Load Balancer, some clients hang while others work. Understand how replies travel back, fix security rules and routes, and handle clients in the same subnet as the backends.

Deepak Bhardwaj4 min read

Applies toOCI Network Load Balancer (private and public), backend sets with source IP preservation, VCN security lists and NSGs

Symptoms

  • Clients through the Network Load Balancer (NLB) time out after enabling preserve source IP on the backend set; with it off, everything works.
  • Clients in the same subnet or VCN as the backends fail; clients from other networks work.
  • Backends see the real client IP but the health checks are fine, so the NLB shows all backends healthy.

What changes when the client IP is preserved

Setting Backend sees Reply goes to
Source IP preservation off The NLB's IP The NLB, which returns it to the client
Source IP preservation on The real client IP Routed towards the client IP by the backend's normal routing; it must pass back through the NLB to be translated correctly

When the client is in a remote network, the reply follows the subnet route table back through the path the NLB handles, and it works. When the client is in the same subnet (or reachable directly), the backend replies straight to the client, bypassing the NLB. The client sent its SYN to the NLB IP but gets a SYN-ACK from the backend IP, and drops it. This is the hairpin problem.

Step 1: check the NLB and backend set configuration

NLB=ocid1.networkloadbalancer.oc1.eu-frankfurt-1.aaaa...

oci nlb network-load-balancer get --network-load-balancer-id $NLB \
  --query 'data.{private:"is-private",preserveSrcDst:"is-preserve-source-destination",symmetric:"is-symmetric-hash-enabled",subnet:"subnet-id"}'

oci nlb backend-set list --network-load-balancer-id $NLB \
  --query 'data.items[].{name:name,preserveSource:"is-preserve-source",policy:policy}' --output table

oci nlb backend-health get --network-load-balancer-id $NLB \
  --backend-set-name bs-app --backend-name 10.0.2.10:443 2>/dev/null

is-preserve-source on the backend set is the per-set client IP preservation flag. is-preserve-source-destination and symmetric hashing on the NLB itself are for transparent ("bump in the wire") designs with network appliances as backends.

Step 2: find where the clients are

List the failing client IPs (from app or client logs) and compare with:

  • The backend subnets' CIDRs.
  • The NLB's subnet.
  • Whether a route sends traffic between them via a gateway or locally within the VCN.

If failing clients are in the same VCN subnet as the backends, it's the hairpin case.

Step 3: fix the hairpin case

Options, from simplest:

Option Effect
Turn off source IP preservation for the backend set used by internal clients Backend sees the NLB IP; replies go back via the NLB. Lose client IP for those flows
Put clients and backends in different subnets with routing that brings replies back through the expected path Removes the direct path
Internal clients call backends directly (or via a separate NLB/backend set without preservation) Separate entry for internal and external traffic
Use proxy protocol / X-Forwarded-For with the flexible Load Balancer (L7) when only HTTP(S) is needed Client IP carried in headers, no routing tricks

Step 4: security rules with preserved client IPs

With preservation on, backends receive packets from the client IPs, not from the NLB subnet. Security lists and NSGs on the backend subnet must allow the client ranges on the service port, and stateless rules (if you use them) must allow the return traffic too.

oci network nsg rules list --nsg-id <backend-nsg> \
  --query 'data[?direction==`INGRESS`].{src:source,proto:protocol,ports:"tcp-options"."destination-port-range"}' --output table

Health checks come from the NLB, so they still succeed even when client traffic is blocked. That's why "all backends healthy, clients failing" is typical.

Tip

Prefer stateful NSG rules for backends behind an NLB. Stateless rules need explicit rules for every return flow, which is where half-open connections come from.

Step 5: test both paths

From an external client and from a client in the backend subnet:

curl -v --connect-timeout 5 https://10.0.1.50/health        # NLB private IP

On a backend, capture to see who sends what:

sudo tcpdump -ni ens3 port 443 and host <client-ip>

A SYN from the client IP followed by a SYN-ACK leaving towards it, but no ACK coming back, confirms the client dropped the reply because it came from the wrong source.

Step 6: transparent mode with network appliances

If the NLB load-balances across firewalls (preserve source and destination), both directions of each flow must hit the same appliance. Enable symmetric hashing on the NLB and route return traffic through the same NLB, as described in OCI's network firewall/NLB reference architectures. Otherwise the same asymmetry problem appears between appliances.

Prevent it next time

  • Decide per backend set whether you need the client IP, and why.
  • Keep internal consumers on a separate path from external ones.
  • Allow client CIDRs explicitly in backend NSGs when preservation is on, and document them.
  • Add a synthetic check from inside the VCN as well as from outside.

Related fixes