Sev1: OCI load balancer returns 502 — every backend critical, health check and security rule fixes

#Symptoms

• Users get 502 Bad Gateway from the load balancer.
• Backend set health is CRITICAL; every backend shows a failed health check.
• The application seems fine when tested directly on the instance.

#1. Read health status

LB=ocid1.loadbalancer.oc1..aaaa
oci lb load-balancer-health get --load-balancer-id $LB
oci lb backend-set-health get --load-balancer-id $LB --backend-set-name bs-web
oci lb backend-health get --load-balancer-id $LB --backend-set-name bs-web --backend-name 10.0.2.15:8080

The backend health output includes health-check-results with a health-check-status per load balancer node, for example CONNECT_FAILED, TIMED_OUT, INVALID_STATUS_CODE, OFFLINE.

#2. Fix the health check

oci lb backend-set get --load-balancer-id $LB --backend-set-name bs-web --query 'data."health-checker"'

Common problems after a release: the URL path now redirects (/ to /login), needs auth, or the app listens on a different port. Update the checker:

oci lb health-checker update --load-balancer-id $LB --backend-set-name bs-web \
  --protocol HTTP --port 8080 --url-path /health --return-code 200 \
  --interval-in-millis 10000 --timeout-in-millis 3000 --retries 3 --force

Test the exact URL from a host in the load balancer subnet:

curl -s -o /dev/null -w "%{http_code}\n" http://10.0.2.15:8080/health

#3. Security lists and NSGs

The load balancer connects from its own subnet's IPs. The backend subnet's security list or the backend NSG must allow ingress from the LB subnet CIDR on the backend port, and the LB subnet must allow egress to the backends.

oci network nsg rules list --nsg-id $BACKEND_NSG --direction INGRESS --output table
oci network security-list get --security-list-id $BACKEND_SL --query 'data."ingress-security-rules"' --output table

If a recent change removed the rule, add it back:

oci network nsg rules add --nsg-id $BACKEND_NSG --security-rules \
  '[{"direction":"INGRESS","protocol":"6","source":"10.0.1.0/24","sourceType":"CIDR_BLOCK","tcpOptions":{"destinationPortRange":{"min":8080,"max":8080}},"description":"LB to app INC12345"}]'

Also check the host firewall on the instances (firewall-cmd --list-all on Oracle Linux; OCI images enable firewalld by default):

sudo firewall-cmd --permanent --add-port=8080/tcp && sudo firewall-cmd --reload

#4. Backend application

sudo ss -ltnp | grep 8080
sudo systemctl status myapp

If the app binds only to 127.0.0.1, the load balancer can't reach it. Bind to the private IP or 0.0.0.0.

#5. Backends set offline or drained

oci lb backend list --load-balancer-id $LB --backend-set-name bs-web \
  --query 'data[].{name:name, offline:offline, drain:drain, backup:backup}' --output table
oci lb backend update --load-balancer-id $LB --backend-set-name bs-web --backend-name 10.0.2.15:8080 \
  --offline false --drain false --weight 1 --backup false

#6. TLS to backends

If the backend set uses SSL, check the backend certificate and the trusted CA bundle on the backend set. An expired backend certificate fails health checks the same way.

#Prevent it next time

• A lightweight unauthenticated /health endpoint that only returns 200 when the app is ready.
• Alarms on backend set health and on the UnHealthyBackendServers-style metrics in the oci_lbaas namespace.
• Keep NSG rules for LB-to-backend in Terraform next to the load balancer.
• Spread backends across fault domains and ADs.