Home / OCI

OCITroubleshootingIntermediate

OCI Load Balancer: backend health is CRITICAL

Read the health checker's status message, then check security rules, the host firewall and the health check settings to bring backends back to OK.

Deepak Bhardwaj3 min read

Applies toOCI Flexible Load Balancer, Oracle Linux/Ubuntu backends

1. Read the health status

LB=ocid1.loadbalancer.oc1...
oci lb backend-set-health get --load-balancer-id $LB --backend-set-name bs-web
oci lb backend-health get --load-balancer-id $LB --backend-set-name bs-web --backend-name 10.0.2.15:8080

The health-check-results field contains a status message. That message tells you where to look:

Message Likely cause Go to
Connection failed / timed out Security rule, host firewall, or the app isn't listening Steps 2–3
Status code mismatch Health URL returns 301/401/404 instead of 200 Step 4
Regex mismatch Response body doesn't match the expected pattern Step 4
Connection reset TLS/protocol mismatch (HTTP check against an HTTPS port) Step 4

Overall states: OK (all healthy), WARNING (some unhealthy), CRITICAL (none healthy), UNKNOWN (no results yet or LB still provisioning).

2. Security lists and NSGs

Health checks come from the load balancer subnet, not from clients. The backend subnet must allow:

  • Ingress TCP on the backend port from the LB subnet CIDR (or from the LB's NSG).
  • The matching return traffic. This happens automatically with stateful rules, but must be added explicitly if you use stateless rules.
oci network security-list get --security-list-id $SL --query "data.\"ingress-security-rules\"" 
oci network nsg rules list --nsg-id $NSG --all --output table

3. On the backend host

sudo ss -ltnp | grep 8080                  # is the app listening, and on 0.0.0.0 rather than 127.0.0.1?
curl -sv http://localhost:8080/health      # does the health URL return 200?
sudo firewall-cmd --list-ports             # Oracle Linux: firewalld is on by default
sudo firewall-cmd --permanent --add-port=8080/tcp && sudo firewall-cmd --reload

Warning

Oracle Linux platform images ship with firewalld enabled and Ubuntu images have iptables rules in place. A wide-open security list doesn't help if the host firewall drops the packet.

4. Check the health checker settings

oci lb backend-set get --load-balancer-id $LB --backend-set-name bs-web --query "data.\"health-checker\""
  • protocol: HTTP or TCP. Use TCP while you debug, to separate network problems from application problems.
  • url-path: should be a lightweight endpoint that doesn't redirect to login.
  • return-code: defaults to 200. Match it to what your health endpoint actually returns.
  • port: leave it at 0 to use the backend port, or set it explicitly for a separate health port.
oci lb health-checker update --load-balancer-id $LB --backend-set-name bs-web \
  --protocol HTTP --port 8080 --url-path /health --return-code 200 \
  --interval-in-millis 10000 --timeout-in-millis 3000 --retries 3

5. Still unhealthy?

  • Use a VCN Network Path Analyzer test from the LB subnet to the backend IP and port to confirm reachability.
  • Check the LB error logs (enable them in Logging) for per-request backend errors.
  • For SSL to the backend, make sure the backend-set SSL config trusts the backend certificate's CA.

Prevent it next time

Alarm on the UnHealthyBackendServers metric for each backend set (namespace oci_lbaas) with a threshold above 0. That gives you a warning before the backend set goes CRITICAL.

Related fixes