OCI Load Balancer 502 and 504 on long requests: tune idle timeout and backend keep-alive

#Symptom

Long-running requests, such as reports, exports or uploads, fail at a suspiciously exact time. Short requests work. Or 502 errors appear at random at low traffic while every backend shows OK.

If the backends are Critical, that's a different problem: see all backends critical.

#How the timeouts interact

There are at least three clocks along the path, and the shortest one ends the request:

Idle timeout counts time with no data moving, not total request time. A request that sends data steadily can run far longer than the idle timeout. A report that thinks for 90 seconds before it sends its first byte will fail with a 60-second idle timeout. The default for HTTP listeners is short (60 seconds, per the documentation at the time of writing), and TCP listeners have a longer one.

#Step 1: get the facts

Check the current listener settings:

oci lb load-balancer get --load-balancer-id "$LB" \
  --query 'data.listeners.*.{name:name,port:port,proto:protocol,idle:"connection-configuration"."idle-timeout",bs:"default-backend-set-name"}'

If this returns nothing useful, view the full JSON with oci lb load-balancer get --load-balancer-id "$LB".

Time the failure from a client:

curl -o /dev/null -sS -w 'code=%{http_code} ttfb=%{time_starttransfer} total=%{time_total}\n' \
  --max-time 600 https://app.contoso.com/reports/slow

A failure at a round number, such as 60.0x seconds, is a timeout. Compare it with the listener value and with your backend settings.

#Step 2: read the access logs

Enable load balancer access and error logs in Logging > Logs > Enable service log, if they aren't already on. Then search:

search "<compartment-ocid>/<log-group-ocid>/<log-ocid>"
| where data.lbStatusCode = '504' or data.lbStatusCode = '502'
| sort by datetime desc

Open a record. Fields worth comparing (names can vary, so check a real record):

How to read the combination:

Metrics in the oci_lbaas namespace help you see the trend. Names can vary, so browse them in the console:

HttpResponses5xx[1m].sum()
BackendTimeouts[1m].sum()

#Step 3: fix 504s on slow requests

First ask whether a request should really be that slow. If a job takes minutes, returning a job ID and polling for the result is more reliable than holding a connection open.

If you do need to allow it, raise the idle timeout. The update command requires the listener's main fields, so include their current values:

oci lb listener update \
  --load-balancer-id "$LB" \
  --listener-name https \
  --default-backend-set-name bs-web \
  --port 443 \
  --protocol HTTP \
  --connection-configuration-idle-timeout 300 \
  --wait-for-state SUCCEEDED

Warning

Updating a listener replaces the settings you pass and may reset those you leave out, such as SSL configuration, hostnames and routing policies. Export the current listener first. Then include everything it already uses, and test on a non-production listener if you can.

Backend timeouts must not be shorter than the listener's:

# nginx as the backend web server
proxy_read_timeout 310s;
proxy_send_timeout 310s;

# gunicorn kills silent workers at this limit, giving a 502 at the LB
gunicorn app:app --timeout 310 --keep-alive 320

Long idle timeouts have a cost. Connections held open use up capacity, and stuck requests stay around for longer, so raise it only as far as you need to.

#Step 4: fix random 502s from keep-alive

The load balancer reuses connections to backends. If a backend closes an idle connection at the moment the load balancer sends a request on it, the client gets a 502.

The backend's keep-alive timeout must be longer than the load balancer's idle timeout.

# nginx: idle timeout 300 s, so choose a higher value
keepalive_timeout 320s;

# Apache httpd
KeepAlive On
KeepAliveTimeout 320

// Node.js
server.keepAliveTimeout = 320000;
server.headersTimeout = 325000;

#Step 5: check WebSocket and streaming endpoints

Connections for WebSockets and server-sent events fail once they sit silent for the idle timeout. Send application heartbeats well inside that limit. For SSE, a comment line every 30 seconds works:

: keepalive

Also turn off response buffering on any reverse proxy in the path, otherwise the heartbeats won't get through.

#Step 6: test the fix

for i in 1 2 3; do
  curl -o /dev/null -sS -w 'code=%{http_code} total=%{time_total}\n' \
    --max-time 600 https://app.contoso.com/reports/slow
done

To test keep-alive, run steady low traffic for an hour across the idle timeout. Random 502s should drop to zero:

hey -z 60m -q 1 -c 2 https://app.contoso.com/health

#Other causes to rule out

#Prevent it next time

• Document the timeout at each hop, and keep them in order: backend keep-alive > listener idle > backend read expectation.
• Alarm on 502 and 504 rates, not only on backend health.
• Move long reports to an async pattern. See Events and Functions automation.
• Related: flexible shapes and NLB and choosing an ingress service.