LLM API calls hang for minutes: client timeouts, retries and connection limits in the OpenAI SDK and boto3

#Symptom

A chat endpoint or batch worker that calls an LLM API stops responding. You don't see errors, only requests that never seem to finish. Then one or more of these turn up:

• Request latency p99 jumps to several minutes while p50 looks normal.
• Web workers or thread pools are exhausted and health checks start failing.
• Logs show APITimeoutError or ReadTimeoutError long after the user gave up.
• The provider dashboard shows far fewer requests than your app thinks it sent.

The usual cause is client defaults. The OpenAI Python SDK waits up to 10 minutes per request and retries some failures twice, so in the worst case one call can block a worker for more than 30 minutes. botocore defaults to a 60-second read timeout with its own retries. Behind a slow proxy, or when the provider is degraded, those defaults keep workers busy for much longer than you expect.

#Quick checks

#1. Find out what the client is actually using

from openai import OpenAI
client = OpenAI()
print(client.timeout, client.max_retries)

import boto3
brt = boto3.client("bedrock-runtime")
cfg = brt.meta.config
print(cfg.connect_timeout, cfg.read_timeout, cfg.retries, cfg.max_pool_connections)

If you see 600 (or a Timeout(timeout=600.0, ...) object) and 2, nobody has set a timeout yet.

#2. Separate network delay from model delay

Time the TCP/TLS connection and the first byte separately:

curl -s -o /dev/null -w "connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n" \
  -H "api-key: $AZURE_OPENAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"ping"}],"max_tokens":5}' \
  "https://<resource>.openai.azure.com/openai/deployments/<deployment>/chat/completions?api-version=<api-version>"

A slow connect or tls points to DNS, a proxy or a private endpoint. A fast connect with a slow ttfb means the model is slow to start answering, which usually comes from a long prompt, a high max_tokens, a reasoning model or a busy deployment.

#3. Check whether retries are hiding the real error

Turn on SDK logging for a single run:

export OPENAI_LOG=debug        # openai SDK request/retry logging
python your_script.py 2>&1 | grep -iE "retry|timeout|status"

For boto3, boto3.set_stream_logger("botocore", logging.DEBUG) prints each attempt. Several attempts for one logical call mean the user-facing latency is the sum of all of them.

#Cause → fix

#Fix: the OpenAI and Azure OpenAI clients

Set a connect timeout and a read timeout separately. Keep connect short and base read on your longest normal response.

import httpx
from openai import OpenAI, AzureOpenAI, APITimeoutError

timeout = httpx.Timeout(60.0, connect=5.0)   # 60 s for read/write/pool, 5 s connect
limits = httpx.Limits(max_connections=100, max_keepalive_connections=20)

client = OpenAI(
    timeout=timeout,
    max_retries=1,
    http_client=httpx.Client(limits=limits, timeout=timeout),
)

azure = AzureOpenAI(
    azure_endpoint="https://<resource>.openai.azure.com",
    api_version="<api-version>",
    timeout=timeout,
    max_retries=1,
)

Override the timeout for a single route without building a new client:

try:
    resp = client.with_options(timeout=15.0, max_retries=0).chat.completions.create(
        model="<model-or-deployment>",
        messages=[{"role": "user", "content": "Summarise this alert in one line: ..."}],
        max_tokens=100,
    )
except APITimeoutError:
    # fall back, return a degraded answer, or queue the work
    ...

Tip

Make one client per process and reuse it. If you create a new client for each request, you lose connection reuse, and every call pays the full TCP and TLS handshake again.

#Streaming changes what "timeout" means

With stream=True, the read timeout applies to the gap between chunks, not to the whole response. A 30-second read timeout can let a long answer stream for two minutes, as long as tokens keep arriving. That's usually what you want for chat. If you also need a hard limit on total time, add your own deadline:

import time
deadline = time.monotonic() + 90
stream = client.chat.completions.create(model="<model>", messages=msgs, stream=True)
for chunk in stream:
    if time.monotonic() > deadline:
        stream.close()
        break
    ...

#Fix: boto3 and Amazon Bedrock Runtime

import boto3
from botocore.config import Config

cfg = Config(
    connect_timeout=5,
    read_timeout=120,
    retries={"max_attempts": 3, "mode": "standard"},   # max_attempts includes the first call
    max_pool_connections=50,
)
brt = boto3.client("bedrock-runtime", region_name="us-east-1", config=cfg)

resp = brt.converse(
    modelId="<model-id-or-inference-profile>",
    messages=[{"role": "user", "content": [{"text": "ping"}]}],
    inferenceConfig={"maxTokens": 50},
)

max_pool_connections defaults to 10. If you share one client across 50 threads, 40 of them wait for a connection before they even send a request. That wait looks like model latency in your traces.

Warning

Don't fix throttling by raising max_attempts. Each retry holds a worker and adds to the load on a quota that's already full. Cap retries, return a clear "busy" response or queue the request, and fix the quota separately.

#Choosing the numbers

Base timeouts on what you measure, not on guesses:

1. Pull p99 latency per route from your traces or gateway logs for the last week.
2. Set the read timeout to about 1.5 to 2 times that p99 for non-streaming calls.
3. Set connect to 3–5 seconds everywhere.
4. Keep the total (timeout × attempts) under the timeout of whatever is upstream: the load balancer, API gateway or the user's patience. If the gateway gives up at 60 seconds, a 60-second client timeout with two retries does nothing useful.

#Prevent it next time

• Put timeouts and retry settings in config, not code defaults, and log them at startup.
• Emit a metric for client-side timeouts that's separate from HTTP errors, and alert when it rises.
• Trace time spent waiting for a pooled connection apart from time spent waiting for the model.
• Keep the gateway or load balancer timeout longer than the client's total time, so the client always fails first with an error it can handle.
• Test with a fault-injection proxy (or a mock endpoint that sleeps) to check that workers come back when the provider stalls.