Handling 429s and latency spikes from LLM APIs

#Why you're getting 429s

LLM endpoints enforce two limits at once:

• RPM: requests per minute.
• TPM: tokens per minute. The provider counts input + max output tokens when it admits the request, not the tokens actually generated.

That second point catches most teams. A request with a 6,000-token prompt and max_tokens: 4000 reserves about 10k TPM even if the reply is 200 tokens. Set max_tokens to what you actually need. It's the cheapest throughput fix there is.

Limits are often evaluated over short windows (for example, per 10 seconds), so bursts get throttled even when the per-minute average looks fine.

#Read the headers

Most providers return the remaining budget and a wait hint:

Log these on every call. They're the most useful signal for capacity planning.

#Retry properly

import random, time
import httpx

RETRYABLE = {429, 500, 502, 503, 504}

def call_with_retry(send, max_attempts=6, base=1.0, cap=30.0):
    """send() performs one HTTP call and returns an httpx.Response."""
    for attempt in range(1, max_attempts + 1):
        resp = send()
        if resp.status_code not in RETRYABLE:
            return resp
        if attempt == max_attempts:
            resp.raise_for_status()
        wait = resp.headers.get("retry-after-ms")
        wait = float(wait) / 1000 if wait else float(resp.headers.get("retry-after", 0) or 0)
        if not wait:
            wait = min(cap, base * 2 ** (attempt - 1))
        time.sleep(wait + random.uniform(0, wait * 0.25))   # jitter avoids synchronized retries

Most official SDKs already do this. In the OpenAI and Anthropic Python SDKs it's controlled by max_retries, so set that instead of wrapping one retry loop around another.

Caution

Don't retry 400 (bad request, content filter) or 401/403. Retrying them only adds latency and noise.

#Spread the load

1. Several deployments or regions behind a gateway. Azure API Management, an AWS API Gateway with Lambda, or LiteLLM can round-robin across deployments and fail over on 429.
2. Separate quota per workload. A batch summarization job shouldn't use up the quota of your customer-facing chat.
3. Queue batch work. Use the provider's batch APIs (usually about 50% cheaper, with results within 24 hours) for anything that doesn't need an immediate answer.

#When to buy provisioned throughput

Provisioned throughput (Azure PTU, Bedrock Provisioned Throughput, OCI dedicated AI clusters) gives you predictable latency and no noisy neighbors. Consider it when:

• Sustained traffic stays high on most business days.
• p95 latency matters more than cost per token.
• You're regularly hitting the maximum pay-as-you-go quota for the region.

Keep a pay-as-you-go deployment as spillover for peaks above the provisioned capacity.

#Latency checklist

• Stream responses (stream: true) so users see the first token in about a second.
• Cache repeated system prompts. Prompt caching cuts time-to-first-token and input cost.
• Use a smaller or faster model for classification and routing, and send only the hard requests to the largest model.
• Alert on p95 time-to-first-token and 429 rate per deployment, not only on errors.