Home / AI & GenAI

AI & GenAIHow-toIntermediate

LLM fallback models and circuit breakers in application code

Keep GenAI features working when a model or provider has problems. Covers a fallback chain of models, circuit breakers, timeouts per stage, which errors to retry, keeping prompts compatible across models, and graceful degradation for users.

Deepak Bhardwaj3 min read

Applies toPython or TypeScript apps calling Azure OpenAI, OpenAI, Anthropic, Bedrock, Vertex AI, OCI GenAI or self-hosted endpoints; libraries such as LiteLLM, tenacity, pybreaker or the gateway's built-in features

1. Fallback chain

Order Target When used
1 Primary model, primary region Normally
2 Same model, another region or deployment Primary throttled or failing
3 Different model or provider (comparable quality) Model or provider outage
4 Smaller or cheaper model, or a cached answer Severe incidents; tell users quality may be lower
5 Graceful message Nothing available

Steps 1–2 belong in the gateway (see the backend pools post). Steps 3–5 often need application logic, because prompts, tools and output formats can differ.

2. Which errors to retry or fall back on

Error Action
429 (rate limit) Retry with backoff, honouring Retry-After; fall back if it persists
500, 502, 503, 504, timeouts, connection resets Retry once, then fall back
400 (bad request, context too long) Don't retry the same request; fix it (truncate) or fall back to a model with a larger context
Content filter blocked Don't fall back to get around the safety filter; return a safe message
401 / 403 Don't retry; alert (configuration or credentials problem)

3. A circuit breaker plus fallback in Python

import time, random
from openai import AzureOpenAI, APITimeoutError, APIConnectionError, RateLimitError, InternalServerError

class Breaker:
    def __init__(self, fails=5, reset_after=30):
        self.fails, self.reset_after, self.count, self.opened = fails, reset_after, 0, 0.0
    def available(self):
        return self.count < self.fails or time.time() - self.opened > self.reset_after
    def success(self): self.count = 0
    def failure(self):
        self.count += 1
        if self.count >= self.fails: self.opened = time.time()

TARGETS = [  # (name, client, deployment)
    ("primary",  AzureOpenAI(azure_endpoint="https://gw.corp.local/aoai", api_version="2024-10-21", api_key="..."), "chat-main"),
    ("fallback", AzureOpenAI(azure_endpoint="https://gw.corp.local/aoai-alt", api_version="2024-10-21", api_key="..."), "chat-alt"),
]
BREAKERS = {name: Breaker() for name, _, _ in TARGETS}
RETRYABLE = (APITimeoutError, APIConnectionError, RateLimitError, InternalServerError)

def chat(messages, timeout=30):
    for name, client, deployment in TARGETS:
        if not BREAKERS[name].available():
            continue
        for attempt in range(2):
            try:
                r = client.with_options(timeout=timeout, max_retries=0).chat.completions.create(
                    model=deployment, messages=messages)
                BREAKERS[name].success()
                return r.choices[0].message.content, name
            except RETRYABLE:
                BREAKERS[name].failure()
                time.sleep((2 ** attempt) + random.random())
    return "The assistant is temporarily unavailable. Please try again in a few minutes.", "none"

In production, keep breaker state shared (Redis) if you run many instances, or rely on the gateway's breakers and keep only the model-level fallback in the app. Use API keys only through the gateway, or managed identity.

4. Keeping prompts compatible

  • Keep a prompt variant per model family in your prompt repository, and test each one with the golden evaluation set.
  • Use structured output with a JSON schema, and validate it. Fallback models may format differently.
  • Check tool and function-calling support and limits for each fallback model.
  • Watch context window differences. Truncate retrieved context to fit the smallest model in the chain.

5. Timeouts

  • Time to first token: 10–20 s before giving up on a streaming call.
  • Total: limit the overall budget (for example 60 s), so a chain of fallbacks doesn't make the user wait 5 minutes.

6. Tell users, and record it

  • Show a subtle notice when a fallback model answered ("Answered with a backup model").
  • Log which target served each request, and alert when the fallback rate rises above normal.

Tip

Test fallbacks on purpose: a weekly synthetic test that forces the primary to fail (a header or flag in non-production) catches broken fallback configuration before a real outage does.

Related fixes