Home / AI & GenAI

AI & GenAIRunbookIntermediate

LLM provider outage playbook: multi-provider readiness, degraded modes and decision points

Prepare for the day a whole LLM provider is unavailable, not just one region. Covers multi-provider readiness work, contract and data checks, degraded service modes, a decision tree during the incident, and returning to normal.

Deepak Bhardwaj3 min read

Applies toOrganisations depending on one or more of Azure OpenAI, OpenAI, Anthropic, Amazon Bedrock, Google Vertex AI, OCI Generative AI, or self-hosted open-weight models

1. Why this is different from a regional outage

A region-level problem is solved with another region from the same provider (see the regional outage runbook). A provider-level outage (global API issue, an account or subscription suspended, a contract or policy change) needs a different provider or self-hosted model.

2. Readiness work (do it before you need it)

  • Second provider approved: security review, data processing terms, data residency, in the approved model list.
  • Accounts, quota and keys set up in the gateway for the second provider. Test them monthly.
  • Prompt variants for the alternate model family, tested with the golden evaluation set and passing your quality bar.
  • Tool and structured-output compatibility checked (function calling, JSON schema).
  • Embedding independence: if the provider's embedding model goes away, your existing index can't be queried with a different embedding model. Options: self-host the embedding model you use, keep a second index with an alternate embedding model for critical corpora, or accept keyword-only search in degraded mode.
  • A self-hosted open-weight model for the most critical use case, if the business needs it (accepting lower quality).

3. Degraded service modes

Agree these with business owners up front:

Mode What users get
Normal Primary provider
Alternate model Second provider; quality may differ slightly; banner shown
Reduced features Search and retrieval only (show source passages without a generated answer); no agents
Read-only / offline Cached FAQ answers; "assistant unavailable" for everything else

4. Decision points during the incident

Provider errors > 20% for 10 min?
 ├─ No  → monitor, regional runbook if one region
 └─ Yes → Is the provider status page reporting a global incident?
          ├─ Yes → switch to the alternate model mode (gateway route change)
          └─ No  → check your own causes (keys, quota, billing, account status)
                   └─ Account or subscription problem → escalate to the vendor account team, switch mode
Alternate provider struggling too (quota)?
 └─ Throttle non-critical apps; move to reduced features for lower tiers

The incident commander decides on the mode, and records the time.

5. Things that go wrong in practice

  • The alternate provider's quota is too low for full production traffic. Pre-agree quota increases, or prioritise apps.
  • Agents with tool calls behave differently on the alternate model. Disable autonomous actions in alternate mode, or require approval.
  • Costs change. Track spend during the incident.

6. Returning to normal

  • Wait for the primary provider to be stable (30–60 minutes of healthy probes).
  • Switch back in stages (canary, then all traffic).
  • Review: time in each mode, user impact, quality scores, cost.

7. Keep it alive

  • A quarterly "game day": run one app on the alternate provider for an hour in production (or in staging with real prompts).
  • Re-run alternate-model evaluations whenever prompts change.

Tip

The biggest blocker in real provider outages is not code, it's approval: legal and security haven't cleared the second provider. Get that done now, while nothing is on fire.

Related fixes