Runbook: model endpoint regional outage (Azure OpenAI, Bedrock, OCI GenAI)

#Symptoms

• A rise in 5xx errors, timeouts or 429s from one model backend.
• Latency (time to first token) far above normal.
• Provider status page or Service Health showing an incident in the region.

#1. Confirm scope (5 minutes)

• Which backends fail: one deployment, one model, one region, or everything?
• Check the provider's status: Azure Service Health / Resource Health, the AWS Health Dashboard, the OCI status page.
• Check it isn't you: expired credentials (401/403), quota exhausted by a runaway job (429 from your own traffic), a recent deployment or network change.

# quick probe through the gateway, bypassing caches
curl -s -o /dev/null -w "%{http_code} %{time_total}\n" -X POST "$GW/openai/deployments/chat-main/chat/completions?api-version=2024-10-21" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"ping"}],"max_tokens":5}'

#2. Shift traffic (if not already automatic)

If backend pools and circuit breakers are set up, traffic should already be moving. Check the per-backend request distribution. If not:

• Remove or deprioritise the failing region's backends in the gateway (APIM backend pool priority, LiteLLM router config).
• Bedrock: switch the model ID to a cross-region inference profile, or to the DR region client.
• Self-hosted fallback: scale it up if it's part of the plan.

#3. Check DR capacity

• DR deployment quota (TPM) versus the current production load. Expect 429s in DR if it was sized only for spillover.
• Throttle non-critical traffic: pause batch jobs, ingestion and evaluations; lower per-team limits in the gateway to protect user-facing apps.

#4. Check quality

• Same model and version in DR? If not, run a quick golden-set evaluation (10–20 queries).
• Content filter policy the same in DR.

#5. Communicate

• Status update to users: "The assistant may be slower than usual." Mention if a backup model is in use.
• Incident channel updates every 30 minutes.

#6. Fail back

• Provider reports the incident resolved, and probes are healthy for 30+ minutes.
• Restore the original gateway priorities, gradually (a canary first).
• Resume batch and ingestion jobs. Watch quota.

#7. Post-incident review

• Time to detect, time to shift traffic, and errors seen by users.
• Was the DR quota enough? Were circuit breakers tuned right?
• Update this runbook and the gateway configuration.

Tip

A synthetic probe every minute, per backend, through the gateway (with a tiny max_tokens) gives you a clear signal on which backend is failing, and costs almost nothing.