Lessons from the 29 September 2026 Azure OpenAI incident in Sweden Central - multi-region failover with API Management and sensible retries

#What happened

Azure's post-incident review 7QL5-Z50 describes intermittent request failures and latency for Azure OpenAI, Foundry Agent Service, Foundry Models and Azure AI services in Sweden Central, from 10:03 to 15:58 UTC on 29 September 2026.

• A planned deployment of an internal service cleared its caches during peak hours.
• Two defects caused most lookups to bypass the cache, sending about 20 times normal traffic to the backend.
• Instances failed health checks that depended on database connections and restarted in a loop. Existing metrics didn't show the connection exhaustion, which slowed diagnosis.

Microsoft's recommendations to customers: review BCDR guidance for Foundry, implement retries with exponential backoff, follow the Well-Architected Framework, monitor at a granular level, and configure Service Health alerts.

An application with deployments in only one region had no way around it.

#1. Deploy the same model in a second region

• Pick a region where the same model and version is available with your deployment type. Data Zone or Global deployments can help, but check data residency requirements first.
• Keep the deployment names the same in both regions, so the gateway can switch without changing request paths.
• Size the second region for the traffic you'd send during failover (quota is per region and subscription).

#2. Put API Management in front

API Management backend pools can spread traffic and fail over, and backends can have a circuit breaker that stops sending traffic to a failing region for a while.

Outline (Bicep-style, check the current schema for your API version):

resource beSweden 'Microsoft.ApiManagement/service/backends@2024-05-01' = {
  parent: apim
  name: 'aoai-swedencentral'
  properties: {
    url: 'https://aoai-sc.openai.azure.com/openai'
    protocol: 'http'
    circuitBreaker: {
      rules: [ {
        name: 'aoai-breaker'
        failureCondition: {
          count: 5
          interval: 'PT1M'
          statusCodeRanges: [ { min: 500, max: 599 }, { min: 429, max: 429 } ]
        }
        tripDuration: 'PT1M'
        acceptRetryAfter: true
      } ]
    }
  }
}

resource pool 'Microsoft.ApiManagement/service/backends@2024-05-01' = {
  parent: apim
  name: 'aoai-pool'
  properties: {
    type: 'Pool'
    pool: {
      services: [
        { id: beSweden.id, priority: 1 }
        { id: beFrance.id, priority: 2 }
      ]
    }
  }
}

Priority 1 takes traffic normally; priority 2 takes it when priority 1's breaker is open. Authenticate APIM to both Azure OpenAI resources with its managed identity (Cognitive Services OpenAI User role), so there are no keys to rotate.

#3. Keep client retries bounded

During this incident, failures were intermittent. Naive retry loops make that worse by multiplying load.

from openai import AzureOpenAI
client = AzureOpenAI(azure_endpoint="https://apim-ai.azure-api.net",
                     api_version="2024-10-21",
                     azure_ad_token_provider=token_provider,
                     max_retries=2,       # SDK retries with backoff on 429/5xx/timeouts
                     timeout=60)

• Let one layer own retries: either the SDK or the gateway, not both multiplied together.
• Honour Retry-After.
• Cap total time per user request, then fail with a clear message.

#4. Monitor per backend

The review notes that missing metrics slowed Microsoft's own diagnosis. Don't repeat that in your layer:

• APIM: request count, failures and latency per backend (use APIM's Application Insights integration or the azure-openai-emit-token-metric policy with a backend dimension).
• Alert on error rate per region, not just total.
• Log the backend chosen for each request, so you can see failover happening.

#5. Test failover

In a test environment, break the primary on purpose (point it at a wrong URL, or block it with a network rule) and confirm traffic moves to the secondary within the breaker window, then moves back.

#Checklist

• Same model and deployment name in a second region
• APIM backend pool with priorities and circuit breakers
• Managed identity from APIM to both resources
• One retry layer, bounded, honouring Retry-After
• Per-backend error and latency metrics with alerts
• Service Health alert for Azure OpenAI Service
• Failover tested

#Sources

• Azure status history - PIR 7QL5-Z50 (Sweden Central)