Highly available LLM endpoints: multi-region backend pools, priorities and circuit breakers in APIM

#1. Target design

APIM sends traffic to priority 1. When those backends are tripped by the circuit breaker (429s from a full PTU, or errors), traffic moves to priority 2, split by weight.

Note

Every deployment in the pool must use the same model and version (and the same deployment name, or rewrite the URL per backend), or answers will vary depending on which backend serves the request.

#2. Backends with circuit breakers (Bicep)

resource beCin 'Microsoft.ApiManagement/service/backends@2024-06-01-preview' = {
  name: '${apim.name}/aoai-ptu-cin'
  properties: {
    url: 'https://aoai-prod-cin.openai.azure.com/openai'
    protocol: 'http'
    circuitBreaker: {
      rules: [ {
        name: 'throttle-or-fail'
        failureCondition: {
          count: 1
          interval: 'PT10S'
          statusCodeRanges: [ { min: 429, max: 429 }, { min: 500, max: 503 } ]
        }
        tripDuration: 'PT10S'
        acceptRetryAfter: true
      } ]
    }
  }
}

resource pool 'Microsoft.ApiManagement/service/backends@2024-06-01-preview' = {
  name: '${apim.name}/aoai-pool'
  properties: {
    type: 'Pool'
    pool: {
      services: [
        { id: '/backends/aoai-ptu-cin', priority: 1 }
        { id: '/backends/aoai-paygo-cin', priority: 2, weight: 50 }
        { id: '/backends/aoai-paygo-sin', priority: 2, weight: 50 }
      ]
    }
  }
}

acceptRetryAfter: true keeps a backend out of the pool for as long as its Retry-After header asks.

#3. Policy: route and retry

<inbound>
  <base />
  <authentication-managed-identity resource="https://cognitiveservices.azure.com" />
  <set-backend-service backend-id="aoai-pool" />
</inbound>
<backend>
  <retry condition="@(context.Response.StatusCode == 429 || context.Response.StatusCode >= 500)"
         count="2" interval="1" first-fast-retry="true">
    <forward-request buffer-request-body="true" />
  </retry>
</backend>

buffer-request-body="true" is needed so the same request body can be sent again to another backend.

#4. Streaming considerations

• Retries only work before the response starts streaming. Once tokens are flowing, a failure ends the stream, and the client must retry.
• Set client and APIM timeouts long enough for long generations, and use stream: true for user-facing chat.

#5. Observability

• Log context.Request.Url / the selected backend in Application Insights so you can see where requests went.
• Track per-backend 429 and 5xx rates, PTU utilisation, and p95 latency per region.
• Alert when priority-2 traffic share rises above the normal level (the PTU is too small, or a region has an issue).

#6. Test failover

• Temporarily set a priority-1 backend URL to an invalid host in a test APIM, or block it with a network rule. Requests should move to priority 2 within seconds.
• Load test above the PTU capacity: the 429s should be absorbed by the spillover backends, and clients shouldn't see them.

#7. Making APIM itself highly available

• Premium (classic) with availability zones and optionally multi-region gateways, or Premium v2 with zone redundancy where available.
• Front Door in front of multi-region APIM gateways for global entry.

Tip

Also handle 429s in the client (exponential backoff with jitter, honouring Retry-After). The gateway reduces 429s a lot, but it can't remove them during a full-capacity event.