Sev1: the LLM gateway is down and every AI app fails at once — APIM or LiteLLM triage and a safe bypass

#Symptoms

• Every app that uses LLMs fails at the same time with 5xx, timeouts, or 401/403.
• Direct calls to the model endpoint (from an admin host) still work.

#1. Locate the failure

# Through the gateway
curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
  https://apim-ai.azure-api.net/openai/deployments/gpt-chat/chat/completions?api-version=2024-10-21 \
  -H "api-key: $APIM_KEY" -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"ping"}],"max_tokens":5}'

# Directly to the backend (admin test only)
curl -s -o /dev/null -w "%{http_code}\n" \
  "https://aoai-prod.openai.azure.com/openai/deployments/gpt-chat/chat/completions?api-version=2024-10-21" \
  -H "api-key: $AOAI_KEY" -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"ping"}],"max_tokens":5}'

The api-version must be one your backend supports.

#2. Azure API Management checks

az apim show -g rg-ai -n apim-ai --query "{state:provisioningState, sku:sku.name, capacity:sku.capacity}"

ApiManagementGatewayLogs
| where TimeGenerated > ago(30m)
| summarize count() by ResponseCode, BackendResponseCode, LastErrorReason
| order by count_ desc

Common causes:

• A policy change: a typo in a policy expression throws on every request. Check the API or product policy's recent revision and roll back.
• Managed identity to backend lost its role (Cognitive Services OpenAI User): backend returns 401 to APIM.
• Named value from Key Vault failed to refresh (vault firewall, deleted secret).
• Capacity: gateway CPU saturated; scale units up.
• Custom domain certificate expired on the gateway hostname.

#3. LiteLLM or self-hosted gateway checks

curl -s http://litellm.ai.svc:4000/health/liveliness
curl -s http://litellm.ai.svc:4000/health/readiness
kubectl get pods -n ai -l app=litellm
kubectl logs -n ai deploy/litellm --tail=100

Typical failures: its database (used for keys and spend tracking) is down, a bad config reload, expired provider keys in its config, or pods OOMKilled under load.

#4. Restore

• Roll back the last gateway change (policy revision, config map, image tag).
• Scale out the gateway if it is overloaded.
• Fix backend auth (restore role assignment, rotate key in the gateway's secret store).

#5. Controlled bypass, only if needed

If the gateway can't be fixed quickly and the business needs AI features back, you can point critical apps directly at the model endpoints.

Caution

Bypassing the gateway also bypasses what it enforces: content safety policies, token limits, logging for audit, per-team quotas and PII redaction. Get explicit approval from the service owner and security, limit it to named critical apps, and set an end time.

Use managed identity or a dedicated key for the bypass, not the gateway's master keys, so you can revoke it cleanly afterwards.

#6. After recovery

• Move apps back to the gateway and revoke bypass credentials.
• Check usage logs for the bypass window to fill gaps in chargeback and audit.

#Prevent it next time

• Run the gateway with redundancy (multiple units/zones, multiple LiteLLM replicas with a PDB).
• Use policy revisions and deploy gateway config through CI with a smoke test call.
• Synthetic probe through the gateway every minute, separate from backend health.
• Document the bypass decision and procedure before you need it.