Sev1: someone regenerated the Azure OpenAI keys and every app gets 401 — restore service and move to Entra ID

#Symptoms

• Every app calling one Azure OpenAI resource fails with 401 Access denied due to invalid subscription key or wrong API endpoint.
• Started at one moment; the activity log shows Regenerate Key.

#1. Confirm what happened

az monitor activity-log list --resource-id <aoai-resource-id> --offset 24h \
  --query "[?contains(operationName.value,'regenerateKey')].{t:eventTimestamp, who:caller}" -o table

The event properties show whether key1 or key2 was regenerated. Apps using the other key are unaffected.

#2. Get the new key into apps

NEWKEY=$(az cognitiveservices account keys list -g rg-ai -n aoai-prod --query key1 -o tsv)
az keyvault secret set --vault-name kv-ai-prod --name aoai-key --value "$NEWKEY" -o none

Then make apps pick it up:

• App Service / Functions with Key Vault references: references refresh periodically; restart to force it.

az webapp restart -g rg-apps -n app-chat

• AKS with the Secrets Store CSI driver: secrets sync on rotation poll interval; restart deployments to be sure.

kubectl rollout restart deployment/chat-api -n ai

• Gateways (APIM named values from Key Vault): refresh the named value or wait for its refresh interval.

Caution

Don't regenerate the other key "to be safe" during the incident. Apps that still work on it would break too. Plan a proper rotation afterwards.

#3. Find apps you didn't know about

Keys get copied into scripts, notebooks and partner systems. Watch for remaining 401s by caller IP or user agent:

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where TimeGenerated > ago(1h)
| where ResultSignature == "401"
| summarize count() by CallerIPAddress, bin(TimeGenerated, 5m)

(Column names vary between the AzureDiagnostics and resource-specific schemas.)

#4. Move to keyless (Entra ID) auth

With Entra ID, there is no shared key to rotate or leak. Give each app's managed identity the data-plane role:

az role assignment create --assignee-object-id <app-mi-principal-id> --assignee-principal-type ServicePrincipal \
  --role "Cognitive Services OpenAI User" --scope <aoai-resource-id>

Client code (Python):

from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import AzureOpenAI

token_provider = get_bearer_token_provider(DefaultAzureCredential(),
                                           "https://cognitiveservices.azure.com/.default")
client = AzureOpenAI(azure_endpoint="https://aoai-prod.openai.azure.com/",
                     azure_ad_token_provider=token_provider, api_version="2024-10-21")

When every app uses Entra ID, disable key auth on the resource:

az resource update --ids <aoai-resource-id> --set properties.disableLocalAuth=true

#5. Proper rotation, if you keep keys

Use the two-key pattern:

1. Apps use key1. Regenerate key2.
2. Update the secret to key2, roll apps, confirm no traffic uses key1.
3. Regenerate key1.

Automate it, and never regenerate the key currently in use.

#Prevent it next time

• Keyless auth with managed identities; disableLocalAuth on production resources.
• Restrict who can call listKeys/regenerateKey (custom roles without Microsoft.CognitiveServices/accounts/regenerateKey/action).
• Alert on regenerateKey events in production.
• Keep an inventory of apps per AI resource.