Sev1: apps get 401/403 after a managed identity or role assignment was removed — find it and restore access

#Symptoms

• App logs show AuthorizationPermissionMismatch, Forbidden, does not have authorization to perform action or ManagedIdentityCredential authentication unavailable.
• Several services break at once: storage, Key Vault, SQL, Service Bus.
• Recent change: Terraform apply, RBAC cleanup script, resource redeploy, or someone toggled the system-assigned identity.

#1. Which identity is the app using?

az webapp identity show -g rg-app -n app-prod
az functionapp identity show -g rg-app -n func-prod
az vm identity show -g rg-app -n vm-app01

Note the principalId. If the system-assigned identity was turned off and on again, it has a new principal ID. Every old role assignment pointed at the old one, which is why everything broke together.

#2. What roles does it have now?

PRINCIPAL=<principal-id>
az role assignment list --assignee $PRINCIPAL --all \
  --query "[].{role:roleDefinitionName, scope:scope}" -o table

An empty list for a production identity is your answer.

#3. Find what was removed

az monitor activity-log list --offset 48h \
  --query "[?operationName.value=='Microsoft.Authorization/roleAssignments/delete'].{time:eventTimestamp, who:caller, scope:resourceId, status:status.value}" \
  -o table

For a fuller picture across subscriptions, query the activity log in Log Analytics:

AzureActivity
| where TimeGenerated > ago(2d)
| where OperationNameValue has "roleAssignments" 
| project TimeGenerated, Caller, OperationNameValue, ActivityStatusValue, _ResourceId
| order by TimeGenerated desc

The properties of a delete event include the role definition ID and principal ID that were removed.

#4. Restore the assignments

az role assignment create --assignee-object-id $PRINCIPAL --assignee-principal-type ServicePrincipal \
  --role "Storage Blob Data Contributor" \
  --scope /subscriptions/<sub>/resourceGroups/rg-data/providers/Microsoft.Storage/storageAccounts/stprod

az role assignment create --assignee-object-id $PRINCIPAL --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope /subscriptions/<sub>/resourceGroups/rg-sec/providers/Microsoft.KeyVault/vaults/kv-prod

Using --assignee-object-id with the principal type avoids Graph lookup delays for new identities.

Tip

Role assignments can take several minutes to apply, and apps cache tokens. Restart the app after restoring roles instead of waiting for tokens to expire.

#5. Data-plane permissions that are not Azure RBAC

Some services keep their own permissions tied to the identity, and those are lost when the principal ID changes:

• Azure SQL: the database user created with CREATE USER [app-prod] FROM EXTERNAL PROVIDER is bound to the old object ID. Drop and recreate it:

DROP USER IF EXISTS [app-prod];
CREATE USER [app-prod] FROM EXTERNAL PROVIDER;
ALTER ROLE db_datareader ADD MEMBER [app-prod];
ALTER ROLE db_datawriter ADD MEMBER [app-prod];

• Key Vault with access policies (not RBAC): add a policy for the new object ID.
• Cosmos DB data plane RBAC: recreate the SQL role assignment for the new principal.

#6. Verify

az webapp restart -g rg-app -n app-prod
az webapp log tail -g rg-app -n app-prod

Watch for 403s to stop. Then check other apps that might share a user-assigned identity.

#Prevent it next time

• Prefer user-assigned identities for production: they survive app redeploys and keep their principal ID.
• Manage role assignments in the same IaC as the app, and protect them from cleanup scripts.
• Put a CanNotDelete lock on critical resource groups; it blocks role assignment deletion at that scope too.
• Alert on roleAssignments/delete in production subscriptions.