Home / Azure

AzureRunbookAdvanced

Sev1: production resource group deleted — what can be recovered, and how fast

A realistic recovery plan after a resource group deletion. What Azure can bring back per resource type, the order to rebuild in, how to find who did it, and the locks and policies that stop it happening again.

Deepak Bhardwaj3 min read

Applies toAzure Resource Manager, Azure Backup, Key Vault, Storage, Azure SQL, Azure CLI 2.50+

First ten minutes

  1. Declare the incident and freeze changes in the subscription.
  2. Check whether the deletion is still running: az group show -n rg-prod-app --query properties.provisioningState returns Deleting while it is in progress. It cannot be cancelled.
  3. Find who and what:
az monitor activity-log list --offset 6h \
  --query "[?operationName.value=='Microsoft.Resources/subscriptions/resourceGroups/delete'].{when:eventTimestamp, who:caller, rg:resourceGroupName, status:status.value}" -o table

If the caller is a pipeline identity, disable that pipeline now.

What can come back

Resource Recovery path Notes
Key Vault az keyvault recover Do this first; other resources depend on keys
Storage account Portal → Restore deleted account Time-limited; RG must exist (re-create it)
Azure SQL database az sql db restore --deleted-time Needs the logical server; deleted servers have limited options
VMs Azure Backup restore (vault in another RG) Vault deleted too? Soft-deleted backup items may survive
App Service, Functions Redeploy from IaC and pipeline Custom domains and certs need re-binding
VNets, NSGs, LBs, public IPs Redeploy from IaC Static public IPs are gone; DNS must change
Recovery Services vault Soft delete keeps backup data for a period Check az backup vault list

Important

Re-create the resource group with the same name and region before restoring storage accounts or other resources that need their original group.

Rebuild order

  1. Resource group, then Key Vault recovery.
  2. Networking (VNet, subnets, NSGs, private DNS links) from IaC.
  3. Data: storage account restore, SQL restore, VM restore from backup.
  4. Compute and apps from pipelines.
  5. DNS and certificates; reissue anything tied to a deleted public IP.
  6. Monitoring, diagnostic settings, alerts.
az group create -n rg-prod-app -l westeurope --tags owner=platform
az keyvault list-deleted --resource-type vault -o table
az deployment group create -g rg-prod-app -f main.bicep -p @prod.parameters.json

No IaC?

Look for a previous deployment template in another subscription or an exported template, and check Azure Resource Graph history in Change Analysis. Without IaC, rebuild from documentation and expect a long RTO; record that as the lesson.

Prevent it next time

az lock create --name no-delete --lock-type CanNotDelete -g rg-prod-app
  • Lock every production resource group; remove locks only through a reviewed change.
  • Keep Recovery Services vaults and Key Vaults in a separate, locked resource group.
  • Don't give pipelines Owner or Contributor at subscription scope.
  • Use Azure Policy to deny resource group deletion outside a break-glass role if your process allows it.
  • Keep everything in IaC and test a full rebuild into an empty subscription.

Related fixes