Home / Azure

AzureRunbookAdvanced

Sev1: Key Vault or secrets deleted, apps failing — recover from soft delete

Recover a deleted Azure Key Vault, secret, key or certificate from soft delete, find who deleted it, and turn on purge protection so the next mistake is a five-minute fix.

Deepak Bhardwaj3 min read

Applies toAzure Key Vault (vault access policy and Azure RBAC models), Azure CLI 2.50+

Symptoms

  • Applications start failing with SecretNotFound, VaultNotFound or DNS errors for <vault>.vault.azure.net.
  • App Service Key Vault references show a red status; AKS pods using the CSI secret store driver fail to mount.
  • Someone ran a cleanup script or a Terraform destroy against the wrong workspace.

1. Confirm what is gone

# Whole vault deleted?
az keyvault list-deleted --resource-type vault -o table

# Vault still exists but individual objects deleted?
az keyvault secret list-deleted --vault-name kv-prod-app -o table
az keyvault key list-deleted --vault-name kv-prod-app -o table
az keyvault certificate list-deleted --vault-name kv-prod-app -o table
What you see Meaning Action
Vault in list-deleted Vault soft-deleted, recoverable until scheduledPurgeDate Step 2
Secret in secret list-deleted Object soft-deleted Step 3
Nothing in either list Purged, or soft delete was off on a very old vault Step 5

2. Recover the vault

az keyvault recover --name kv-prod-app --location westeurope

The vault comes back with its objects, access policies and network rules. Private endpoints are a separate resource: if they were deleted with the resource group, re-create them and check the private DNS A record.

Warning

You cannot create a new vault with the same name while the old one is soft-deleted. Do not "fix" the outage by pointing apps at a new vault name — recover the original instead.

3. Recover individual objects

az keyvault secret recover --vault-name kv-prod-app --name sql-conn-string
az keyvault key recover --vault-name kv-prod-app --name cmk-storage
az keyvault certificate recover --vault-name kv-prod-app --name wildcard-contoso

Recovery is asynchronous. Poll az keyvault secret show until it returns the secret before you restart apps.

4. Restart consumers and verify

az webapp restart -g rg-app -n app-prod
kubectl rollout restart deploy/api -n prod     # pods using the CSI driver

App Service caches Key Vault references; a restart forces a refresh. For customer-managed keys (Storage, SQL TDE), check the resource status — some services take a few minutes to detect the key again.

5. If the object was purged

  • Restore from a backup blob if you took one with az keyvault secret backup / az keyvault key backup.
  • Otherwise re-issue: rotate the database password, re-import the certificate, generate a new key.
  • For a purged customer-managed key, data encrypted with it is not recoverable. This is why purge protection matters.

6. Find out who did it

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where OperationName in ("SecretDelete", "VaultDelete", "KeyDelete", "CertificateDelete")
| project TimeGenerated, OperationName, id_s, CallerIPAddress, identity_claim_upn_s
| order by TimeGenerated desc

Column names depend on whether your diagnostic setting uses the Azure diagnostics or resource-specific mode. Vault deletion also appears in the Activity log:

az monitor activity-log list --offset 24h \
  --query "[?contains(operationName.value,'vaults/delete')].{when:eventTimestamp, who:caller}" -o table

Prevent it next time

az keyvault update --name kv-prod-app --enable-purge-protection true
az lock create --name no-delete --lock-type CanNotDelete --resource-group rg-security
  • Purge protection cannot be turned off once enabled — that is the point.
  • Put production vaults in a resource group with a CanNotDelete lock.
  • Back up secrets and certificates you cannot regenerate.
  • Alert on SecretDelete and VaultDelete events.

Related fixes