Azure Databricks managed resource group errors - deny assignments, Azure Policy and locks that break clusters

#Symptom

One of these shows up, often right after a governance change on the subscription:

• An admin with Owner tries to delete a VM or disk, tag a resource, change the storage account firewall or add a role assignment in the databricks-rg-* resource group. It fails with denied by deny assignment and the name System deny assignment created by Azure Databricks.
• Clusters stop starting. The termination reason is a cloud provider error that contains RequestDisallowedByPolicy.
• Clusters show as terminated in Databricks but their VMs stay in Azure. Or termination errors show ScopeLocked.
• Remediation tasks for a DeployIfNotExists policy (agents, diagnostics, backup) fail for every VM in the managed resource group.

All of these come from the same design. Azure Databricks owns the managed resource group and protects it with a deny assignment. The only principal that can change resources there is the Azure Databricks service, which is in the assignment's exclusion list. Your Owner role doesn't override that. Policies and locks still apply to the group, though, so they can block Databricks itself.

#Find the managed resource group

RG=rg-data-prod
WS=dbw-data-prod

MRG_ID=$(az databricks workspace show -g $RG -n $WS --query managedResourceGroupId -o tsv)
MRG=${MRG_ID##*/}
echo $MRG_ID

#Quick map

#Checks

#1. Confirm it is the Databricks deny assignment

az rest --method get \
  --url "https://management.azure.com${MRG_ID}/providers/Microsoft.Authorization/denyAssignments?api-version=2022-04-01" \
  --query "value[].{name:properties.denyAssignmentName, actions:properties.permissions[0].actions, excludedPrincipals:properties.excludePrincipals[].id}" -o json

If the name is System deny assignment created by Azure Databricks, you're dealing with the expected protection, not a missing role. You can't edit or remove it, and assigning yourself more roles won't help.

#2. Find the failed operation and who ran it

az monitor activity-log list --resource-group $MRG --offset 1d --status Failed \
  --query "[].{time:eventTimestamp, op:operationName.value, caller:caller, sub:subStatus.value}" -o table

If the caller is the Azure Databricks service and the operation is Microsoft.Compute/virtualMachines/write or Microsoft.Network/networkInterfaces/write, a policy or lock is blocking Databricks itself.

#3. Find the policy behind RequestDisallowedByPolicy

az policy state list --resource-group $MRG \
  --filter "complianceState eq 'NonCompliant'" \
  --query "[].{policy:policyDefinitionName, assignment:policyAssignmentName, effect:policyDefinitionAction, resource:resourceType}" -o table

The full error in the Databricks event log also names the policy assignment. Open the cluster, go to Event log, and expand the TERMINATING event.

databricks clusters events --json '{"cluster_id": "0612-101530-abcd123", "event_types": ["TERMINATING"], "limit": 5}' -o json

Policies that commonly cause this:

• Require a tag on resources, with the deny effect.
• Allowed VM SKUs or allowed locations.
• Deny public IP addresses, on workspaces without secure cluster connectivity.
• Require disk encryption sets or specific managed disk settings.

#4. Look for inherited locks

az lock list --resource-group $MRG -o table
az lock list --query "[?!contains(id, '/resourceGroups/')]" -o table   # subscription-level locks

A CanNotDelete lock at the subscription scope applies to the managed resource group as well. Databricks then can't delete the VMs and disks of terminated clusters, and you keep paying for them.

#Fixes

#Change Databricks resources the supported way

# Workspace tags are applied to the managed resource group and the resources Databricks creates
# (propagation details vary by resource type; check Cost Management after a cluster restart)
az resource tag --ids $(az databricks workspace show -g $RG -n $WS --query id -o tsv) \
  --tags costcenter=cc-1234 env=prod --is-incremental

#Exempt the managed resource group from a blocking policy

Take this route when the policy's goal is already met another way (tags through cluster policies, for example) or doesn't apply to ephemeral cluster VMs.

az policy exemption create \
  --name exempt-databricks-mrg-require-tags \
  --display-name "Databricks managed RG - tags enforced by cluster policy" \
  --policy-assignment "/subscriptions/<sub-id>/providers/Microsoft.Authorization/policyAssignments/require-tags" \
  --exemption-category Mitigated \
  --scope "$MRG_ID" \
  --expires-on 2027-11-10T00:00:00Z \
  --description "Tags applied through workspace tags and compute policies"

Warning

An exemption on the policy assignment covers every policy in an initiative unless you limit it. Use --policy-definition-reference-ids to exempt only the definition that blocks Databricks.

Better long term: give the policy an exclusion through notScopes for databricks-rg-* groups. Or change the policy to modify so it adds the tag instead of denying the request.

#Remove or re-scope the lock

Caution

Removing a subscription-level lock takes away delete protection from every resource group in the subscription. Before you delete it, add CanNotDelete locks to the resource groups that need protection, but not to databricks-rg-*.

az lock create --name protect-data-prod --lock-type CanNotDelete --resource-group rg-data-prod
az lock delete --name sub-no-delete --subscription <sub-id>

Then restart one cluster and confirm the leftover VMs disappear:

az vm list -g $MRG --query "[].{name:name, state:provisioningState}" -o table

#Deploy agents another way

Policy-driven VM extensions don't work in the managed resource group, and cluster VMs only live a few hours anyway. Exempt the group from those DeployIfNotExists assignments. For security monitoring, use Databricks audit logs, system tables and diagnostic settings on the workspace resource. If you really need a host agent, install it with a cluster-scoped init script, and test it on a non-production cluster first.

#Prevent it next time

• Put an exclusion for databricks-rg-* (or notScopes per workspace) in your policy baseline before you roll out deny policies to a subscription with Databricks.
• Run az policy state list on each managed resource group after every policy assignment change, then start one test cluster.
• Don't put delete locks at the subscription level where Databricks workspaces run. Lock individual resource groups instead.
• Enforce tags, VM sizes and spot use in Databricks compute policies. Don't try to enforce them on the VMs Databricks creates.
• Add a check to on-call notes: RequestDisallowedByPolicy in a cluster termination reason means a policy change, not a Databricks outage.