Databricks compute policies - limiting VM sizes, enforcing auto-termination and tags without blocking users

#Why policies

Without policies, anyone with cluster creation rights can start a 50-node cluster of the largest VMs with no auto-termination and no cost-centre tag. Policies let you:

• Limit VM types and sizes.
• Cap workers and require autoscaling.
• Force auto-termination on interactive clusters.
• Pin runtime versions (LTS) and access mode.
• Set custom tags for cost allocation.

#Policy definition basics

A policy is JSON. Each attribute path has a rule type:

#Example: interactive analyst policy

{
  "spark_version": { "type": "unlimited", "defaultValue": "auto:latest-lts" },
  "node_type_id": { "type": "allowlist", "values": ["Standard_D4ds_v5", "Standard_D8ds_v5", "Standard_E8ds_v5"], "defaultValue": "Standard_D4ds_v5" },
  "driver_node_type_id": { "type": "allowlist", "values": ["Standard_D4ds_v5", "Standard_D8ds_v5", "Standard_E8ds_v5"] },
  "autoscale.min_workers": { "type": "range", "minValue": 1, "maxValue": 2, "defaultValue": 1 },
  "autoscale.max_workers": { "type": "range", "maxValue": 8, "defaultValue": 4 },
  "num_workers": { "type": "forbidden" },
  "autotermination_minutes": { "type": "range", "minValue": 10, "maxValue": 60, "defaultValue": 30 },
  "data_security_mode": { "type": "fixed", "value": "USER_ISOLATION" },
  "custom_tags.CostCenter": { "type": "regex", "pattern": "^CC[0-9]{4}$" },
  "custom_tags.Team": { "type": "unlimited" },
  "azure_attributes.availability": { "type": "fixed", "value": "ON_DEMAND_AZURE" }
}

• num_workers is forbidden, so the user must use autoscaling.
• USER_ISOLATION is the standard (shared) access mode with Unity Catalog. Use SINGLE_USER for dedicated compute.
• Users must enter a cost centre that matches the pattern.

#Example: job compute policy

{
  "cluster_type": { "type": "fixed", "value": "job" },
  "spark_version": { "type": "regex", "pattern": "^1[5-9]\\.[0-9]+\\.x-scala.*" },
  "node_type_id": { "type": "allowlist", "values": ["Standard_D8ds_v5", "Standard_D16ds_v5", "Standard_E16ds_v5"] },
  "autoscale.max_workers": { "type": "range", "maxValue": 20 },
  "azure_attributes.availability": { "type": "allowlist", "values": ["SPOT_WITH_FALLBACK_AZURE", "ON_DEMAND_AZURE"], "defaultValue": "SPOT_WITH_FALLBACK_AZURE" },
  "azure_attributes.first_on_demand": { "type": "fixed", "value": 1 },
  "custom_tags.CostCenter": { "type": "regex", "pattern": "^CC[0-9]{4}$" }
}

Adjust the runtime regex to the versions you support. Check the exact string format with databricks clusters spark-versions.

#Create and assign with the CLI

The definition field in the API request is a JSON string, not an object. Build the request with jq so the quoting is right:

jq -n --arg def "$(jq -c . analyst-policy.json)" '{name:"Analyst - interactive", definition:$def}' > req.json
databricks cluster-policies create --json @req.json

databricks cluster-policies list --output json | jq -r '.[] | [.policy_id, .name] | @tsv'

Grant a group permission to use it:

databricks cluster-policies set-permissions <policy-id> --json '{
  "access_control_list": [{"group_name": "analysts", "permission_level": "CAN_USE"}]
}'

#Remove unrestricted creation

Policies only work if users can't bypass them. In the workspace admin settings, remove Unrestricted cluster creation from the users group and grant it only to a small platform admin group. Users then see only the policies they can use.

Warning

Check existing clusters before you tighten access. Clusters created outside a policy keep running, and users may not be able to edit or restart them once the entitlement is removed. Tell users and give them time to move to a policy.

#Find clusters outside policies

databricks clusters list --output json | \
  jq -r '.[] | select(.policy_id == null) | [.cluster_id, .cluster_name, .creator_user_name] | @tsv'

#Policy families and serverless

• Databricks provides policy families (Personal Compute, Shared Compute, Job Compute) that you can base your policies on and override.
• Serverless compute isn't controlled by compute policies. Use budget policies (serverless usage policies) for tagging serverless usage, and workspace settings to control access.

#Good practice

• Two to four policies per workspace: personal, shared analytics, jobs, ML/GPU.
• Keep policies in Git and deploy them with Terraform or asset bundles.
• Review the allowed node types twice a year as new VM generations appear.

#Checklist

• Policies per persona with VM allowlists and worker caps
• Auto-termination enforced on interactive compute
• Access mode fixed for Unity Catalog
• Cost tags required
• Unrestricted cluster creation removed from users
• Clusters outside policies migrated