Home / Data Platform / Databricks & Spark

Data PlatformHow-toIntermediate

Databricks cluster policies: compute governance that cuts cost and support tickets

Design a small set of cluster policies per persona that enforce auto-termination, approved node types and runtimes, tags, spot usage and Unity Catalog access modes. Includes policy JSON, rollout steps and how to measure compliance.

Deepak Bhardwaj23 Oct 20262 min read

Applies toDatabricks on Azure, AWS and GCP (Premium/Enterprise); classic compute

1. Policies per persona

Policy	For	Key rules
`personal-compute`	Analysts, exploration	Single node, small VM, auto-terminate 30 min
`shared-standard`	Teams sharing a cluster	Standard (shared) access mode, autoscale 1–4, 60 min auto-terminate
`job-compute`	All scheduled jobs	Job clusters only, spot with fallback, approved runtimes, mandatory tags
`ml-gpu`	Data scientists	Approved GPU types, max workers, auto-terminate 60 min

Remove unrestricted cluster creation from most users; they create compute only through policies.

2. Example: job compute policy

{
  "cluster_type": { "type": "fixed", "value": "job" },
  "spark_version": { "type": "regex", "pattern": "1[5-9]\\.[0-9]+\\.x-(photon-)?scala2\\.(12|13)" },
  "node_type_id": { "type": "allowlist", "values": ["Standard_D8ds_v5", "Standard_D16ds_v5", "Standard_E8ds_v5"] },
  "driver_node_type_id": { "type": "allowlist", "values": ["Standard_D8ds_v5", "Standard_E8ds_v5"] },
  "autoscale.max_workers": { "type": "range", "maxValue": 20, "defaultValue": 8 },
  "azure_attributes.availability": { "type": "fixed", "value": "SPOT_WITH_FALLBACK_AZURE" },
  "azure_attributes.first_on_demand": { "type": "fixed", "value": 1 },
  "data_security_mode": { "type": "allowlist", "values": ["SINGLE_USER", "USER_ISOLATION"] },
  "custom_tags.team": { "type": "unlimited", "isOptional": false },
  "custom_tags.cost_center": { "type": "unlimited", "isOptional": false }
}

(Adjust the runtime regex to the LTS versions you support; on AWS use aws_attributes.*.)

3. Example: personal compute

{
  "spark_conf.spark.databricks.cluster.profile": { "type": "fixed", "value": "singleNode", "hidden": true },
  "num_workers": { "type": "fixed", "value": 0, "hidden": true },
  "node_type_id": { "type": "allowlist", "values": ["Standard_D4ds_v5", "Standard_D8ds_v5"], "defaultValue": "Standard_D4ds_v5" },
  "autotermination_minutes": { "type": "fixed", "value": 30, "hidden": true },
  "data_security_mode": { "type": "fixed", "value": "SINGLE_USER" },
  "custom_tags.team": { "type": "unlimited", "isOptional": false }
}

4. Deploy as code

resource "databricks_cluster_policy" "job" {
  name       = "job-compute"
  definition = file("${path.module}/policies/job-compute.json")
}
resource "databricks_permissions" "job_policy" {
  cluster_policy_id = databricks_cluster_policy.job.id
  access_control { group_name = "grp-data-engineers" permission_level = "CAN_USE" }
}

Policy families (Databricks-provided templates) are a good starting point; override what you need.

5. Roll out

Create policies; give groups CAN_USE.
Report existing clusters that don't comply (system.compute.clusters — policy_id null, no auto-termination).
Announce, then remove unrestricted creation; existing clusters must be edited to use a policy.
Review monthly: policy usage, untagged spend, exception requests.

SELECT cluster_name, owned_by, policy_id, auto_termination_minutes, dbr_version
FROM system.compute.clusters
WHERE delete_time IS NULL AND (policy_id IS NULL OR auto_termination_minutes IS NULL OR auto_termination_minutes = 0);

Tip
Keep the number of policies small (4–6). Too many policies confuse users and lead to "please give me unrestricted access" requests.