Azure Databricks cluster still running overnight: auto termination and SQL warehouse auto stop

#Symptom

You come in on Monday and find an all-purpose cluster has been up all weekend. The Azure bill shows VMs that nobody remembers starting. Auto termination might be off, or it might be on and not doing what you expected.

#How it works

For an all-purpose cluster, "inactive" means no Spark jobs, no Structured Streaming and no JDBC or ODBC calls for the configured time.

#Step 1: find what's running now

databricks clusters list --output json \
  | jq -r '.[] | select(.state=="RUNNING") | [.cluster_id, .cluster_name, .creator_user_name, .autotermination_minutes, .cluster_source] | @tsv'

databricks warehouses list --output json \
  | jq -r '.[] | [.id, .name, .state, .auto_stop_mins] | @tsv'

The list output format depends on your CLI version. If jq fails, look at the raw output first.

autotermination_minutes of 0 means the feature is off: the cluster runs until someone stops it.

Find all such clusters using system tables, if your workspace has them enabled:

SELECT cluster_id, cluster_name, owned_by, auto_termination_minutes, change_time
FROM system.compute.clusters
WHERE delete_time IS NULL
  AND cluster_source IN ('UI', 'API')
  AND COALESCE(auto_termination_minutes, 0) = 0
QUALIFY ROW_NUMBER() OVER (PARTITION BY cluster_id ORDER BY change_time DESC) = 1;

#Step 2: why isn't an idle-looking cluster stopping?

Look at Compute > your cluster > Event log too. It records who started and resized a cluster, and any termination. An automatic job may be what's starting it again.

Remember: a notebook left open in a browser, with no running cell, doesn't count as activity, but a running cell does.

#Step 3: stop it now

databricks clusters delete 0123-456789-abcde123

Despite the name, clusters delete terminates the cluster and keeps its configuration, so you can start it again. clusters permanent-delete is the one that removes it.

Warning

Check before you stop anything. Terminating a cluster kills running queries and loses data cached or held in notebook memory. Check the Spark UI and ask the owner, especially if a long-running notebook is attached.

databricks warehouses stop <warehouse-id>

#Step 4: set auto termination

In the UI, go to Compute > cluster > Edit, and set Terminate after N minutes of inactivity. The allowed range begins at 10 minutes.

With the CLI, the edit call needs the cluster's core settings, so start from what's there:

databricks clusters get 0123-456789-abcde123 > cluster.json

Edit the file, keeping only the settings you need, then add "autotermination_minutes": 30:

{
  "cluster_id": "0123-456789-abcde123",
  "cluster_name": "team-analytics",
  "spark_version": "15.4.x-scala2.12",
  "node_type_id": "Standard_D4ds_v5",
  "num_workers": 2,
  "autotermination_minutes": 30
}

databricks clusters edit --json @cluster.json

Editing a running cluster may restart it, so do it when nobody is using it. Fields that you leave out can reset to their defaults, so include anything you've customized, such as Spark config and libraries.

Suggested values:

#Step 5: set SQL warehouse auto stop

databricks warehouses edit <warehouse-id> --auto-stop-mins 10

Serverless warehouses start quickly, so a short auto stop works well. Classic and Pro warehouses take longer to start, so people may prefer a longer value. A dashboard with a scheduled refresh can keep a warehouse awake. Check Query history if it never stops.

#Step 6: enforce it with a compute policy

Setting each cluster by hand doesn't last. A compute policy makes the rule stick. Workspace admins can create it:

{
  "autotermination_minutes": {
    "type": "range",
    "minValue": 10,
    "maxValue": 60,
    "defaultValue": 30
  },
  "custom_tags.team": {
    "type": "unlimited",
    "isOptional": false
  }
}

databricks cluster-policies create --json '{
  "name": "standard-interactive",
  "definition": "{\"autotermination_minutes\":{\"type\":\"range\",\"minValue\":10,\"maxValue\":60,\"defaultValue\":30}}"
}'

Then give users access to that policy, and remove their unrestricted cluster creation right, so new clusters must use it.

#Check the savings

SELECT usage_metadata.cluster_id,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 7
  AND usage_metadata.cluster_id IS NOT NULL
GROUP BY 1
ORDER BY 2 DESC
LIMIT 20;

Remember that Azure also bills separately for the VMs, so total cost is more than DBUs alone. The billing queries post has more options, and cost tags shows how to see costs by team in Azure.

#Prevent it next time

• Use one policy for all interactive compute, with a maximum auto-termination value.
• Run the Step 1 query weekly, and send the results to the team.
• Move scheduled work from all-purpose to job compute, which ends with each run.
• Ask teams to stop clusters at the end of the day instead of waiting for the timer.