Home / Databricks on Azure

Databricks on AzureRunbookIntermediate

Sev1: Databricks spend spiking today — find the runaway compute and stop it

Treat a sudden Azure Databricks cost jump as an incident. Use system.billing tables to find the workspace, SKU, cluster or job behind it, stop the waste safely and put budgets, policies and auto-termination in place.

Deepak Bhardwaj2 min read

Applies toAzure Databricks with Unity Catalog system tables (system.billing, system.compute, system.lakeflow), cluster policies, budgets

1. Where is the money going today?

SELECT workspace_id, sku_name, usage_metadata.cluster_id, usage_metadata.job_id,
       usage_metadata.warehouse_id, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_date >= current_date() - 1
GROUP BY ALL
ORDER BY dbus DESC
LIMIT 20;

Compare with the same day last week. Join system.billing.list_prices for an approximate cost (list price, not your discount).

2. Identify the resource

-- Clusters: who owns them and how they're configured
SELECT cluster_id, cluster_name, owned_by, worker_node_type, max_autoscale_workers, auto_termination_minutes
FROM system.compute.clusters
WHERE cluster_id IN ('0925-101530-abc123')
QUALIFY ROW_NUMBER() OVER (PARTITION BY cluster_id ORDER BY change_time DESC) = 1;

-- Jobs
SELECT job_id, name, creator_id FROM system.lakeflow.jobs WHERE job_id = '1234567890'
QUALIFY ROW_NUMBER() OVER (PARTITION BY job_id ORDER BY change_time DESC) = 1;

3. Usual suspects

Pattern Cause
All-purpose cluster with DBUs every hour all night auto_termination_minutes = 0 or a notebook keeping it busy
A job's DBUs 10x normal Data volume spike, infinite loop, while True polling, retries
Streaming job on a big cluster 24/7 Continuous trigger where a scheduled availableNow would do
SQL warehouse never stopping A tool pinging it every few minutes; auto-stop disabled
GPU nodes ML experiment left running

4. Stop it safely

databricks clusters delete <cluster-id>          # terminates (doesn't remove config)
databricks jobs cancel-all-runs --job-id <job-id>
databricks warehouses stop <warehouse-id>

Tell the owner before terminating anything that may be mid-write; for jobs, prefer cancelling and re-running later.

5. Guardrails

Cluster policy for all-purpose compute:

{
  "autotermination_minutes": { "type": "range", "maxValue": 60, "defaultValue": 30 },
  "autoscale.max_workers": { "type": "range", "maxValue": 8 },
  "node_type_id": { "type": "allowlist", "values": ["Standard_D8ds_v5", "Standard_E8ds_v5"] },
  "custom_tags.costcenter": { "type": "unlimited", "isOptional": false }
}
  • Remove unrestricted cluster creation from users; give them the policy instead.
  • Budgets with alerts per workspace or tag (account console → Budgets), and Azure Cost Management budgets on the resource group.
  • Job runs: timeouts (timeout_seconds) on every production job.

Tip

A daily report of the top 10 clusters/jobs by DBUs, sent to the owners, prevents most spikes from lasting more than a day.

Prevent it next time

  • Auto-termination enforced by policy; tags required for cost allocation.
  • Timeouts on jobs and statement timeouts on SQL warehouses.
  • Alert on daily DBUs above a threshold per workspace from system.billing.usage.

Related fixes