Introduction to Databricks compute types: all-purpose clusters, job compute, SQL warehouses and serverless

#Four kinds of compute

On Azure Databricks, all-purpose compute is billed at a higher DBU rate than job compute for the same VMs (check current pricing for your tier and region). Running scheduled jobs on an always-on all-purpose cluster is one of the most common sources of avoidable cost.

#All-purpose clusters

Good for people working interactively:

• Shared by several users (standard access mode) or one user (dedicated access mode).
• Libraries can be installed on the cluster.
• Stay up between commands, so results come back quickly.

Set auto-termination so forgotten clusters stop:

databricks clusters get <cluster-id> | jq '.autotermination_minutes'

Tip

Use a cluster policy that forces autotermination_minutes between 10 and 60 for all-purpose clusters. Without it, someone will set 0 (never) eventually.

#Job compute

A job compute cluster is defined in the job and lives only for that run:

• Each run gets a fresh cluster: no leftover state, no library conflicts with other users.
• Cheaper DBU rate than all-purpose.
• Several tasks in one job can share one job cluster (define it once under job_clusters and reference it by job_cluster_key).

Example job definition (JSON for the Jobs API / CLI):

{
  "name": "orders-daily",
  "job_clusters": [
    {
      "job_cluster_key": "etl",
      "new_cluster": {
        "spark_version": "15.4.x-scala2.12",
        "node_type_id": "Standard_D8ds_v5",
        "autoscale": { "min_workers": 2, "max_workers": 8 },
        "data_security_mode": "SINGLE_USER",
        "custom_tags": { "costcenter": "sales-data" }
      }
    }
  ],
  "tasks": [
    {
      "task_key": "bronze",
      "job_cluster_key": "etl",
      "notebook_task": { "notebook_path": "/Repos/data/orders/bronze" }
    },
    {
      "task_key": "silver",
      "depends_on": [ { "task_key": "bronze" } ],
      "job_cluster_key": "etl",
      "notebook_task": { "notebook_path": "/Repos/data/orders/silver" }
    }
  ],
  "schedule": { "quartz_cron_expression": "0 30 2 * * ?", "timezone_id": "Asia/Kolkata" }
}

databricks jobs create --json @orders-daily.json

Check the runtime versions available in your workspace with databricks clusters spark-versions, and VM types with databricks clusters list-node-types.

#SQL warehouses

For SQL users and BI tools (Power BI, Tableau, DBSQL dashboards):

• Serverless warehouses start in seconds and are managed by Databricks.
• Pro and classic run in your subscription and take a few minutes to start.
• Size (2X-Small ... 4X-Large) sets per-cluster compute; scaling (min/max clusters) handles concurrency.

Use a SQL warehouse, not an all-purpose cluster, for Power BI. It's built for many small concurrent queries.

#Serverless compute for notebooks and jobs

Where enabled in your workspace and region, serverless compute starts almost instantly and you don't pick VM sizes. Trade-offs to know:

• Network access to your private resources needs network connectivity configurations (NCC) and private endpoints or firewall rules set up by an admin.
• Limited control over Spark configs and init scripts; libraries go in the notebook environment.
• Billing is per serverless DBU; check pricing and use budget policies/tags.

#Moving a job off an all-purpose cluster

Find jobs that use an existing cluster:

databricks jobs list --output json | jq -r '.[] | .job_id' | while read id; do
  databricks jobs get $id --output json | \
    jq -r --arg id "$id" '.settings.tasks[]? | select(.existing_cluster_id != null) | "\($id)\t\(.task_key)\t\(.existing_cluster_id)"'
done

For each one:

1. Copy the cluster's spark version, node type, libraries and Spark config into a job_clusters entry.
2. Replace existing_cluster_id with job_cluster_key in each task, and move libraries to the task's libraries list.
3. Run it once manually, compare duration and output.
4. Reduce the all-purpose cluster's size or delete it if it was only there for jobs.

Warning

Jobs that relied on state left on a long-running cluster (temp views, files on local disk, cached tables, libraries someone installed by hand) will fail on fresh job compute. That's a good thing to find, but test before the next scheduled run.

#Picking quickly

#Prevent problems later

• Cluster policies: one for interactive work (auto-termination, max size), one for jobs.
• Tag all compute with cost center and owner (custom_tags) so Azure Cost Management and system tables can split costs.
• Review system.billing.usage monthly for all-purpose DBUs used by job runs.
• Keep job definitions in Databricks Asset Bundles in Git, so compute settings are reviewed like code.