Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Transformation: production jobs off all-purpose clusters and onto job compute or serverless

Stop running scheduled production jobs on shared interactive clusters. Move them to job clusters or serverless jobs, handle libraries and init scripts, set policies and timeouts, and compare cost and reliability.

Deepak Bhardwaj3 min read

Applies toAzure Databricks Workflows, job clusters, serverless compute for jobs and notebooks, Unity Catalog

Why

All-purpose cluster for jobs Job cluster / serverless
Higher DBU rate Lower rate (job compute) or pay per use (serverless)
Jobs compete with people's notebooks Isolated per run
Library and config drift by hand Defined in the job, versioned
Runs 24/7 if auto-termination is off Exists only during the run

1. Find jobs on all-purpose clusters

SELECT j.job_id, j.name, t.task_key, t.existing_cluster_id
FROM system.lakeflow.job_tasks t
JOIN system.lakeflow.jobs j USING (workspace_id, job_id)
WHERE t.existing_cluster_id IS NOT NULL AND t.delete_time IS NULL
QUALIFY ROW_NUMBER() OVER (PARTITION BY t.job_id, t.task_key ORDER BY t.change_time DESC) = 1;

Column availability in system tables changes over time — or list jobs with the CLI and check existing_cluster_id per task.

2. Choose the target per job

Job Target
SQL/Python/notebook tasks without custom native libraries Serverless jobs — fast start, no cluster tuning
Needs init scripts, specific node types, GPUs, custom Spark configs Job cluster
Many short tasks in one job Shared job cluster across tasks (job_cluster_key)

3. Job cluster definition (bundle)

resources:
  jobs:
    sales_daily:
      job_clusters:
        - job_cluster_key: main
          new_cluster:
            spark_version: 15.4.x-scala2.12
            node_type_id: Standard_E8ds_v5
            autoscale: { min_workers: 2, max_workers: 8 }
            policy_id: ${var.job_policy_id}
            data_security_mode: SINGLE_USER
      tasks:
        - task_key: load
          job_cluster_key: main
          notebook_task: { notebook_path: ../src/load_sales.py }
          libraries: [ { whl: /Volumes/ops/libs/wheels/sales_lib-2.1.0-py3-none-any.whl } ]
      timeout_seconds: 7200
      max_concurrent_runs: 1

4. Serverless version

      tasks:
        - task_key: load
          notebook_task: { notebook_path: ../src/load_sales.py }
          environment_key: default
      environments:
        - environment_key: default
          spec:
            client: "1"
            dependencies: [ "/Volumes/ops/libs/wheels/sales_lib-2.1.0-py3-none-any.whl", "openpyxl==3.1.5" ]

Serverless jobs don't support everything (for example init scripts, some Spark configs, RDD APIs, certain languages). Check limitations and test each job.

5. Things that break during the move

  • Code relying on state left on the interactive cluster (cached tables, temp views, files on local disk, libraries installed by someone weeks ago).
  • spark.conf values set in the cluster UI — move them into the job/cluster definition.
  • Startup time: job clusters take minutes to start; chain tasks on one job cluster or use pools/serverless.

6. Compare and switch

Run the job on the new compute in a test schedule; compare runtime, DBUs (from system.billing.usage by job_id) and output. Then switch the production schedule and remove the all-purpose cluster's job access.

Tip

A cluster policy for job clusters (node types, max workers, tags) keeps the new setup from drifting into the old one.

Checklist

  • Inventory of jobs using all-purpose clusters
  • Target chosen per job; job specs in bundles
  • Hidden dependencies on interactive clusters removed
  • Timeouts, concurrency, policies set
  • Cost/runtime compared; all-purpose clusters reserved for people

Related fixes