Your first Databricks cluster: which runtime, access mode, size and auto-termination to choose

#The goal

You open Compute in Azure Databricks, select Create compute, and face a long form. Most of it can stay at the defaults. This post explains the handful of settings that affect whether your code works and how much it costs.

If you're brand new, begin with workspace first steps. If you only want to run SQL, you might not need a cluster at all. See SQL warehouse basics and choosing compute.

#The short answer

For a learning or development cluster:

#Setting 1: the runtime

The Databricks Runtime is the bundle of Spark, Delta and libraries. Pick a version marked LTS, unless you need a newer feature.

• Your code and libraries must match the Spark and Python versions it contains.
• ML runtimes add machine learning libraries, and take longer to start. Use them only if you need them.
• Photon speeds up many SQL and DataFrame workloads, and it costs more per hour.

List what's available in your workspace:

databricks clusters spark-versions --output json | grep -E '"name".*LTS' | head -10

For upgrades later, see LTS runtime upgrades.

#Setting 2: the access mode

This is the setting that causes the most confusion, as it changes what your code is allowed to do.

Standard mode has limits, such as no RDD APIs and restricted sparkContext access. If you hit those, read Standard access mode errors. To use Unity Catalog, you must choose Standard or Dedicated.

#Setting 3: single node or multi-node

For a first cluster, single node is the right choice. See single-node dev compute. When you do use workers, start with a range like 1 to 4. A big maximum can be an expensive surprise, and autoscaling needs tuning to work well.

#Setting 4: node size

Each VM family suits a different kind of work:

Start small, and scale up only when the Spark UI shows that you need to. See the VM family guide. List what your workspace offers:

databricks clusters list-node-types --output json | grep '"node_type_id"' | head -20

Check the VM's availability and your quota for the region, because the cluster won't start without them. See launch failures.

#Setting 5: auto-termination

Always set this. An all-purpose cluster with no auto-termination will run, and bill, until someone stops it.

Clusters terminate after the set number of idle minutes. A terminated cluster keeps its configuration, and you can restart it. For more on this, see idle clusters and auto-stop.

Warning

You pay twice for a running cluster: Databricks DBUs, and the Azure VMs. Idle all-purpose clusters are one of the most common sources of waste. If a job needs to run on a schedule, use job compute instead, which costs less and shuts down when finished.

#Create it from the CLI

Save this as cluster.json, and replace the runtime and node type with values from your workspace:

{
  "cluster_name": "dev-asha",
  "spark_version": "<lts-runtime-key>",
  "node_type_id": "Standard_D4ds_v5",
  "num_workers": 2,
  "data_security_mode": "SINGLE_USER",
  "single_user_name": "asha@contoso.com",
  "autotermination_minutes": 30,
  "custom_tags": {
    "owner": "asha",
    "purpose": "dev"
  }
}

In the API, SINGLE_USER is Dedicated mode and USER_ISOLATION is Standard. Create it and check it:

databricks clusters create --json @cluster.json

databricks clusters list --output json | grep -E '"cluster_name"|"state"'

If you're new to the CLI, see the CLI cheat sheet. Azure tags also show up in your cost reports, as long as tags are propagating.

#Test it

Attach a notebook to the cluster, and run this:

print(spark.version)
spark.range(1000000).selectExpr("sum(id) AS total").show()

Then find out which catalogs you can see:

SHOW CATALOGS;
SELECT current_user(), current_catalog();

If a catalog you expect isn't listed, check your grants with Unity Catalog basics.

#Start, stop and delete

databricks clusters start <cluster-id>

# "delete" means terminate here. The configuration is kept
databricks clusters delete <cluster-id>

# This removes it for good
databricks clusters permanent-delete <cluster-id>

Caution

permanent-delete can't be undone, and any libraries and settings on that cluster are lost. Keep important configuration in a cluster policy or in code.

#Common problems

#Prevent it next time

• Create a personal development cluster with a short auto-termination time.
• Share a Standard cluster within the team, instead of having many idle ones.
• Have an admin set cluster policies, so that good defaults are automatic.
• Tag each cluster with an owner, so that you can tell who's paying.