Home / Data Platform / Databricks & Spark

Data PlatformHow-toBeginner

Right-sizing standalone Databricks compute: single-node clusters, job clusters and serverless

When a single-node cluster is the right choice, how to configure it, how it differs from multi-node job clusters and serverless compute, and policies that stop small workloads from burning large budgets.

Deepak Bhardwaj2 min read

Applies toDatabricks on Azure, AWS and GCP

When single-node is right

  • Development and debugging with sample data.
  • Small jobs (a few GB) using pandas, Polars, scikit-learn or Spark in local mode.
  • ML training that runs on one large (or GPU) machine anyway.

A single-node cluster runs the driver only; Spark uses local threads (local[*]). No worker VMs, no shuffle over the network.

Configure it

In the UI: Compute → Create → Single node. Or in a job/bundle:

job_clusters:
  - job_cluster_key: small
    new_cluster:
      spark_version: 15.4.x-scala2.12
      node_type_id: Standard_E8ds_v5
      num_workers: 0
      spark_conf:
        spark.databricks.cluster.profile: singleNode
        spark.master: "local[*, 4]"
      custom_tags:
        ResourceClass: SingleNode

local[*, 4] allows 4 task retries, closer to cluster behaviour.

Choose the compute type

Need Compute
Interactive dev, small data Single-node all-purpose cluster with auto-termination 30 min
Scheduled pipeline Job cluster (created per run, cheaper DBU rate) or serverless jobs
Large shuffle-heavy ETL Multi-node job cluster with autoscaling
SQL/BI SQL warehouse (serverless where available)
Quick start without cluster management Serverless notebooks/jobs

Policies

{
  "spark_conf.spark.databricks.cluster.profile": { "type": "fixed", "value": "singleNode" },
  "num_workers": { "type": "fixed", "value": 0 },
  "node_type_id": { "type": "allowlist", "values": ["Standard_E4ds_v5", "Standard_E8ds_v5"] },
  "autotermination_minutes": { "type": "range", "maxValue": 60, "defaultValue": 30 }
}

Give developers this "Personal Compute" style policy by default, and multi-node policies only on request.

Signs you've outgrown single-node

  • Driver memory errors / OOM on joins or collect().
  • Jobs that take hours where data size suggests minutes.
  • Spark UI shows long single-threaded stages on large inputs.

Then switch to a job cluster with workers, or rewrite the step to push work to Spark/SQL instead of pandas on the driver.

Reliability notes

Single-node = single point of failure for that run. For scheduled jobs, rely on job retries (task-level max_retries, min_retry_interval_millis) and idempotent writes, so a lost VM just reruns the task.

Related fixes