Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Optimization: tuning Databricks autoscaling on Azure - min/max workers, scale-down and when to use fixed size

Autoscaling helps uneven workloads but can also waste money or slow jobs. Set minimum and maximum workers from observed usage, understand how scale-down works for jobs versus interactive clusters, use fixed-size clusters for steady batch jobs, and check node utilisation in system tables.

Deepak Bhardwaj2 min read

Applies toAzure Databricks classic clusters (job and all-purpose) with autoscaling; Lakeflow Declarative Pipelines enhanced autoscaling

Autoscale or fixed?

Workload Recommendation
Batch job with steady parallelism Fixed size, tuned from past runs
Batch job with very uneven stages (big shuffle, then small writes) Autoscale with sensible max
Shared all-purpose cluster with varied users Autoscale with low min and auto-termination
Streaming Fixed size, or pipelines with enhanced autoscaling
Lakeflow Declarative Pipelines Enhanced autoscaling

Autoscaling adds VMs on demand, which takes minutes on Azure. For short jobs, that delay can cost more than a fixed-size cluster.

Pick min and max from evidence

-- Average CPU per cluster over its nodes (last 7 days)
SELECT cluster_id,
       AVG(cpu_user_percent + cpu_system_percent) AS avg_cpu,
       MAX(cpu_user_percent + cpu_system_percent) AS peak_cpu,
       COUNT(DISTINCT instance_id) AS nodes_seen
FROM system.compute.node_timeline
WHERE start_time >= current_date() - 7
GROUP BY cluster_id
ORDER BY avg_cpu;
Observation Action
Cluster always at max workers, high CPU Raise max or optimise the job
Cluster rarely above min Lower max; maybe use fixed small size
Low CPU across nodes at max Bottleneck is elsewhere (skew, I/O, driver); more nodes won't help
Min workers high on interactive clusters Lower min; idle workers cost money

Scale-down behaviour

  • Job clusters with optimized autoscaling can scale down when executors are idle, but nodes holding shuffle files may be kept.
  • All-purpose clusters scale down more conservatively. Combine with auto-termination for idle time.
  • Check current runtime docs for exact scale-down timing.

Guardrails

  • Cluster policies cap autoscale.max_workers per team or job type.
  • Separate policies for interactive and job clusters.
  • Alert when a cluster sits at max for long periods (capacity or code issue).

Tip

If autoscaling keeps adding nodes but run time doesn't improve, look at the Spark UI for skew or a single long task. More workers can't speed up one straggler.

Checklist

  • Autoscale vs fixed chosen per workload
  • Min/max set from utilisation data
  • Auto-termination on interactive clusters
  • Max limited by cluster policy
  • Clusters at max reviewed for skew or bottlenecks

Related fixes