Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Optimization: choosing Azure VM families for Databricks clusters - memory, compute, storage-optimized and local SSD

The VM type under a classic cluster decides how much memory per core, local disk for caching and shuffle, and price per hour you get. Match the family to the workload shape using Spark UI evidence, prefer fewer larger nodes for shuffle-heavy work, use local-SSD types for disk cache, and re-test after each change.

Deepak Bhardwaj2 min read

Applies toAzure Databricks classic job and all-purpose clusters (VM families such as D, E, F, L series and their newer generations); not serverless compute, where Databricks picks the hardware

Workload shape → VM family

Workload Signs in Spark UI / metrics Family to try
General ETL, joins of medium tables Balanced CPU and memory, modest spill General purpose (D-series, newer generations)
Large joins/aggregations, wide tables, caching Spill to disk, GC time high, OOM Memory optimized (E-series)
CPU-heavy transforms, parsing, UDFs, ML feature work CPU near 100%, little spill Compute optimized (F-series)
Repeated reads of the same Delta data (BI, interactive) Same files scanned again and again Storage optimized with local NVMe (L-series) or types with large local SSD for disk cache
Deep learning GPU work GPU series (NC/ND families) with GPU runtimes

Newer VM generations usually give better price/performance. Check which types your region offers and your vCPU quota per family before switching (see the vCPU quota Sev1 post).

Fewer large vs many small nodes

Fewer, larger workers Many small workers
Less shuffle over the network, more memory per executor Finer autoscaling steps
Better for joins/aggregations Better for embarrassingly parallel tasks
A lost node hurts more More per-node overhead

A common sweet spot for ETL is mid-size nodes (e.g. 8–16 cores). Test with your own jobs.

Driver sizing

  • The driver needs more memory when you collect(), broadcast large tables, or run many concurrent streams/notebooks.
  • For most jobs, the driver can be the same or smaller than workers.

Measure, change, measure

  1. Pick one heavy job. Record run time, DBU and VM cost from system tables for a few runs.
  2. Change one thing (family or size).
  3. Compare cost per run, not only runtime.
-- Cost per run for a job (DBU side; VM cost comes from Azure Cost Management)
SELECT usage_metadata.job_run_id, SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE usage_metadata.job_id = '<job_id>'
  AND usage_date >= current_date() - 30
GROUP BY 1 ORDER BY 1 DESC;

Tip

Remember both bills: Databricks DBUs and Azure VM/disk charges. A VM type with a higher hourly price can still be cheaper per run if the job finishes much faster.

Checklist

  • Workload shape read from Spark UI and metrics
  • VM family chosen to match
  • Regional availability and quota checked
  • Node count/size tested on real jobs
  • Cost per run compared (DBU + VM)
  • Choice enforced through cluster policies

Related fixes