Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Optimization: Azure spot VMs for Databricks workers - which jobs, fallback settings and eviction handling

Spot VMs cut the VM part of Databricks cost for workloads that can tolerate losing workers. Keep the driver on-demand, use spot with fallback to on-demand for workers, choose which jobs qualify, and watch eviction rates and run-time impact so savings don't turn into missed SLAs.

Deepak Bhardwaj2 min read

Applies toAzure Databricks classic clusters (job and all-purpose) using azure_attributes availability settings

How it works on Azure Databricks

  • azure_attributes.availability: ON_DEMAND_AZURE, SPOT_AZURE or SPOT_WITH_FALLBACK_AZURE.
  • first_on_demand: how many nodes (starting with the driver) are always on-demand.
  • spot_bid_max_price: -1 means pay up to the on-demand price (never evicted for price, only for capacity).
{
  "spark_version": "15.4.x-scala2.12",
  "node_type_id": "Standard_E8ds_v5",
  "autoscale": { "min_workers": 2, "max_workers": 12 },
  "azure_attributes": {
    "first_on_demand": 1,
    "availability": "SPOT_WITH_FALLBACK_AZURE",
    "spot_bid_max_price": -1
  }
}

first_on_demand: 1 keeps the driver on-demand; losing the driver fails the job.

Which jobs suit spot

Good fit Poor fit
Batch ETL with retries and slack before the deadline Tight-SLA jobs close to their deadline
Idempotent backfills Long single-stage jobs where losing nodes restarts much work
Dev/test clusters Streaming with large state on workers
Large shuffles? Only with care Jobs that already struggle with stability

When a spot worker is evicted, Spark recomputes lost tasks/shuffle data. Short tasks and Delta checkpoints keep the impact small.

Make jobs tolerant

  • Job and task retries configured (see the reliability post).
  • Avoid caching critical data only on workers.
  • Use SPOT_WITH_FALLBACK_AZURE so the cluster can still start when spot capacity is short.
  • Spread across VM types that have spot capacity in your region.

Measure the trade-off

Metric Source
Eviction/node-loss events Cluster event log (NODES_LOST, spot preemption reasons)
Run time change Job run history
Savings Azure Cost Management VM cost by cluster tag 
# Cluster events for a job cluster: look for NODES_LOST and spot-related reasons
databricks clusters events <cluster-id> --output json \
  | jq '.[] | select(.type=="NODES_LOST" or .type=="UPSIZE_COMPLETED") | {type, timestamp, details}'

For a fleet view, system.compute.node_timeline shows per-node lifetimes; nodes that end long before their cluster did are often evicted spot workers.

Warning

Spot savings apply to the Azure VM cost, not to Databricks DBUs. If evictions make a job run 40% longer, the extra DBUs can cancel out the VM saving.

Checklist

  • Driver on-demand (first_on_demand ≥ 1)
  • Spot with fallback for workers
  • Only jobs with slack and retries moved to spot
  • Evictions and run time monitored
  • Savings measured on VM cost, DBU impact checked
  • Spot settings applied through cluster policies

Related fixes