Home / Databricks on Azure

Databricks on AzureHow-toIntermediate

Optimization: Databricks instance pools on Azure - faster cluster starts without paying for idle VMs

Instance pools keep ready VMs so clusters start in less time, but idle pool VMs are billed by Azure. Size min idle and idle timeouts to your schedule, preload runtimes, separate pools for drivers and workers, and decide whether serverless now does the job better.

Deepak Bhardwaj2 min read

Applies toAzure Databricks classic clusters using instance pools

What pools do

  • Clusters attached to a pool take VMs from it instead of requesting new ones from Azure.
  • Idle VMs in a pool cost Azure VM charges (no DBUs while idle).
  • Pools can preload a Databricks Runtime image to cut start time further.

Key settings

Setting Effect Guidance
min_idle_instances VMs kept warm at all times 0 for most pools; >0 only where start time matters every hour
idle_instance_autotermination_minutes How long extra idle VMs live Short (e.g. 10–30 min) unless jobs come in waves
max_capacity Cap on VMs Set to protect quota and budget
preloaded_spark_versions Runtime ready on idle VMs Match the runtime your jobs use
Availability (on-demand/spot) Pool-level Separate spot pool for workers 
{
  "instance_pool_name": "etl-workers-e8ds",
  "node_type_id": "Standard_E8ds_v5",
  "min_idle_instances": 0,
  "max_capacity": 60,
  "idle_instance_autotermination_minutes": 15,
  "preloaded_spark_versions": ["15.4.x-scala2.12"],
  "azure_attributes": { "availability": "SPOT_WITH_FALLBACK_AZURE", "spot_bid_max_price": -1 },
  "custom_tags": { "team": "data-eng" }
}

Pool patterns

Pattern Use
Driver pool (on-demand) + worker pool (spot) Reliable drivers, cheap workers
Pool per VM type and team Clear cost attribution via tags
Shared pool for a batch window Many jobs starting together (e.g. 02:00) reuse warm VMs as each finishes

Is the pool paying off?

Check How
Start time saved Compare cluster start durations with and without the pool
Idle VM hours Pool usage metrics/Azure cost for pool-tagged VMs
Hit rate Were clusters served from idle VMs or did the pool have to grow?

If jobs run rarely, a pool with min_idle_instances: 0 gives little benefit; serverless jobs may be the better answer for start time.

Warning

A pool with min_idle_instances set to a few large VMs runs 24×7 on the Azure bill. Review pools quarterly and remove those that no cluster uses.

Checklist

  • Min idle set to 0 unless justified
  • Idle timeout short
  • Max capacity set
  • Runtime preloaded to match jobs
  • Driver/worker and spot/on-demand split considered
  • Pool tags for cost; unused pools removed

Related fixes