Home / Databricks on Azure

Databricks on AzureTroubleshootingIntermediate

Sev1: Databricks clusters fail with cloud provider launch errors — Azure vCPU quota and VM stockouts

Recover when job clusters can't get VMs from Azure: quota exceeded for a VM family, regional capacity stockouts, or spot evictions. Read the termination reason, switch node types or capacity type fast, and request quota before the next peak.

Deepak Bhardwaj2 min read

Applies toAzure Databricks classic compute (job and all-purpose clusters, pools), Azure subscription quotas

Symptoms

  • Cluster start fails with termination reasons like AZURE_QUOTA_EXCEEDED_EXCEPTION, CLOUD_PROVIDER_LAUNCH_FAILURE or CLOUD_PROVIDER_RESOURCE_STOCKOUT.
  • Clusters start with fewer workers than requested and jobs run slowly.
  • Spot-based clusters lose workers repeatedly.

1. Read the reason

databricks clusters events <cluster-id> --output json | jq '.events[] | select(.type=="TERMINATING" or .type=="FAILED_TO_EXPAND_DISK" or .type=="NODES_LOST") | {type, details}'

The details.reason.parameters often contain the Azure error message, including the VM family and the limit.

2. Quota exceeded

az vm list-usage -l westeurope -o table | grep -Ei "family|Total Regional"

Each VM family (for example Standard EDSv5 Family vCPUs) has its own limit, plus the total regional vCPU limit. Peaks from many jobs starting together hit them.

Quick fixes:

  • Switch some jobs to a different VM family with spare quota (e.g. from Standard_E8ds_v5 to Standard_E8as_v5 or Standard_D16ds_v5).
  • Reduce max_workers on non-critical jobs; stagger schedules.

Then request more quota (portal → Quotas → Compute, or a support request). Small increases are often approved automatically.

3. Capacity stockout (not your quota)

The region temporarily has no capacity for that VM size.

  • Try another size in the same family or another family.
  • Use multiple instance types via pools or fleet options where available, so the cluster can use whatever is available.
  • Serverless jobs avoid VM procurement in your subscription entirely — a good escape hatch for critical workloads.

4. Spot problems

"azure_attributes": { "availability": "SPOT_WITH_FALLBACK_AZURE", "first_on_demand": 1, "spot_bid_max_price": -1 }
  • SPOT_WITH_FALLBACK_AZURE falls back to on-demand when spot isn't available.
  • first_on_demand: 1 keeps the driver on on-demand capacity so an eviction doesn't kill the job.
  • For SLA-critical jobs, use on-demand workers.

5. Get tonight's critical jobs through

  1. Pause non-critical schedules.
  2. Edit critical job clusters to a node type with available quota/capacity (or serverless).
  3. Re-run failed runs (databricks jobs repair-run <run-id> --rerun-all-failed-tasks).

Tip

Keep a documented alternate node type for each critical job (tested once), so the switch during an incident is a one-line change in the bundle or job definition.

Prevent it next time

  • Quota headroom of ~30% above observed peak per VM family; review monthly.
  • Cluster policies that limit max workers per job.
  • Alerts on cluster launch failures from job run results or system tables (system.compute / system.lakeflow tables).

Related fixes