Sev1: Azure Databricks clusters won't start — VNet-injected subnets out of IP addresses

#Symptoms

• Clusters and job runs fail to start; termination reason such as SUBNET_EXHAUSTED_FAILURE, or launch errors mentioning the subnet having no available addresses.
• Happens at peak (many jobs at 02:00) or after autoscaling limits were raised.

#1. Confirm

databricks clusters events <cluster-id> --event-types TERMINATING --output json | jq '.events[0].details.reason'

Each cluster node uses one IP in the host (public) subnet and one in the container (private) subnet. So the maximum number of nodes running at once in the workspace is roughly the number of usable IPs in the smaller subnet (Azure reserves 5 per subnet).

az network vnet subnet show -g rg-dbx-net --vnet-name vnet-dbx -n snet-dbx-private --query "{prefix:addressPrefix}" -o table
# Cluster node NICs live in the workspace's managed resource group; count them at peak
az network nic list -g <managed-resource-group> --query "length(@)"

#2. Immediate relief

• Terminate idle all-purpose clusters and stop non-critical jobs.
• Lower max_workers in autoscaling for large non-critical jobs.
• Stagger the 02:00 job pile-up (move some schedules by 30–60 minutes).
• Use pools carefully: idle pool instances also hold IPs.

databricks clusters list --output json | jq -r '.[] | select(.state=="RUNNING") | [.cluster_id, .cluster_name, (.num_workers // .autoscale.max_workers)] | @tsv'

#3. Durable fix: bigger subnets

You can change the subnets of an existing VNet-injected workspace to larger ones (in the same VNet) — check the current Azure Databricks documentation for the "update workspace network configuration" procedure and its restrictions. Steps in outline:

1. Add address space to the VNet if needed (az network vnet update --address-prefixes).
2. Create new, larger host and container subnets with the Databricks delegation (Microsoft.Databricks/workspaces) and NSG.
3. Update the workspace to use the new subnets (ARM/Bicep/Terraform update of customPublicSubnetName/customPrivateSubnetName).
4. Restart running clusters so they move.

az network vnet subnet create -g rg-dbx-net --vnet-name vnet-dbx -n snet-dbx-private-v2 \
  --address-prefixes 10.20.8.0/22 --delegations Microsoft.Databricks/workspaces --network-security-group nsg-dbx

Warning

Subnet changes on a workspace need all clusters stopped during the switch on some configurations. Plan a short window and check the documented constraints first.

#4. Use less IP space

• Serverless jobs and SQL warehouses run in Databricks' serverless compute plane and don't consume your subnet IPs.
• Fewer, larger nodes instead of many small ones for the same core count.
• Job clusters that terminate on completion instead of long-lived all-purpose clusters.

#Prevent it next time

• Size subnets for peak concurrent nodes plus headroom (a /22 or larger for busy workspaces).
• Track concurrent running nodes vs. subnet capacity; alert at 80%.
• Spread heavy schedules across time.