Home / Databricks on Azure

Databricks on AzureTroubleshootingAdvanced

Sev1: Azure Databricks clusters fail to connect to the control plane — SCC relay and Private Link issues

Fix clusters that terminate during start because they can't reach the Databricks control plane (secure cluster connectivity relay) or Azure dependencies after a firewall, UDR or DNS change. Check required egress, Private Link DNS and NSG rules.

Deepak Bhardwaj3 min read

Applies toAzure Databricks with VNet injection, secure cluster connectivity (no public IP), back-end Private Link, Azure Firewall / NVAs

Symptoms

  • Clusters terminate at start with network-related reasons, for example tunnel/relay setup failures or CONTROL_PLANE_REQUEST_FAILURE, or driver unreachable.
  • Started right after a firewall rule change, a new UDR to an NVA, or DNS changes in the hub.

1. What a classic cluster needs to reach

Destination Why
SCC relay (control plane) Cluster ↔ control plane tunnel
Workspace webapp / control plane APIs Cluster registration
Workspace artifact and log storage Runtime images, logs
Hive metastore (legacy) Only if used
Event Hubs endpoint Logging / telemetry
Your ADLS, Key Vault, package mirrors Data and libraries

The exact FQDNs/IPs and service tags per region are listed in the Azure Databricks "user-defined route settings" documentation. With Azure Firewall you can use the AzureDatabricks service tag for control plane addresses plus FQDN rules for storage endpoints.

2. Find what changed

az network route-table route list -g rg-dbx-net --route-table-name rt-dbx -o table
az network nsg rule list -g rg-dbx-net --nsg-name nsg-dbx --include-default -o table
az monitor activity-log list -g rg-hub --offset 12h --query "[?contains(operationName.value,'firewallPolicies') || contains(operationName.value,'routeTables')].{t:eventTimestamp, op:operationName.value, who:caller}" -o table

Firewall logs show denies from the Databricks subnets:

AZFWApplicationRule
| where TimeGenerated > ago(2h) and SourceIp startswith "10.20."
| where Action == "Deny"
| summarize count() by Fqdn, DestinationPort

3. NSG rules

The workspace's network intent policy adds required NSG rules on the delegated subnets. Don't remove or override them with higher-priority denies. Required patterns include inbound/outbound between the two Databricks subnets (worker-to-worker) and outbound to AzureDatabricks, Storage, EventHub (and Sql if legacy metastore) service tags.

With back-end Private Link, clusters reach the control plane via a private endpoint (databricks_ui_api sub-resource) in the transit/data plane VNet.

  • DNS: privatelink.azuredatabricks.net zone must be linked to the workspace VNet (or resolvable through your resolver), and resolve the workspace and relay names to private IPs.
  • The private endpoint must be approved and in the right VNet.
nslookup adb-1234567890123456.7.azuredatabricks.net     # from a VM in the Databricks VNet
az network private-endpoint list -g rg-dbx-net --query "[].{name:name, state:privateLinkServiceConnections[0].privateLinkServiceConnectionState.status}" -o table

5. Asymmetric routing

A UDR sending the control-plane traffic through a firewall, while return traffic takes another path, breaks connections. Follow the documented UDR guidance: routes for control plane IPs (or service tag) with next hop Internet, or symmetric routing through the firewall with SNAT.

Tip

Keep a tiny "network canary" job: a single-node cluster that starts every hour and runs SELECT 1. It alerts you to network breakage before the 02:00 batch does.

Prevent it next time

  • Treat Databricks subnets' NSGs, UDRs and firewall rules as a protected baseline in IaC with review from the data platform team.
  • Use service tags rather than hard-coded IPs where possible.
  • Canary cluster job with alerting.

Related fixes