Azure Databricks cluster can't reach a database or API: DNS, NSG, route and firewall checks

A JDBC read fails with Connection timed out, UnknownHostException or The TCP/IP connection to the host ... has failed, or a REST call from a notebook hangs and then errors. The workspace UI works and other jobs run, but this one destination can't be reached. Almost every time the cause is on the network path between the cluster subnets and the target, not in Spark.

This post covers classic compute in a VNet-injected workspace. Serverless compute leaves Azure from Databricks-managed networks and is controlled by network connectivity configurations (NCCs) and network policies, so the fix there is different.

#Read the error first

#Step 1: test from the cluster itself

Run this in a notebook on the same cluster (or the same job cluster definition) that fails. It uses only the Python standard library, so it works in every access mode:

import socket, time

def probe(host, port, timeout=5):
    try:
        addrs = sorted({a[4][0] for a in socket.getaddrinfo(host, port, proto=socket.IPPROTO_TCP)})
    except socket.gaierror as e:
        return f"{host}: DNS FAILED ({e})"
    for ip in addrs:
        t = time.time()
        try:
            with socket.create_connection((ip, port), timeout=timeout):
                return f"{host} -> {ip}:{port} OPEN in {time.time()-t:.2f}s"
        except Exception as e:
            last = f"{host} -> {ip}:{port} {type(e).__name__}: {e}"
    return last

for host, port in [("sql-prod-01.database.windows.net", 1433),
                   ("onprem-sql.corp.contoso.com", 1433),
                   ("api.partner.example.com", 443)]:
    print(probe(host, port))

Look at three things in the output:

1. DNS FAILED: go to Step 2.
2. Resolved IP is wrong, for example a public IP when the target has a private endpoint: go to Step 2.
3. timeout: DNS is fine, packets are dropped. Go to Step 3.

Tip

Run the probe on the driver and on the executors. The driver and workers can sit on different hosts. To run it on workers, use sc.parallelize(range(4), 4).map(lambda _: probe("onprem-sql.corp.contoso.com", 1433)).collect(). If only the workers fail, look at NSG rules or on-prem firewall rules that allow a single IP instead of the subnet ranges.

#Step 2: DNS

Find the workspace VNet and subnets (in the CLI output these sit under parameters; field names can vary by API version):

az databricks workspace show -g rg-dbx-prod -n dbx-prod \
  --query "{vnet:parameters.customVirtualNetworkId.value, host:parameters.customPublicSubnetName.value, container:parameters.customPrivateSubnetName.value, npip:parameters.enableNoPublicIp.value}" -o json

Check which DNS servers the VNet hands out. An empty list means Azure-provided DNS (168.63.129.16):

az network vnet show --ids <vnet-resource-id> --query "dhcpOptions.dnsServers" -o json

For a private endpoint target (Azure SQL, Storage, Key Vault, a Synapse or Cosmos DB endpoint), the matching privatelink.* zone must resolve from the workspace VNet:

# Is the zone linked to the workspace VNet (or to a hub whose DNS resolver the VNet uses)?
az network private-dns link vnet list -g rg-dns -z privatelink.database.windows.net -o table

# Does the zone have an A record for the server?
az network private-dns record-set a list -g rg-dns -z privatelink.database.windows.net \
  --query "[].{name:name, ip:aRecords[0].ipv4Address}" -o table

Warning

Cluster nodes pick up VNet DNS settings when they boot. After you change DNS servers or forwarders, restart running clusters. Job clusters get the change on their next run. Restarting an all-purpose cluster kills attached notebooks and streams, so warn users first.

#Step 3: NSG and routes

DNS returns the right IP but the connection times out. Check what both workspace subnets allow and where their traffic goes:

az network vnet subnet show -g rg-net -n snet-dbx-container --vnet-name vnet-dbx-prod \
  --query "{nsg:networkSecurityGroup.id, routeTable:routeTable.id, natGateway:natGateway.id}" -o json

az network nsg rule list -g rg-net --nsg-name nsg-dbx-prod \
  --query "[?direction=='Outbound'].{p:priority, name:name, access:access, dst:destinationAddressPrefix, ports:destinationPortRange}" -o table

az network route-table route list -g rg-net --route-table-name rt-dbx-prod -o table

The rules Databricks adds to the NSG cover only the control plane and intra-cluster traffic. A custom outbound Deny at a lower priority number than your allow rule blocks ports such as 1433, 1521, 5432 and 443.

If your account can read the workspace's managed resource group, you can see the effective result on a real cluster NIC:

az network nic list -g <managed-rg-name> --query "[].name" -o tsv | head -3
az network nic show-effective-route-table -g <managed-rg-name> -n <nic-name> -o table
az network nic list-effective-nsg -g <managed-rg-name> -n <nic-name>

Common route problems:

• A 0.0.0.0/0 route to a firewall or NVA, and the firewall has no rule for the destination.
• On-prem prefixes that are missing because the gateway route propagation is disabled on the route table, or the hub doesn't advertise them.
• No outbound path at all: no NAT gateway, no firewall and no public IPs (secure cluster connectivity). Azure is retiring default outbound access, so give the subnets an explicit egress path, usually a NAT gateway or a firewall.

#Step 4: the firewall in the middle

If traffic goes through Azure Firewall, its logs show the drop. With resource-specific diagnostic tables enabled:

AZFWNetworkRule
| where TimeGenerated > ago(1h)
| where SourceIp startswith "10.20."            // your Databricks subnet prefix
| where DestinationPort in (1433, 1521, 5432)
| summarize hits = count() by SourceIp, DestinationIp, DestinationPort, Action
| order by hits desc

AZFWApplicationRule
| where TimeGenerated > ago(1h)
| where SourceIp startswith "10.20." and Action == "Deny"
| summarize hits = count() by Fqdn, DestinationPort
| order by hits desc

Allow both workspace subnet ranges as the source. For HTTPS APIs behind TLS inspection, either exclude the FQDN from inspection or add the inspection CA to the cluster trust store with an init script. The first is usually less work.

#Step 5: the target's own firewall

If the path is open, the target can still refuse you:

# Azure SQL: is public access off, and what IPs are allowed?
az sql server show -g rg-sql -n sql-prod-01 --query "{public:publicNetworkAccess}" -o json
az sql server firewall-rule list -g rg-sql --server sql-prod-01 -o table
az network private-endpoint list -g rg-sql --query "[].{name:name, subnet:subnet.id, state:privateLinkServiceConnections[0].privateLinkServiceConnectionState.status}" -o table

• Through a private endpoint: Azure SQL sees the private source IP. Public firewall rules don't apply. Check that the private endpoint connection is Approved.
• Over the internet: the target sees your NAT gateway or firewall public IP, not the cluster IP. Allow that address:

az network nat gateway show -g rg-net -n nat-dbx-prod --query "publicIpAddresses[].id" -o tsv \
  | xargs -I{} az network public-ip show --ids {} --query ipAddress -o tsv

• On-prem SQL Server or Oracle: the on-prem firewall must allow the Databricks subnet CIDRs. Requests that allow one tested IP break as soon as the cluster autoscales onto new nodes.

#Fix summary

Caution

Don't fix a timeout by adding a broad Allow Any Any outbound NSG rule or by enabling public network access on the database. Those changes outlive the incident. Open only the destination and port, and record the change with an expiry review.

#Prevent it next time

• Keep a list of every external source each workspace connects to (host, port, path: private endpoint, on-prem or internet), and keep it in the same repo as the network IaC.
• Add the probe notebook as the first task of critical jobs that read external sources, so a network failure shows up as a clear task error rather than a 15-minute JDBC timeout.
• Manage privatelink.* zones and their VNet links centrally with Azure Policy, so a new workspace VNet is linked automatically.
• Send Azure Firewall and NSG flow logs to Log Analytics, and alert on denies from the Databricks subnet ranges.
• When a workload moves to serverless, repeat the connectivity setup in the NCC. Classic-compute network rules don't apply to serverless.