Databricks serverless can't reach ADLS behind a firewall: 403 errors fixed with network connectivity configurations (NCC)

#Symptoms

• A query on a serverless SQL warehouse fails with 403 This request is not authorized to perform this operation or AuthorizationFailure from storage, while the same table works on a classic cluster.
• Serverless notebooks or jobs fail reading abfss:// paths or external volumes with timeouts or 403s.
• Everything worked until the storage team set the account to Selected networks or Disabled public access.

#Why it happens

Classic compute runs in your VNet (VNet injection), so storage firewall rules for your subnets or private endpoints in your VNet cover it. Serverless compute runs in the Databricks serverless compute plane, not your VNet. Your storage firewall doesn't know it.

Network connectivity configurations (NCCs) are an account-level object that give serverless a known way in:

#Step 1: Confirm it's networking, not permissions

On a classic cluster with the same identity, does SELECT work? If yes, the Unity Catalog grants and storage credential are fine; it's network.

Check the storage account's network settings:

az storage account show -n stdatalakeprod -g rg-data \
  --query "{public:publicNetworkAccess, default:networkRuleSet.defaultAction, vnetRules:networkRuleSet.virtualNetworkRules[].virtualNetworkResourceId, resourceRules:networkRuleSet.resourceAccessRules}" -o json

Storage diagnostic logs show the denied calls and the caller's IP/network; useful if you have them turned on:

StorageBlobLogs
| where TimeGenerated > ago(1h)
| where StatusCode == 403
| project TimeGenerated, OperationName, StatusText, CallerIpAddress, Uri

StatusText of AuthorizationFailure / IP-related messages point to the firewall.

#Step 2: Create or find an NCC and attach it to the workspace

NCCs are managed in the account console (accounts.azuredatabricks.net) under Security > Network connectivity configurations, or through the Account API. An NCC is regional and can be attached to several workspaces in that region.

With the Databricks CLI configured for the account (host https://accounts.azuredatabricks.net, your account ID):

databricks account network-connectivity create-network-connectivity-configuration \
  --json '{"name":"ncc-centralindia-prod","region":"centralindia"}'

databricks account network-connectivity list-network-connectivity-configurations

Attach it to the workspace (account console: Workspaces > your workspace > Network connectivity configuration, or the workspace update API). Command names can change between CLI releases; databricks account network-connectivity --help lists the current ones.

Tip

After attaching an NCC, serverless compute can take several minutes to pick up the change. Stop and start the SQL warehouse to make sure it uses the new configuration.

#Step 3a: Storage with "selected networks": add the NCC subnets

Get the stable subnet IDs from the NCC (account console shows them under Default rules, or in the API response under egress_config.default_rules.azure_service_endpoint_rule.subnets).

Add each subnet to the storage firewall:

for SUBNET in $(cat ncc-subnets.txt); do
  az storage account network-rule add -g rg-data --account-name stdatalakeprod --subnet "$SUBNET"
done

These subnets belong to Databricks-managed VNets in the region. The Azure CLI may need --subnet as a full resource ID, which is what the NCC gives you.

#Step 3b: Storage with public access disabled: private endpoint rules

Create a private endpoint rule in the NCC for each storage account and sub-resource you need (dfs for ADLS Gen2, and blob too if anything uses the blob endpoint):

databricks account network-connectivity create-private-endpoint-rule <ncc-id> \
  --json '{"resource_id":"/subscriptions/<sub>/resourceGroups/rg-data/providers/Microsoft.Storage/storageAccounts/stdatalakeprod","group_id":"dfs"}'

Then approve the pending private endpoint connection on the storage account:

az network private-endpoint-connection list --id \
  /subscriptions/<sub>/resourceGroups/rg-data/providers/Microsoft.Storage/storageAccounts/stdatalakeprod \
  --query "[?properties.privateLinkServiceConnectionState.status=='Pending'].{name:name, id:id}" -o table

az network private-endpoint-connection approve --id <connection-id> \
  --description "Databricks serverless NCC"

Until it's approved, the rule stays pending and serverless still can't connect.

Warning

Each NCC has limits on the number of private endpoint rules and storage accounts it can serve (check current Databricks limits). Plan one NCC per region shared by workspaces, and list which storage accounts it covers.

#Step 4: Don't forget the other resources

Serverless needs a path to everything the workload touches, not only the main lake:

#Step 5: Test

-- On the serverless SQL warehouse
LIST 'abfss://landing@stdatalakeprod.dfs.core.windows.net/sales/';
SELECT COUNT(*) FROM main.sales.orders_bronze;

In a serverless notebook:

display(dbutils.fs.ls("/Volumes/main/sales/landing/"))

#Symptom → fix table

#Prevent it next time

• Create the NCC with the workspace in your landing zone automation, before anyone enables serverless.
• Keep a list (in Terraform) of every storage account and service serverless must reach, and create NCC rules from it.
• Agree a change process with the storage team: firewall changes on lake accounts must include the NCC subnets or private endpoints.
• Add a scheduled serverless test query per catalog that alerts on failure.